streams

Creates, updates, deletes, gets or lists a streams resource.

Overview

Name	streams
Type	Resource
Id	google.datastream.streams

Fields

The following fields are returned by SELECT queries:

get

Name	Datatype	Description
name	string	Output only. Identifier. The stream's name.
backfillAll	object	Automatically backfill objects included in the stream source configuration. Specific objects can be excluded. (id: BackfillAllStrategy)
backfillNone	object	Do not automatically backfill any objects. (id: BackfillNoneStrategy)
createTime	string (google-datetime)	Output only. The creation time of the stream.
customerManagedEncryptionKey	string	Immutable. A reference to a KMS encryption key. If provided, it will be used to encrypt the data. If left blank, data will be encrypted using an internal Stream-specific encryption key provisioned through KMS.
destinationConfig	object	Required. Destination connection profile configuration. (id: DestinationConfig)
displayName	string	Required. Display name.
errors	array	Output only. Errors on the Stream.
labels	object	Labels.
lastRecoveryTime	string (google-datetime)	Output only. If the stream was recovered, the time of the last recovery. Note: This field is currently experimental.
ruleSets	array	Optional. Rule sets to apply to the stream.
satisfiesPzi	boolean	Output only. Reserved for future use.
satisfiesPzs	boolean	Output only. Reserved for future use.
sourceConfig	object	Required. Source connection profile configuration. (id: SourceConfig)
state	string	The state of the stream. (STATE_UNSPECIFIED, NOT_STARTED, RUNNING, PAUSED, MAINTENANCE, FAILED, FAILED_PERMANENTLY, STARTING, DRAINING)
updateTime	string (google-datetime)	Output only. The last update time of the stream.

list

Name	Datatype	Description
name	string	Output only. Identifier. The stream's name.
backfillAll	object	Automatically backfill objects included in the stream source configuration. Specific objects can be excluded. (id: BackfillAllStrategy)
backfillNone	object	Do not automatically backfill any objects. (id: BackfillNoneStrategy)
createTime	string (google-datetime)	Output only. The creation time of the stream.
customerManagedEncryptionKey	string	Immutable. A reference to a KMS encryption key. If provided, it will be used to encrypt the data. If left blank, data will be encrypted using an internal Stream-specific encryption key provisioned through KMS.
destinationConfig	object	Required. Destination connection profile configuration. (id: DestinationConfig)
displayName	string	Required. Display name.
errors	array	Output only. Errors on the Stream.
labels	object	Labels.
lastRecoveryTime	string (google-datetime)	Output only. If the stream was recovered, the time of the last recovery. Note: This field is currently experimental.
ruleSets	array	Optional. Rule sets to apply to the stream.
satisfiesPzi	boolean	Output only. Reserved for future use.
satisfiesPzs	boolean	Output only. Reserved for future use.
sourceConfig	object	Required. Source connection profile configuration. (id: SourceConfig)
state	string	The state of the stream. (STATE_UNSPECIFIED, NOT_STARTED, RUNNING, PAUSED, MAINTENANCE, FAILED, FAILED_PERMANENTLY, STARTING, DRAINING)
updateTime	string (google-datetime)	Output only. The last update time of the stream.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	projectsId, locationsId, streamsId		Use this method to get details about a stream.
list	select	projectsId, locationsId	pageToken, filter, pageSize, orderBy	Use this method to list streams in a project and location.
create	insert	projectsId, locationsId	requestId, validateOnly, streamId, force	Use this method to create a stream.
patch	update	projectsId, locationsId, streamsId	requestId, updateMask, validateOnly, force	Use this method to update the configuration of a stream.
delete	delete	projectsId, locationsId, streamsId	requestId	Use this method to delete a stream.
run	exec	projectsId, locationsId, streamsId		Use this method to start, resume or recover a stream with a non default CDC strategy.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
locationsId	string
projectsId	string
streamsId	string
filter	string
force	boolean
orderBy	string
pageSize	integer (int32)
pageToken	string
requestId	string
streamId	string
updateMask	string (google-fieldmask)
validateOnly	boolean

SELECT examples

get

Use this method to get details about a stream.

SELECT
name,
backfillAll,
backfillNone,
createTime,
customerManagedEncryptionKey,
destinationConfig,
displayName,
errors,
labels,
lastRecoveryTime,
ruleSets,
satisfiesPzi,
satisfiesPzs,
sourceConfig,
state,
updateTime
FROM google.datastream.streams
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND streamsId = '{{ streamsId }}' -- required
;

list

Use this method to list streams in a project and location.

SELECT
name,
backfillAll,
backfillNone,
createTime,
customerManagedEncryptionKey,
destinationConfig,
displayName,
errors,
labels,
lastRecoveryTime,
ruleSets,
satisfiesPzi,
satisfiesPzs,
sourceConfig,
state,
updateTime
FROM google.datastream.streams
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND pageToken = '{{ pageToken }}'
AND filter = '{{ filter }}'
AND pageSize = '{{ pageSize }}'
AND orderBy = '{{ orderBy }}'
;

INSERT examples

create

Use this method to create a stream.

INSERT INTO google.datastream.streams (
data__displayName,
data__destinationConfig,
data__backfillAll,
data__sourceConfig,
data__labels,
data__customerManagedEncryptionKey,
data__backfillNone,
data__state,
data__ruleSets,
projectsId,
locationsId,
requestId,
validateOnly,
streamId,
force
)
SELECT 
'{{ displayName }}',
'{{ destinationConfig }}',
'{{ backfillAll }}',
'{{ sourceConfig }}',
'{{ labels }}',
'{{ customerManagedEncryptionKey }}',
'{{ backfillNone }}',
'{{ state }}',
'{{ ruleSets }}',
'{{ projectsId }}',
'{{ locationsId }}',
'{{ requestId }}',
'{{ validateOnly }}',
'{{ streamId }}',
'{{ force }}'
RETURNING
name,
done,
error,
metadata,
response
;

Manifest

# Description fields are for documentation purposes
- name: streams
props:
  - name: projectsId
    value: "{{ projectsId }}"
    description: Required parameter for the streams resource.
  - name: locationsId
    value: "{{ locationsId }}"
    description: Required parameter for the streams resource.
  - name: displayName
    value: "{{ displayName }}"
    description: |
      Required. Display name.
  - name: destinationConfig
    description: |
      Required. Destination connection profile configuration.
    value:
      bigqueryDestinationConfig:
        appendOnly: "{{ appendOnly }}"
        dataFreshness: "{{ dataFreshness }}"
        singleTargetDataset:
          datasetId: "{{ datasetId }}"
        sourceHierarchyDatasets:
          datasetTemplate:
            datasetIdPrefix: "{{ datasetIdPrefix }}"
            location: "{{ location }}"
            kmsKeyName: "{{ kmsKeyName }}"
          projectId: "{{ projectId }}"
        blmtConfig:
          rootPath: "{{ rootPath }}"
          bucket: "{{ bucket }}"
          tableFormat: "{{ tableFormat }}"
          connectionName: "{{ connectionName }}"
          fileFormat: "{{ fileFormat }}"
        merge: "{{ merge }}"
      gcsDestinationConfig:
        avroFileFormat: "{{ avroFileFormat }}"
        jsonFileFormat:
          schemaFileFormat: "{{ schemaFileFormat }}"
          compression: "{{ compression }}"
        path: "{{ path }}"
        fileRotationMb: {{ fileRotationMb }}
        fileRotationInterval: "{{ fileRotationInterval }}"
      destinationConnectionProfile: "{{ destinationConnectionProfile }}"
  - name: backfillAll
    description: |
      Automatically backfill objects included in the stream source configuration. Specific objects can be excluded.
    value:
      mongodbExcludedObjects:
        databases:
          - database: "{{ database }}"
            collections: "{{ collections }}"
      salesforceExcludedObjects:
        objects:
          - objectName: "{{ objectName }}"
            fields: "{{ fields }}"
      spannerExcludedObjects:
        schemas:
          - tables: "{{ tables }}"
            schema: "{{ schema }}"
      saasExcludedObjects:
        objects:
          - properties: "{{ properties }}"
            objectName: "{{ objectName }}"
      oracleExcludedObjects:
        oracleSchemas:
          - schema: "{{ schema }}"
            oracleTables: "{{ oracleTables }}"
      mysqlExcludedObjects:
        mysqlDatabases:
          - database: "{{ database }}"
            mysqlTables: "{{ mysqlTables }}"
      sqlServerExcludedObjects:
        schemas:
          - schema: "{{ schema }}"
            tables: "{{ tables }}"
      postgresqlExcludedObjects:
        postgresqlSchemas:
          - schema: "{{ schema }}"
            postgresqlTables: "{{ postgresqlTables }}"
  - name: sourceConfig
    description: |
      Required. Source connection profile configuration.
    value:
      serviceNowSourceConfig:
        includeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        excludeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        pollingInterval: "{{ pollingInterval }}"
      postgresqlSourceConfig:
        includeObjects:
          postgresqlSchemas:
            - schema: "{{ schema }}"
              postgresqlTables: "{{ postgresqlTables }}"
        publication: "{{ publication }}"
        replicationSlot: "{{ replicationSlot }}"
        excludeObjects:
          postgresqlSchemas:
            - schema: "{{ schema }}"
              postgresqlTables: "{{ postgresqlTables }}"
        maxConcurrentBackfillTasks: {{ maxConcurrentBackfillTasks }}
      mysqlSourceConfig:
        excludeObjects:
          mysqlDatabases:
            - database: "{{ database }}"
              mysqlTables: "{{ mysqlTables }}"
        maxConcurrentBackfillTasks: {{ maxConcurrentBackfillTasks }}
        binaryLogPosition: "{{ binaryLogPosition }}"
        gtid: "{{ gtid }}"
        maxConcurrentCdcTasks: {{ maxConcurrentCdcTasks }}
        includeObjects:
          mysqlDatabases:
            - database: "{{ database }}"
              mysqlTables: "{{ mysqlTables }}"
      spannerSourceConfig:
        changeStreamName: "{{ changeStreamName }}"
        includeObjects:
          schemas:
            - tables: "{{ tables }}"
              schema: "{{ schema }}"
        fgacRole: "{{ fgacRole }}"
        maxConcurrentCdcTasks: {{ maxConcurrentCdcTasks }}
        backfillDataBoostEnabled: {{ backfillDataBoostEnabled }}
        excludeObjects:
          schemas:
            - tables: "{{ tables }}"
              schema: "{{ schema }}"
        maxConcurrentBackfillTasks: {{ maxConcurrentBackfillTasks }}
        spannerRpcPriority: "{{ spannerRpcPriority }}"
      salesforceSourceConfig:
        pollingInterval: "{{ pollingInterval }}"
        includeObjects:
          objects:
            - objectName: "{{ objectName }}"
              fields: "{{ fields }}"
        excludeObjects:
          objects:
            - objectName: "{{ objectName }}"
              fields: "{{ fields }}"
      sourceConnectionProfile: "{{ sourceConnectionProfile }}"
      oracleSourceConfig:
        includeObjects:
          oracleSchemas:
            - schema: "{{ schema }}"
              oracleTables: "{{ oracleTables }}"
        logMiner: "{{ logMiner }}"
        maxConcurrentCdcTasks: {{ maxConcurrentCdcTasks }}
        maxConcurrentBackfillTasks: {{ maxConcurrentBackfillTasks }}
        excludeObjects:
          oracleSchemas:
            - schema: "{{ schema }}"
              oracleTables: "{{ oracleTables }}"
        binaryLogParser:
          logFileDirectories:
            onlineLogDirectory: "{{ onlineLogDirectory }}"
            archivedLogDirectory: "{{ archivedLogDirectory }}"
          oracleAsmLogFileAccess: "{{ oracleAsmLogFileAccess }}"
        dropLargeObjects: "{{ dropLargeObjects }}"
        streamLargeObjects: "{{ streamLargeObjects }}"
      mongodbSourceConfig:
        includeObjects:
          databases:
            - database: "{{ database }}"
              collections: "{{ collections }}"
        excludeObjects:
          databases:
            - database: "{{ database }}"
              collections: "{{ collections }}"
        maxConcurrentBackfillTasks: {{ maxConcurrentBackfillTasks }}
        jsonMode: "{{ jsonMode }}"
      salesforceMarketingCloudSourceConfig:
        includeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        excludeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        fullRefreshPollingInterval: "{{ fullRefreshPollingInterval }}"
        pollingInterval: "{{ pollingInterval }}"
      dataverseSourceConfig:
        includeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        excludeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        pollingInterval: "{{ pollingInterval }}"
      sqlServerSourceConfig:
        includeObjects:
          schemas:
            - schema: "{{ schema }}"
              tables: "{{ tables }}"
        transactionLogs: "{{ transactionLogs }}"
        maxConcurrentCdcTasks: {{ maxConcurrentCdcTasks }}
        excludeObjects:
          schemas:
            - schema: "{{ schema }}"
              tables: "{{ tables }}"
        maxConcurrentBackfillTasks: {{ maxConcurrentBackfillTasks }}
        changeTables: "{{ changeTables }}"
      workdaySourceConfig:
        pollingInterval: "{{ pollingInterval }}"
        includeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
        excludeObjects:
          objects:
            - properties: "{{ properties }}"
              objectName: "{{ objectName }}"
  - name: labels
    value: "{{ labels }}"
    description: |
      Labels.
  - name: customerManagedEncryptionKey
    value: "{{ customerManagedEncryptionKey }}"
    description: |
      Immutable. A reference to a KMS encryption key. If provided, it will be used to encrypt the data. If left blank, data will be encrypted using an internal Stream-specific encryption key provisioned through KMS.
  - name: backfillNone
    value: "{{ backfillNone }}"
    description: |
      Do not automatically backfill any objects.
  - name: state
    value: "{{ state }}"
    description: |
      The state of the stream.
    valid_values: ['STATE_UNSPECIFIED', 'NOT_STARTED', 'RUNNING', 'PAUSED', 'MAINTENANCE', 'FAILED', 'FAILED_PERMANENTLY', 'STARTING', 'DRAINING']
  - name: ruleSets
    description: |
      Optional. Rule sets to apply to the stream.
    value:
      - customizationRules: "{{ customizationRules }}"
        objectFilter:
          sourceObjectIdentifier:
            mongodbIdentifier:
              database: "{{ database }}"
              collection: "{{ collection }}"
            salesforceIdentifier:
              objectName: "{{ objectName }}"
            sqlServerIdentifier:
              schema: "{{ schema }}"
              table: "{{ table }}"
            spannerIdentifier:
              schema: "{{ schema }}"
              table: "{{ table }}"
            mysqlIdentifier:
              database: "{{ database }}"
              table: "{{ table }}"
            oracleIdentifier:
              schema: "{{ schema }}"
              table: "{{ table }}"
            postgresqlIdentifier:
              schema: "{{ schema }}"
              table: "{{ table }}"
  - name: requestId
    value: "{{ requestId }}"
  - name: validateOnly
    value: {{ validateOnly }}
  - name: streamId
    value: "{{ streamId }}"
  - name: force
    value: {{ force }}

UPDATE examples

patch

Use this method to update the configuration of a stream.

UPDATE google.datastream.streams
SET 
data__displayName = '{{ displayName }}',
data__destinationConfig = '{{ destinationConfig }}',
data__backfillAll = '{{ backfillAll }}',
data__sourceConfig = '{{ sourceConfig }}',
data__labels = '{{ labels }}',
data__customerManagedEncryptionKey = '{{ customerManagedEncryptionKey }}',
data__backfillNone = '{{ backfillNone }}',
data__state = '{{ state }}',
data__ruleSets = '{{ ruleSets }}'
WHERE 
projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND streamsId = '{{ streamsId }}' --required
AND requestId = '{{ requestId}}'
AND updateMask = '{{ updateMask}}'
AND validateOnly = {{ validateOnly}}
AND force = {{ force}}
RETURNING
name,
done,
error,
metadata,
response;

DELETE examples

delete

Use this method to delete a stream.

DELETE FROM google.datastream.streams
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND streamsId = '{{ streamsId }}' --required
AND requestId = '{{ requestId }}'
;

Lifecycle Methods

run

Use this method to start, resume or recover a stream with a non default CDC strategy.

EXEC google.datastream.streams.run 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@streamsId='{{ streamsId }}' --required 
@@json=
'{
"cdcStrategy": "{{ cdcStrategy }}", 
"force": {{ force }}
}'
;