data_scans

Creates, updates, deletes, gets or lists a data_scans resource.

Overview

Name	data_scans
Type	Resource
Id	google.dataplex.data_scans

Fields

The following fields are returned by SELECT queries:

projects_locations_data_scans_get

Name	Datatype	Description
name	string	Output only. Identifier. The relative resource name of the scan, of the form: projects/{project}/locations/{location_id}/dataScans/{datascan_id}, where project refers to a project_id or project_number and location_id refers to a Google Cloud region.
createTime	string (google-datetime)	Output only. The time when the scan was created.
data	object	Required. The data source for DataScan. (id: GoogleCloudDataplexV1DataSource)
dataDiscoveryResult	object	Output only. The result of a data discovery scan. (id: GoogleCloudDataplexV1DataDiscoveryResult)
dataDiscoverySpec	object	Settings for a data discovery scan. (id: GoogleCloudDataplexV1DataDiscoverySpec)
dataDocumentationResult	object	Output only. The result of a data documentation scan. (id: GoogleCloudDataplexV1DataDocumentationResult)
dataDocumentationSpec	object	Settings for a data documentation scan. (id: GoogleCloudDataplexV1DataDocumentationSpec)
dataProfileResult	object	Output only. The result of a data profile scan. (id: GoogleCloudDataplexV1DataProfileResult)
dataProfileSpec	object	Settings for a data profile scan. (id: GoogleCloudDataplexV1DataProfileSpec)
dataQualityResult	object	Output only. The result of a data quality scan. (id: GoogleCloudDataplexV1DataQualityResult)
dataQualitySpec	object	Settings for a data quality scan. (id: GoogleCloudDataplexV1DataQualitySpec)
description	string	Optional. Description of the scan. Must be between 1-1024 characters.
displayName	string	Optional. User friendly display name. Must be between 1-256 characters.
executionIdentity	object	Optional. Immutable. The identity to run the datascan. If not specified, defaults to the Dataplex Service Agent. (id: GoogleCloudDataplexV1ExecutionIdentity)
executionSpec	object	Optional. DataScan execution settings.If not specified, the fields in it will use their default values. (id: GoogleCloudDataplexV1DataScanExecutionSpec)
executionStatus	object	Output only. Status of the data scan execution. (id: GoogleCloudDataplexV1DataScanExecutionStatus)
labels	object	Optional. User-defined labels for the scan.
state	string	Output only. Current state of the DataScan. (STATE_UNSPECIFIED, ACTIVE, CREATING, DELETING, ACTION_REQUIRED)
type	string	Output only. The type of DataScan. (DATA_SCAN_TYPE_UNSPECIFIED, DATA_QUALITY, DATA_PROFILE, DATA_DISCOVERY, DATA_DOCUMENTATION, UNSTRUCTURED_DATA_PROFILE)
uid	string	Output only. System generated globally unique ID for the scan. This ID will be different if the scan is deleted and re-created with the same name.
unstructuredDataProfileResult	object	Output only. The result of an unstructured data profile scan. (id: GoogleCloudDataplexV1UnstructuredDataProfileResult)
unstructuredDataProfileSpec	object	Optional. Settings for an unstructured data profile scan. (id: GoogleCloudDataplexV1UnstructuredDataProfileSpec)
updateTime	string (google-datetime)	Output only. The time when the scan was last updated.

projects_locations_data_scans_list

Name	Datatype	Description
name	string	Output only. Identifier. The relative resource name of the scan, of the form: projects/{project}/locations/{location_id}/dataScans/{datascan_id}, where project refers to a project_id or project_number and location_id refers to a Google Cloud region.
createTime	string (google-datetime)	Output only. The time when the scan was created.
data	object	Required. The data source for DataScan. (id: GoogleCloudDataplexV1DataSource)
dataDiscoveryResult	object	Output only. The result of a data discovery scan. (id: GoogleCloudDataplexV1DataDiscoveryResult)
dataDiscoverySpec	object	Settings for a data discovery scan. (id: GoogleCloudDataplexV1DataDiscoverySpec)
dataDocumentationResult	object	Output only. The result of a data documentation scan. (id: GoogleCloudDataplexV1DataDocumentationResult)
dataDocumentationSpec	object	Settings for a data documentation scan. (id: GoogleCloudDataplexV1DataDocumentationSpec)
dataProfileResult	object	Output only. The result of a data profile scan. (id: GoogleCloudDataplexV1DataProfileResult)
dataProfileSpec	object	Settings for a data profile scan. (id: GoogleCloudDataplexV1DataProfileSpec)
dataQualityResult	object	Output only. The result of a data quality scan. (id: GoogleCloudDataplexV1DataQualityResult)
dataQualitySpec	object	Settings for a data quality scan. (id: GoogleCloudDataplexV1DataQualitySpec)
description	string	Optional. Description of the scan. Must be between 1-1024 characters.
displayName	string	Optional. User friendly display name. Must be between 1-256 characters.
executionIdentity	object	Optional. Immutable. The identity to run the datascan. If not specified, defaults to the Dataplex Service Agent. (id: GoogleCloudDataplexV1ExecutionIdentity)
executionSpec	object	Optional. DataScan execution settings.If not specified, the fields in it will use their default values. (id: GoogleCloudDataplexV1DataScanExecutionSpec)
executionStatus	object	Output only. Status of the data scan execution. (id: GoogleCloudDataplexV1DataScanExecutionStatus)
labels	object	Optional. User-defined labels for the scan.
state	string	Output only. Current state of the DataScan. (STATE_UNSPECIFIED, ACTIVE, CREATING, DELETING, ACTION_REQUIRED)
type	string	Output only. The type of DataScan. (DATA_SCAN_TYPE_UNSPECIFIED, DATA_QUALITY, DATA_PROFILE, DATA_DISCOVERY, DATA_DOCUMENTATION, UNSTRUCTURED_DATA_PROFILE)
uid	string	Output only. System generated globally unique ID for the scan. This ID will be different if the scan is deleted and re-created with the same name.
unstructuredDataProfileResult	object	Output only. The result of an unstructured data profile scan. (id: GoogleCloudDataplexV1UnstructuredDataProfileResult)
unstructuredDataProfileSpec	object	Optional. Settings for an unstructured data profile scan. (id: GoogleCloudDataplexV1UnstructuredDataProfileSpec)
updateTime	string (google-datetime)	Output only. The time when the scan was last updated.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
projects_locations_data_scans_get	select	projectsId, locationsId, dataScansId	view	Gets a DataScan resource.
projects_locations_data_scans_list	select	projectsId, locationsId	filter, pageSize, orderBy, pageToken	Lists DataScans.
projects_locations_data_scans_create	insert	projectsId, locationsId	dataScanId, validateOnly	Creates a DataScan resource.
projects_locations_data_scans_patch	update	projectsId, locationsId, dataScansId	updateMask, validateOnly	Updates a DataScan resource.
projects_locations_data_scans_delete	delete	projectsId, locationsId, dataScansId	force	Deletes a DataScan resource.
projects_locations_data_scans_generate_data_quality_rules	exec	projectsId, locationsId, dataScansId		Generates recommended data quality rules based on the results of a data profiling scan.Use the recommendations to build rules for a data quality scan.
projects_locations_data_scans_run	exec	projectsId, locationsId, dataScansId		Runs an on-demand execution of a DataScan

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
dataScansId	string
locationsId	string
projectsId	string
dataScanId	string
filter	string
force	boolean
orderBy	string
pageSize	integer (int32)
pageToken	string
updateMask	string (google-fieldmask)
validateOnly	boolean
view	string

SELECT examples

projects_locations_data_scans_get

Gets a DataScan resource.

SELECT
name,
createTime,
data,
dataDiscoveryResult,
dataDiscoverySpec,
dataDocumentationResult,
dataDocumentationSpec,
dataProfileResult,
dataProfileSpec,
dataQualityResult,
dataQualitySpec,
description,
displayName,
executionIdentity,
executionSpec,
executionStatus,
labels,
state,
type,
uid,
unstructuredDataProfileResult,
unstructuredDataProfileSpec,
updateTime
FROM google.dataplex.data_scans
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND dataScansId = '{{ dataScansId }}' -- required
AND view = '{{ view }}'
;

projects_locations_data_scans_list

Lists DataScans.

SELECT
name,
createTime,
data,
dataDiscoveryResult,
dataDiscoverySpec,
dataDocumentationResult,
dataDocumentationSpec,
dataProfileResult,
dataProfileSpec,
dataQualityResult,
dataQualitySpec,
description,
displayName,
executionIdentity,
executionSpec,
executionStatus,
labels,
state,
type,
uid,
unstructuredDataProfileResult,
unstructuredDataProfileSpec,
updateTime
FROM google.dataplex.data_scans
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND filter = '{{ filter }}'
AND pageSize = '{{ pageSize }}'
AND orderBy = '{{ orderBy }}'
AND pageToken = '{{ pageToken }}'
;

INSERT examples

projects_locations_data_scans_create

Creates a DataScan resource.

INSERT INTO google.dataplex.data_scans (
data__unstructuredDataProfileSpec,
data__description,
data__dataDocumentationSpec,
data__dataDiscoverySpec,
data__executionIdentity,
data__dataProfileSpec,
data__labels,
data__dataQualitySpec,
data__executionSpec,
data__data,
data__displayName,
projectsId,
locationsId,
dataScanId,
validateOnly
)
SELECT 
'{{ unstructuredDataProfileSpec }}',
'{{ description }}',
'{{ dataDocumentationSpec }}',
'{{ dataDiscoverySpec }}',
'{{ executionIdentity }}',
'{{ dataProfileSpec }}',
'{{ labels }}',
'{{ dataQualitySpec }}',
'{{ executionSpec }}',
'{{ data }}',
'{{ displayName }}',
'{{ projectsId }}',
'{{ locationsId }}',
'{{ dataScanId }}',
'{{ validateOnly }}'
RETURNING
name,
done,
error,
metadata,
response
;

Manifest

# Description fields are for documentation purposes
- name: data_scans
props:
  - name: projectsId
    value: "{{ projectsId }}"
    description: Required parameter for the data_scans resource.
  - name: locationsId
    value: "{{ locationsId }}"
    description: Required parameter for the data_scans resource.
  - name: unstructuredDataProfileSpec
    description: |
      Optional. Settings for an unstructured data profile scan.
    value:
      graphProfilePublishingEnabled: {{ graphProfilePublishingEnabled }}
      customizedPrompt: "{{ customizedPrompt }}"
      globalEndpointEnabled: {{ globalEndpointEnabled }}
  - name: description
    value: "{{ description }}"
    description: |
      Optional. Description of the scan. Must be between 1-1024 characters.
  - name: dataDocumentationSpec
    description: |
      Settings for a data documentation scan.
    value:
      catalogPublishingEnabled: {{ catalogPublishingEnabled }}
      generationScopes:
        - "{{ generationScopes }}"
  - name: dataDiscoverySpec
    description: |
      Settings for a data discovery scan.
    value:
      bigqueryPublishingConfig:
        tableType: "{{ tableType }}"
        location: "{{ location }}"
        project: "{{ project }}"
        connection: "{{ connection }}"
      storageConfig:
        csvOptions:
          delimiter: "{{ delimiter }}"
          quote: "{{ quote }}"
          headerRows: {{ headerRows }}
          typeInferenceDisabled: {{ typeInferenceDisabled }}
          encoding: "{{ encoding }}"
        unstructuredDataOptions:
          semanticInferenceEnabled: {{ semanticInferenceEnabled }}
          globalEndpointEnabled: {{ globalEndpointEnabled }}
        excludePatterns:
          - "{{ excludePatterns }}"
        jsonOptions:
          encoding: "{{ encoding }}"
          typeInferenceDisabled: {{ typeInferenceDisabled }}
        includePatterns:
          - "{{ includePatterns }}"
  - name: executionIdentity
    description: |
      Optional. Immutable. The identity to run the datascan. If not specified, defaults to the Dataplex Service Agent.
    value:
      dataplexServiceAgent: "{{ dataplexServiceAgent }}"
      userCredential: "{{ userCredential }}"
      serviceAccount:
        email: "{{ email }}"
  - name: dataProfileSpec
    description: |
      Settings for a data profile scan.
    value:
      includeFields:
        fieldNames:
          - "{{ fieldNames }}"
      catalogPublishingEnabled: {{ catalogPublishingEnabled }}
      mode: "{{ mode }}"
      samplingPercent: {{ samplingPercent }}
      rowFilter: "{{ rowFilter }}"
      postScanActions:
        bigqueryExport:
          resultsTable: "{{ resultsTable }}"
      excludeFields:
        fieldNames:
          - "{{ fieldNames }}"
  - name: labels
    value: "{{ labels }}"
    description: |
      Optional. User-defined labels for the scan.
  - name: dataQualitySpec
    description: |
      Settings for a data quality scan.
    value:
      enableCatalogBasedRules: {{ enableCatalogBasedRules }}
      rowFilter: "{{ rowFilter }}"
      postScanActions:
        notificationReport:
          scoreThresholdTrigger:
            scoreThreshold: {{ scoreThreshold }}
          recipients:
            emails:
              - "{{ emails }}"
          jobFailureTrigger: "{{ jobFailureTrigger }}"
          jobEndTrigger: "{{ jobEndTrigger }}"
        bigqueryExport:
          resultsTable: "{{ resultsTable }}"
      rules:
        - setExpectation:
            values:
              - "{{ values }}"
          dimension: "{{ dimension }}"
          description: "{{ description }}"
          attributes: "{{ attributes }}"
          templateReference:
            name: "{{ name }}"
            resolvedSql: "{{ resolvedSql }}"
            values: "{{ values }}"
            ruleTemplate:
              capabilities:
                - "{{ capabilities }}"
              name: "{{ name }}"
              dimension: "{{ dimension }}"
              inputParameters: "{{ inputParameters }}"
              sqlCollection:
                - query: "{{ query }}"
          threshold: {{ threshold }}
          rowConditionExpectation:
            sqlExpression: "{{ sqlExpression }}"
          sqlAssertion:
            sqlStatement: "{{ sqlStatement }}"
          regexExpectation:
            regex: "{{ regex }}"
          statisticRangeExpectation:
            minValue: "{{ minValue }}"
            strictMinEnabled: {{ strictMinEnabled }}
            strictMaxEnabled: {{ strictMaxEnabled }}
            statistic: "{{ statistic }}"
            maxValue: "{{ maxValue }}"
          ruleSource:
            rulePathElements:
              - entryLinkSource:
                  entryLink: "{{ entryLink }}"
                  entryLinkType: "{{ entryLinkType }}"
                entrySource:
                  entryType: "{{ entryType }}"
                  displayName: "{{ displayName }}"
                  entry: "{{ entry }}"
          uniquenessExpectation: "{{ uniquenessExpectation }}"
          debugQueries: "{{ debugQueries }}"
          name: "{{ name }}"
          nonNullExpectation: "{{ nonNullExpectation }}"
          column: "{{ column }}"
          ignoreNull: {{ ignoreNull }}
          suspended: {{ suspended }}
          tableConditionExpectation:
            sqlExpression: "{{ sqlExpression }}"
          rangeExpectation:
            minValue: "{{ minValue }}"
            strictMinEnabled: {{ strictMinEnabled }}
            strictMaxEnabled: {{ strictMaxEnabled }}
            maxValue: "{{ maxValue }}"
      filter: "{{ filter }}"
      catalogPublishingEnabled: {{ catalogPublishingEnabled }}
      samplingPercent: {{ samplingPercent }}
  - name: executionSpec
    description: |
      Optional. DataScan execution settings.If not specified, the fields in it will use their default values.
    value:
      trigger:
        schedule:
          cron: "{{ cron }}"
        onDemand: "{{ onDemand }}"
        oneTime:
          ttlAfterScanCompletion: "{{ ttlAfterScanCompletion }}"
      field: "{{ field }}"
  - name: data
    description: |
      Required. The data source for DataScan.
    value:
      resource: "{{ resource }}"
      entity: "{{ entity }}"
  - name: displayName
    value: "{{ displayName }}"
    description: |
      Optional. User friendly display name. Must be between 1-256 characters.
  - name: dataScanId
    value: "{{ dataScanId }}"
  - name: validateOnly
    value: {{ validateOnly }}

UPDATE examples

projects_locations_data_scans_patch

Updates a DataScan resource.

UPDATE google.dataplex.data_scans
SET 
data__unstructuredDataProfileSpec = '{{ unstructuredDataProfileSpec }}',
data__description = '{{ description }}',
data__dataDocumentationSpec = '{{ dataDocumentationSpec }}',
data__dataDiscoverySpec = '{{ dataDiscoverySpec }}',
data__executionIdentity = '{{ executionIdentity }}',
data__dataProfileSpec = '{{ dataProfileSpec }}',
data__labels = '{{ labels }}',
data__dataQualitySpec = '{{ dataQualitySpec }}',
data__executionSpec = '{{ executionSpec }}',
data__data = '{{ data }}',
data__displayName = '{{ displayName }}'
WHERE 
projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND dataScansId = '{{ dataScansId }}' --required
AND updateMask = '{{ updateMask}}'
AND validateOnly = {{ validateOnly}}
RETURNING
name,
done,
error,
metadata,
response;

DELETE examples

projects_locations_data_scans_delete

Deletes a DataScan resource.

DELETE FROM google.dataplex.data_scans
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND dataScansId = '{{ dataScansId }}' --required
AND force = '{{ force }}'
;

Lifecycle Methods

projects_locations_data_scans_generate_data_quality_rules

Generates recommended data quality rules based on the results of a data profiling scan.Use the recommendations to build rules for a data quality scan.

EXEC google.dataplex.data_scans.projects_locations_data_scans_generate_data_quality_rules 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@dataScansId='{{ dataScansId }}' --required
;

projects_locations_data_scans_run

Runs an on-demand execution of a DataScan

EXEC google.dataplex.data_scans.projects_locations_data_scans_run 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@dataScansId='{{ dataScansId }}' --required
;