Skip to main content

data_sources

Creates, updates, deletes, gets or lists a data_sources resource.

Overview​

Namedata_sources
TypeResource
Idgoogle.bigquerydatatransfer.data_sources

Fields​

The following fields are returned by SELECT queries:

Successful response

NameDatatypeDescription
namestringOutput only. Data source resource name.
authorizationTypestringIndicates the type of authorization.
clientIdstringData source client id which should be used to receive refresh token.
dataRefreshTypestringSpecifies whether the data source supports automatic data refresh for the past few days, and how it's supported. For some data sources, data might not be complete until a few days later, so it's useful to refresh data automatically.
dataSourceIdstringData source id.
defaultDataRefreshWindowDaysinteger (int32)Default data refresh window on days. Only meaningful when data_refresh_type = SLIDING_WINDOW.
defaultSchedulestringDefault data transfer schedule. Examples of valid schedules include: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00.
descriptionstringUser friendly data source description string.
displayNamestringUser friendly data source name.
helpUrlstringUrl for the help document for this data source.
manualRunsDisabledbooleanDisables backfilling and manual run scheduling for the data source.
minimumScheduleIntervalstring (google-duration)The minimum interval for scheduler to schedule runs.
parametersarrayData source parameters.
scopesarrayApi auth scopes for which refresh token needs to be obtained. These are scopes needed by a data source to prepare data and ingest them into BigQuery, e.g., https://www.googleapis.com/auth/bigquery
supportsCustomSchedulebooleanSpecifies whether the data source supports a user defined schedule, or operates on the default schedule. When set to true, user can override default schedule.
supportsMultipleTransfersbooleanDeprecated. This field has no effect.
transferTypestringDeprecated. This field has no effect.
updateDeadlineSecondsinteger (int32)The number of seconds to wait for an update from the data source before the Data Transfer Service marks the transfer as FAILED.

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
projects_locations_data_sources_getselectprojectsId, locationsId, dataSourcesIdRetrieves a supported data source and returns its settings.
projects_data_sources_getselectprojectsId, dataSourcesIdRetrieves a supported data source and returns its settings.
projects_locations_data_sources_listselectprojectsId, locationsIdpageToken, pageSizeLists supported data sources and returns their settings.
projects_data_sources_listselectprojectsIdpageToken, pageSizeLists supported data sources and returns their settings.
projects_enroll_data_sourcesexecprojectsIdEnroll data sources in a user project. This allows users to create transfer configurations for these data sources. They will also appear in the ListDataSources RPC and as such, will appear in the BigQuery UI, and the documents can be found in the public guide for BigQuery Web UI and Data Transfer Service.
projects_data_sources_check_valid_credsexecprojectsId, dataSourcesIdReturns true if valid credentials exist for the given data source and requesting user.
projects_locations_data_sources_check_valid_credsexecprojectsId, locationsId, dataSourcesIdReturns true if valid credentials exist for the given data source and requesting user.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
dataSourcesIdstring
locationsIdstring
projectsIdstring
pageSizeinteger (int32)
pageTokenstring

SELECT examples​

Retrieves a supported data source and returns its settings.

SELECT
name,
authorizationType,
clientId,
dataRefreshType,
dataSourceId,
defaultDataRefreshWindowDays,
defaultSchedule,
description,
displayName,
helpUrl,
manualRunsDisabled,
minimumScheduleInterval,
parameters,
scopes,
supportsCustomSchedule,
supportsMultipleTransfers,
transferType,
updateDeadlineSeconds
FROM google.bigquerydatatransfer.data_sources
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND dataSourcesId = '{{ dataSourcesId }}' -- required;

Lifecycle Methods​

Enroll data sources in a user project. This allows users to create transfer configurations for these data sources. They will also appear in the ListDataSources RPC and as such, will appear in the BigQuery UI, and the documents can be found in the public guide for BigQuery Web UI and Data Transfer Service.

EXEC google.bigquerydatatransfer.data_sources.projects_enroll_data_sources 
@projectsId='{{ projectsId }}' --required 
@@json=
'{
"dataSourceIds": "{{ dataSourceIds }}"
}';