Skip to main content

transfer_configs

Creates, updates, deletes, gets or lists a transfer_configs resource.

Overview​

Nametransfer_configs
TypeResource
Idgoogle.bigquerydatatransfer.transfer_configs

Fields​

The following fields are returned by SELECT queries:

Successful response

NameDatatypeDescription
namestringIdentifier. The resource name of the transfer config. Transfer config names have the form either projects/{project_id}/locations/{region}/transferConfigs/{config_id} or projects/{project_id}/transferConfigs/{config_id}, where config_id is usually a UUID, even though it is not guaranteed or required. The name is ignored when creating a transfer config.
dataRefreshWindowDaysinteger (int32)The number of days to look back to automatically refresh the data. For example, if data_refresh_window_days = 10, then every day BigQuery reingests data for [today-10, today-1], rather than ingesting data for just [today-1]. Only valid if the data source supports the feature. Set the value to 0 to use the default value.
dataSourceIdstringData source ID. This cannot be changed once data transfer is created. The full list of available data source IDs can be returned through an API call: https://cloud.google.com/bigquery-transfer/docs/reference/datatransfer/rest/v1/projects.locations.dataSources/list
datasetRegionstringOutput only. Region in which BigQuery dataset is located.
destinationDatasetIdstringThe BigQuery target dataset id.
disabledbooleanIs this config disabled. When set to true, no runs will be scheduled for this transfer config.
displayNamestringUser specified display name for the data transfer.
emailPreferencesobjectEmail notifications will be sent according to these preferences to the email address of the user who owns this transfer config. (id: EmailPreferences)
encryptionConfigurationobjectThe encryption configuration part. Currently, it is only used for the optional KMS key name. The BigQuery service account of your project must be granted permissions to use the key. Read methods will return the key name applied in effect. Write methods will apply the key if it is present, or otherwise try to apply project default keys if it is absent. (id: EncryptionConfiguration)
errorobjectOutput only. Error code with detailed information about reason of the latest config failure. (id: Status)
nextRunTimestring (google-datetime)Output only. Next time when data transfer will run.
notificationPubsubTopicstringPub/Sub topic where notifications will be sent after transfer runs associated with this transfer config finish. The format for specifying a pubsub topic is: projects/{project_id}/topics/{topic_id}
ownerInfoobjectOutput only. Information about the user whose credentials are used to transfer data. Populated only for transferConfigs.get requests. In case the user information is not available, this field will not be populated. (id: UserInfo)
paramsobjectParameters specific to each data source. For more information see the bq tab in the 'Setting up a data transfer' section for each data source. For example the parameters for Cloud Storage transfers are listed here: https://cloud.google.com/bigquery-transfer/docs/cloud-storage-transfer#bq
schedulestringData transfer schedule. If the data source does not support a custom schedule, this should be empty. If it is empty, the default value for the data source will be used. The specified times are in UTC. Examples of valid format: 1st,3rd monday of month 15:30, every wed,fri of jan,jun 13:15, and first sunday of quarter 00:00. See more explanation about the format here: https://cloud.google.com/appengine/docs/flexible/python/scheduling-jobs-with-cron-yaml#the_schedule_format NOTE: The minimum interval time between recurring transfers depends on the data source; refer to the documentation for your data source.
scheduleOptionsobjectOptions customizing the data transfer schedule. (id: ScheduleOptions)
scheduleOptionsV2objectOptions customizing different types of data transfer schedule. This field replaces "schedule" and "schedule_options" fields. ScheduleOptionsV2 cannot be used together with ScheduleOptions/Schedule. (id: ScheduleOptionsV2)
statestringOutput only. State of the most recently updated transfer run.
updateTimestring (google-datetime)Output only. Data transfer modification time. Ignored by server on input.
userIdstring (int64)Deprecated. Unique ID of the user on whose behalf transfer is done.

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
projects_locations_transfer_configs_getselectprojectsId, locationsId, transferConfigsIdReturns information about a data transfer config.
projects_transfer_configs_getselectprojectsId, transferConfigsIdReturns information about a data transfer config.
projects_locations_transfer_configs_listselectprojectsId, locationsIddataSourceIds, pageToken, pageSizeReturns information about all transfer configs owned by a project in the specified location.
projects_transfer_configs_listselectprojectsIddataSourceIds, pageToken, pageSizeReturns information about all transfer configs owned by a project in the specified location.
projects_locations_transfer_configs_createinsertprojectsId, locationsIdauthorizationCode, versionInfo, serviceAccountNameCreates a new data transfer configuration.
projects_transfer_configs_createinsertprojectsIdauthorizationCode, versionInfo, serviceAccountNameCreates a new data transfer configuration.
projects_locations_transfer_configs_patchupdateprojectsId, locationsId, transferConfigsIdauthorizationCode, updateMask, versionInfo, serviceAccountNameUpdates a data transfer configuration. All fields must be set, even if they are not updated.
projects_transfer_configs_patchupdateprojectsId, transferConfigsIdauthorizationCode, updateMask, versionInfo, serviceAccountNameUpdates a data transfer configuration. All fields must be set, even if they are not updated.
projects_locations_transfer_configs_deletedeleteprojectsId, locationsId, transferConfigsIdDeletes a data transfer configuration, including any associated transfer runs and logs.
projects_transfer_configs_deletedeleteprojectsId, transferConfigsIdDeletes a data transfer configuration, including any associated transfer runs and logs.
projects_transfer_configs_schedule_runsexecprojectsId, transferConfigsIdCreates transfer runs for a time range [start_time, end_time]. For each date - or whatever granularity the data source supports - in the range, one transfer run is created. Note that runs are created per UTC time in the time range. DEPRECATED: use StartManualTransferRuns instead.
projects_transfer_configs_start_manual_runsexecprojectsId, transferConfigsIdManually initiates transfer runs. You can schedule these runs in two ways: 1. For a specific point in time using the 'requested_run_time' parameter. 2. For a period between 'start_time' (inclusive) and 'end_time' (exclusive). If scheduling a single run, it is set to execute immediately (schedule_time equals the current time). When scheduling multiple runs within a time range, the first run starts now, and subsequent runs are delayed by 15 seconds each.
projects_locations_transfer_configs_schedule_runsexecprojectsId, locationsId, transferConfigsIdCreates transfer runs for a time range [start_time, end_time]. For each date - or whatever granularity the data source supports - in the range, one transfer run is created. Note that runs are created per UTC time in the time range. DEPRECATED: use StartManualTransferRuns instead.
projects_locations_transfer_configs_start_manual_runsexecprojectsId, locationsId, transferConfigsIdManually initiates transfer runs. You can schedule these runs in two ways: 1. For a specific point in time using the 'requested_run_time' parameter. 2. For a period between 'start_time' (inclusive) and 'end_time' (exclusive). If scheduling a single run, it is set to execute immediately (schedule_time equals the current time). When scheduling multiple runs within a time range, the first run starts now, and subsequent runs are delayed by 15 seconds each.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
locationsIdstring
projectsIdstring
transferConfigsIdstring
authorizationCodestring
dataSourceIdsstring
pageSizeinteger (int32)
pageTokenstring
serviceAccountNamestring
updateMaskstring (google-fieldmask)
versionInfostring

SELECT examples​

Returns information about a data transfer config.

SELECT
name,
dataRefreshWindowDays,
dataSourceId,
datasetRegion,
destinationDatasetId,
disabled,
displayName,
emailPreferences,
encryptionConfiguration,
error,
nextRunTime,
notificationPubsubTopic,
ownerInfo,
params,
schedule,
scheduleOptions,
scheduleOptionsV2,
state,
updateTime,
userId
FROM google.bigquerydatatransfer.transfer_configs
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND transferConfigsId = '{{ transferConfigsId }}' -- required;

INSERT examples​

Creates a new data transfer configuration.

INSERT INTO google.bigquerydatatransfer.transfer_configs (
data__name,
data__destinationDatasetId,
data__displayName,
data__dataSourceId,
data__params,
data__schedule,
data__scheduleOptions,
data__scheduleOptionsV2,
data__dataRefreshWindowDays,
data__disabled,
data__userId,
data__notificationPubsubTopic,
data__emailPreferences,
data__encryptionConfiguration,
projectsId,
locationsId,
authorizationCode,
versionInfo,
serviceAccountName
)
SELECT 
'{{ name }}',
'{{ destinationDatasetId }}',
'{{ displayName }}',
'{{ dataSourceId }}',
'{{ params }}',
'{{ schedule }}',
'{{ scheduleOptions }}',
'{{ scheduleOptionsV2 }}',
{{ dataRefreshWindowDays }},
{{ disabled }},
'{{ userId }}',
'{{ notificationPubsubTopic }}',
'{{ emailPreferences }}',
'{{ encryptionConfiguration }}',
'{{ projectsId }}',
'{{ locationsId }}',
'{{ authorizationCode }}',
'{{ versionInfo }}',
'{{ serviceAccountName }}'
RETURNING
name,
dataRefreshWindowDays,
dataSourceId,
datasetRegion,
destinationDatasetId,
disabled,
displayName,
emailPreferences,
encryptionConfiguration,
error,
nextRunTime,
notificationPubsubTopic,
ownerInfo,
params,
schedule,
scheduleOptions,
scheduleOptionsV2,
state,
updateTime,
userId
;

UPDATE examples​

Updates a data transfer configuration. All fields must be set, even if they are not updated.

UPDATE google.bigquerydatatransfer.transfer_configs
SET 
data__name = '{{ name }}',
data__destinationDatasetId = '{{ destinationDatasetId }}',
data__displayName = '{{ displayName }}',
data__dataSourceId = '{{ dataSourceId }}',
data__params = '{{ params }}',
data__schedule = '{{ schedule }}',
data__scheduleOptions = '{{ scheduleOptions }}',
data__scheduleOptionsV2 = '{{ scheduleOptionsV2 }}',
data__dataRefreshWindowDays = {{ dataRefreshWindowDays }},
data__disabled = {{ disabled }},
data__userId = '{{ userId }}',
data__notificationPubsubTopic = '{{ notificationPubsubTopic }}',
data__emailPreferences = '{{ emailPreferences }}',
data__encryptionConfiguration = '{{ encryptionConfiguration }}'
WHERE 
projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND transferConfigsId = '{{ transferConfigsId }}' --required
AND authorizationCode = '{{ authorizationCode}}'
AND updateMask = '{{ updateMask}}'
AND versionInfo = '{{ versionInfo}}'
AND serviceAccountName = '{{ serviceAccountName}}'
RETURNING
name,
dataRefreshWindowDays,
dataSourceId,
datasetRegion,
destinationDatasetId,
disabled,
displayName,
emailPreferences,
encryptionConfiguration,
error,
nextRunTime,
notificationPubsubTopic,
ownerInfo,
params,
schedule,
scheduleOptions,
scheduleOptionsV2,
state,
updateTime,
userId;

DELETE examples​

Deletes a data transfer configuration, including any associated transfer runs and logs.

DELETE FROM google.bigquerydatatransfer.transfer_configs
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND transferConfigsId = '{{ transferConfigsId }}' --required;

Lifecycle Methods​

Creates transfer runs for a time range [start_time, end_time]. For each date - or whatever granularity the data source supports - in the range, one transfer run is created. Note that runs are created per UTC time in the time range. DEPRECATED: use StartManualTransferRuns instead.

EXEC google.bigquerydatatransfer.transfer_configs.projects_transfer_configs_schedule_runs 
@projectsId='{{ projectsId }}' --required, 
@transferConfigsId='{{ transferConfigsId }}' --required 
@@json=
'{
"startTime": "{{ startTime }}", 
"endTime": "{{ endTime }}"
}';