Skip to main content

pipelines

Creates, updates, deletes, gets or lists a pipelines resource.

Overview​

Namepipelines
TypeResource
Idgoogle.datapipelines.pipelines

Fields​

The following fields are returned by SELECT queries:

Successful response

NameDatatypeDescription
namestringThe pipeline name. For example: projects/PROJECT_ID/locations/LOCATION_ID/pipelines/PIPELINE_ID. * PROJECT_ID can contain letters ([A-Za-z]), numbers ([0-9]), hyphens (-), colons (:), and periods (.). For more information, see Identifying projects. * LOCATION_ID is the canonical ID for the pipeline's location. The list of available locations can be obtained by calling google.cloud.location.Locations.ListLocations. Note that the Data Pipelines service is not available in all regions. It depends on Cloud Scheduler, an App Engine application, so it's only available in App Engine regions. * PIPELINE_ID is the ID of the pipeline. Must be unique for the selected project and location.
createTimestring (google-datetime)Output only. Immutable. The timestamp when the pipeline was initially created. Set by the Data Pipelines service.
displayNamestringRequired. The display name of the pipeline. It can contain only letters ([A-Za-z]), numbers ([0-9]), hyphens (-), and underscores (_).
jobCountinteger (int32)Output only. Number of jobs.
lastUpdateTimestring (google-datetime)Output only. Immutable. The timestamp when the pipeline was last modified. Set by the Data Pipelines service.
pipelineSourcesobjectImmutable. The sources of the pipeline (for example, Dataplex). The keys and values are set by the corresponding sources during pipeline creation.
scheduleInfoobjectInternal scheduling information for a pipeline. If this information is provided, periodic jobs will be created per the schedule. If not, users are responsible for creating jobs externally. (id: GoogleCloudDatapipelinesV1ScheduleSpec)
schedulerServiceAccountEmailstringOptional. A service account email to be used with the Cloud Scheduler job. If not specified, the default compute engine service account will be used.
statestringRequired. The state of the pipeline. When the pipeline is created, the state is set to 'PIPELINE_STATE_ACTIVE' by default. State changes can be requested by setting the state to stopping, paused, or resuming. State cannot be changed through UpdatePipeline requests.
typestringRequired. The type of the pipeline. This field affects the scheduling of the pipeline and the type of metrics to show for the pipeline.
workloadobjectWorkload information for creating new jobs. (id: GoogleCloudDatapipelinesV1Workload)

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
getselectprojectsId, locationsId, pipelinesIdLooks up a single pipeline. Returns a "NOT_FOUND" error if no such pipeline exists. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.
listselectprojectsId, locationsIdfilter, pageSize, pageTokenLists pipelines. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.
createinsertprojectsId, locationsIdCreates a pipeline. For a batch pipeline, you can pass scheduler information. Data Pipelines uses the scheduler information to create an internal scheduler that runs jobs periodically. If the internal scheduler is not configured, you can use RunPipeline to run jobs.
patchupdateprojectsId, locationsId, pipelinesIdupdateMaskUpdates a pipeline. If successful, the updated Pipeline is returned. Returns NOT_FOUND if the pipeline doesn't exist. If UpdatePipeline does not return successfully, you can retry the UpdatePipeline request until you receive a successful response.
deletedeleteprojectsId, locationsId, pipelinesIdDeletes a pipeline. If a scheduler job is attached to the pipeline, it will be deleted.
stopexecprojectsId, locationsId, pipelinesIdFreezes pipeline execution permanently. If there's a corresponding scheduler entry, it's deleted, and the pipeline state is changed to "ARCHIVED". However, pipeline metadata is retained.
runexecprojectsId, locationsId, pipelinesIdCreates a job for the specified pipeline directly. You can use this method when the internal scheduler is not configured and you want to trigger the job directly or through an external system. Returns a "NOT_FOUND" error if the pipeline doesn't exist. Returns a "FORBIDDEN" error if the user doesn't have permission to access the pipeline or run jobs for the pipeline.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
locationsIdstring
pipelinesIdstring
projectsIdstring
filterstring
pageSizeinteger (int32)
pageTokenstring
updateMaskstring (google-fieldmask)

SELECT examples​

Looks up a single pipeline. Returns a "NOT_FOUND" error if no such pipeline exists. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.

SELECT
name,
createTime,
displayName,
jobCount,
lastUpdateTime,
pipelineSources,
scheduleInfo,
schedulerServiceAccountEmail,
state,
type,
workload
FROM google.datapipelines.pipelines
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND pipelinesId = '{{ pipelinesId }}' -- required;

INSERT examples​

Creates a pipeline. For a batch pipeline, you can pass scheduler information. Data Pipelines uses the scheduler information to create an internal scheduler that runs jobs periodically. If the internal scheduler is not configured, you can use RunPipeline to run jobs.

INSERT INTO google.datapipelines.pipelines (
data__name,
data__displayName,
data__type,
data__state,
data__workload,
data__scheduleInfo,
data__schedulerServiceAccountEmail,
data__pipelineSources,
projectsId,
locationsId
)
SELECT 
'{{ name }}',
'{{ displayName }}',
'{{ type }}',
'{{ state }}',
'{{ workload }}',
'{{ scheduleInfo }}',
'{{ schedulerServiceAccountEmail }}',
'{{ pipelineSources }}',
'{{ projectsId }}',
'{{ locationsId }}'
RETURNING
name,
createTime,
displayName,
jobCount,
lastUpdateTime,
pipelineSources,
scheduleInfo,
schedulerServiceAccountEmail,
state,
type,
workload
;

UPDATE examples​

Updates a pipeline. If successful, the updated Pipeline is returned. Returns NOT_FOUND if the pipeline doesn't exist. If UpdatePipeline does not return successfully, you can retry the UpdatePipeline request until you receive a successful response.

UPDATE google.datapipelines.pipelines
SET 
data__name = '{{ name }}',
data__displayName = '{{ displayName }}',
data__type = '{{ type }}',
data__state = '{{ state }}',
data__workload = '{{ workload }}',
data__scheduleInfo = '{{ scheduleInfo }}',
data__schedulerServiceAccountEmail = '{{ schedulerServiceAccountEmail }}',
data__pipelineSources = '{{ pipelineSources }}'
WHERE 
projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND pipelinesId = '{{ pipelinesId }}' --required
AND updateMask = '{{ updateMask}}'
RETURNING
name,
createTime,
displayName,
jobCount,
lastUpdateTime,
pipelineSources,
scheduleInfo,
schedulerServiceAccountEmail,
state,
type,
workload;

DELETE examples​

Deletes a pipeline. If a scheduler job is attached to the pipeline, it will be deleted.

DELETE FROM google.datapipelines.pipelines
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND pipelinesId = '{{ pipelinesId }}' --required;

Lifecycle Methods​

Freezes pipeline execution permanently. If there's a corresponding scheduler entry, it's deleted, and the pipeline state is changed to "ARCHIVED". However, pipeline metadata is retained.

EXEC google.datapipelines.pipelines.stop 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@pipelinesId='{{ pipelinesId }}' --required;