Skip to main content

workflow_templates

Creates, updates, deletes, gets or lists a workflow_templates resource.

Overview

Nameworkflow_templates
TypeResource
Idgoogle.dataproc.workflow_templates

Fields

The following fields are returned by SELECT queries:

Successful response

NameDatatypeDescription
idstring
namestringOutput only. The resource name of the workflow template, as described in https://cloud.google.com/apis/design/resource_names. For projects.regions.workflowTemplates, the resource name of the template has the following format: projects/{project_id}/regions/{region}/workflowTemplates/{template_id} For projects.locations.workflowTemplates, the resource name of the template has the following format: projects/{project_id}/locations/{location}/workflowTemplates/{template_id}
createTimestring (google-datetime)Output only. The time template was created.
dagTimeoutstring (google-duration)Optional. Timeout duration for the DAG of jobs, expressed in seconds (see JSON representation of duration (https://developers.google.com/protocol-buffers/docs/proto3#json)). The timeout duration must be from 10 minutes ("600s") to 24 hours ("86400s"). The timer begins when the first job is submitted. If the workflow is running at the end of the timeout period, any remaining jobs are cancelled, the workflow is ended, and if the workflow was running on a managed cluster, the cluster is deleted.
encryptionConfigobjectOptional. Encryption settings for encrypting workflow template job arguments. (id: GoogleCloudDataprocV1WorkflowTemplateEncryptionConfig)
jobsarrayRequired. The Directed Acyclic Graph of Jobs to submit.
labelsobjectOptional. The labels to associate with this template. These labels will be propagated to all jobs and clusters created by the workflow instance.Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt).Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt).No more than 32 labels can be associated with a template.
parametersarrayOptional. Template parameters whose values are substituted into the template. Values for parameters must be provided when the template is instantiated.
placementobjectRequired. WorkflowTemplate scheduling information. (id: WorkflowTemplatePlacement)
updateTimestring (google-datetime)Output only. The time template was last updated.
versioninteger (int32)Optional. Used to perform a consistent read-modify-write.This field should be left blank for a CreateWorkflowTemplate request. It is required for an UpdateWorkflowTemplate request, and must match the current server version. A typical update template flow would fetch the current template with a GetWorkflowTemplate request, which will return the current template with the version field filled in with the current server version. The user updates other fields in the template, then returns it as part of the UpdateWorkflowTemplate request.

Methods

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
projects_regions_workflow_templates_getselectprojectsId, regionsId, workflowTemplatesIdversionRetrieves the latest workflow template.Can retrieve previously instantiated template by specifying optional version parameter.
projects_locations_workflow_templates_getselectprojectsId, locationsId, workflowTemplatesIdversionRetrieves the latest workflow template.Can retrieve previously instantiated template by specifying optional version parameter.
projects_regions_workflow_templates_listselectprojectsId, regionsIdpageSize, pageTokenLists workflows that match the specified filter in the request.
projects_locations_workflow_templates_listselectprojectsId, locationsIdpageSize, pageTokenLists workflows that match the specified filter in the request.
projects_regions_workflow_templates_createinsertprojectsId, regionsIdCreates new workflow template.
projects_locations_workflow_templates_createinsertprojectsId, locationsIdCreates new workflow template.
projects_regions_workflow_templates_updatereplaceprojectsId, regionsId, workflowTemplatesIdUpdates (replaces) workflow template. The updated template must contain version that matches the current server version.
projects_locations_workflow_templates_updatereplaceprojectsId, locationsId, workflowTemplatesIdUpdates (replaces) workflow template. The updated template must contain version that matches the current server version.
projects_regions_workflow_templates_deletedeleteprojectsId, regionsId, workflowTemplatesIdversionDeletes a workflow template. It does not cancel in-progress workflows.
projects_locations_workflow_templates_deletedeleteprojectsId, locationsId, workflowTemplatesIdversionDeletes a workflow template. It does not cancel in-progress workflows.
projects_regions_workflow_templates_instantiateexecprojectsId, regionsId, workflowTemplatesIdInstantiates a template and begins execution.The returned Operation can be used to track execution of workflow by polling operations.get. The Operation will complete when entire workflow is finished.The running workflow can be aborted via operations.cancel. This will cause any inflight jobs to be cancelled and workflow-owned clusters to be deleted.The Operation.metadata will be WorkflowMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#workflowmetadata). Also see Using WorkflowMetadata (https://cloud.google.com/dataproc/docs/concepts/workflows/debugging#using_workflowmetadata).On successful completion, Operation.response will be Empty.
projects_regions_workflow_templates_instantiate_inlineexecprojectsId, regionsIdrequestIdInstantiates a template and begins execution.This method is equivalent to executing the sequence CreateWorkflowTemplate, InstantiateWorkflowTemplate, DeleteWorkflowTemplate.The returned Operation can be used to track execution of workflow by polling operations.get. The Operation will complete when entire workflow is finished.The running workflow can be aborted via operations.cancel. This will cause any inflight jobs to be cancelled and workflow-owned clusters to be deleted.The Operation.metadata will be WorkflowMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#workflowmetadata). Also see Using WorkflowMetadata (https://cloud.google.com/dataproc/docs/concepts/workflows/debugging#using_workflowmetadata).On successful completion, Operation.response will be Empty.
projects_locations_workflow_templates_instantiateexecprojectsId, locationsId, workflowTemplatesIdInstantiates a template and begins execution.The returned Operation can be used to track execution of workflow by polling operations.get. The Operation will complete when entire workflow is finished.The running workflow can be aborted via operations.cancel. This will cause any inflight jobs to be cancelled and workflow-owned clusters to be deleted.The Operation.metadata will be WorkflowMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#workflowmetadata). Also see Using WorkflowMetadata (https://cloud.google.com/dataproc/docs/concepts/workflows/debugging#using_workflowmetadata).On successful completion, Operation.response will be Empty.
projects_locations_workflow_templates_instantiate_inlineexecprojectsId, locationsIdrequestIdInstantiates a template and begins execution.This method is equivalent to executing the sequence CreateWorkflowTemplate, InstantiateWorkflowTemplate, DeleteWorkflowTemplate.The returned Operation can be used to track execution of workflow by polling operations.get. The Operation will complete when entire workflow is finished.The running workflow can be aborted via operations.cancel. This will cause any inflight jobs to be cancelled and workflow-owned clusters to be deleted.The Operation.metadata will be WorkflowMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#workflowmetadata). Also see Using WorkflowMetadata (https://cloud.google.com/dataproc/docs/concepts/workflows/debugging#using_workflowmetadata).On successful completion, Operation.response will be Empty.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
locationsIdstring
projectsIdstring
regionsIdstring
workflowTemplatesIdstring
pageSizeinteger (int32)
pageTokenstring
requestIdstring
versioninteger (int32)

SELECT examples

Retrieves the latest workflow template.Can retrieve previously instantiated template by specifying optional version parameter.

SELECT
id,
name,
createTime,
dagTimeout,
encryptionConfig,
jobs,
labels,
parameters,
placement,
updateTime,
version
FROM google.dataproc.workflow_templates
WHERE projectsId = '{{ projectsId }}' -- required
AND regionsId = '{{ regionsId }}' -- required
AND workflowTemplatesId = '{{ workflowTemplatesId }}' -- required
AND version = '{{ version }}';

INSERT examples

Creates new workflow template.

INSERT INTO google.dataproc.workflow_templates (
data__id,
data__version,
data__labels,
data__placement,
data__jobs,
data__parameters,
data__dagTimeout,
data__encryptionConfig,
projectsId,
regionsId
)
SELECT
'{{ id }}',
{{ version }},
'{{ labels }}',
'{{ placement }}',
'{{ jobs }}',
'{{ parameters }}',
'{{ dagTimeout }}',
'{{ encryptionConfig }}',
'{{ projectsId }}',
'{{ regionsId }}'
RETURNING
id,
name,
createTime,
dagTimeout,
encryptionConfig,
jobs,
labels,
parameters,
placement,
updateTime,
version
;

REPLACE examples

Updates (replaces) workflow template. The updated template must contain version that matches the current server version.

REPLACE google.dataproc.workflow_templates
SET
data__id = '{{ id }}',
data__version = {{ version }},
data__labels = '{{ labels }}',
data__placement = '{{ placement }}',
data__jobs = '{{ jobs }}',
data__parameters = '{{ parameters }}',
data__dagTimeout = '{{ dagTimeout }}',
data__encryptionConfig = '{{ encryptionConfig }}'
WHERE
projectsId = '{{ projectsId }}' --required
AND regionsId = '{{ regionsId }}' --required
AND workflowTemplatesId = '{{ workflowTemplatesId }}' --required
RETURNING
id,
name,
createTime,
dagTimeout,
encryptionConfig,
jobs,
labels,
parameters,
placement,
updateTime,
version;

DELETE examples

Deletes a workflow template. It does not cancel in-progress workflows.

DELETE FROM google.dataproc.workflow_templates
WHERE projectsId = '{{ projectsId }}' --required
AND regionsId = '{{ regionsId }}' --required
AND workflowTemplatesId = '{{ workflowTemplatesId }}' --required
AND version = '{{ version }}';

Lifecycle Methods

Instantiates a template and begins execution.The returned Operation can be used to track execution of workflow by polling operations.get. The Operation will complete when entire workflow is finished.The running workflow can be aborted via operations.cancel. This will cause any inflight jobs to be cancelled and workflow-owned clusters to be deleted.The Operation.metadata will be WorkflowMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#workflowmetadata). Also see Using WorkflowMetadata (https://cloud.google.com/dataproc/docs/concepts/workflows/debugging#using_workflowmetadata).On successful completion, Operation.response will be Empty.

EXEC google.dataproc.workflow_templates.projects_regions_workflow_templates_instantiate 
@projectsId='{{ projectsId }}' --required,
@regionsId='{{ regionsId }}' --required,
@workflowTemplatesId='{{ workflowTemplatesId }}' --required
@@json=
'{
"version": {{ version }},
"requestId": "{{ requestId }}",
"parameters": "{{ parameters }}"
}';