templates

Creates, updates, deletes, gets or lists a templates resource.

Overview

Name	templates
Type	Resource
Id	google.dataflow.templates

Fields

The following fields are returned by SELECT queries:

projects_locations_templates_get

Name	Datatype	Description
metadata	object	The template metadata describing the template name, available parameters, etc. (id: TemplateMetadata)
runtimeMetadata	object	Describes the runtime metadata with SDKInfo and available parameters. (id: RuntimeMetadata)
status	object	The status of the get template request. Any problems with the request will be indicated in the error_details. (id: Status)
templateType	string	Template Type. (UNKNOWN, LEGACY, FLEX)

projects_templates_get

Name	Datatype	Description
metadata	object	The template metadata describing the template name, available parameters, etc. (id: TemplateMetadata)
runtimeMetadata	object	Describes the runtime metadata with SDKInfo and available parameters. (id: RuntimeMetadata)
status	object	The status of the get template request. Any problems with the request will be indicated in the error_details. (id: Status)
templateType	string	Template Type. (UNKNOWN, LEGACY, FLEX)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
projects_locations_templates_get	select	projectId, location	gcsPath, view	Get the template associated with a template. To get the template, we recommend using projects.locations.templates.get with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.get is not recommended, because only templates that are running in us-central1 are retrieved. # IAM Permissions Requires the resourcemanager.projects.get permission on the project.
projects_templates_get	select	projectId	gcsPath, location, view	Get the template associated with a template. To get the template, we recommend using projects.locations.templates.get with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.get is not recommended, because only templates that are running in us-central1 are retrieved. # IAM Permissions Requires the resourcemanager.projects.get permission on the project.
projects_locations_templates_create	insert	projectId, location		Creates a Cloud Dataflow job from a template. Do not enter confidential information when you supply string values using the API. To create a job, we recommend using projects.locations.templates.create with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.create is not recommended, because your job will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get
projects_templates_create	insert	projectId		Creates a Cloud Dataflow job from a template. Do not enter confidential information when you supply string values using the API. To create a job, we recommend using projects.locations.templates.create with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.create is not recommended, because your job will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get
projects_locations_templates_launch	exec	projectId, location	dynamicTemplate.gcsPath, dynamicTemplate.stagingLocation, gcsPath, validateOnly	Launches a template. To launch a template, we recommend using projects.locations.templates.launch with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.launch is not recommended, because jobs launched from the template will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get
projects_templates_launch	exec	projectId	dynamicTemplate.gcsPath, dynamicTemplate.stagingLocation, gcsPath, location, validateOnly	Launches a template. To launch a template, we recommend using projects.locations.templates.launch with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.launch is not recommended, because jobs launched from the template will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
location	string
projectId	string
dynamicTemplate.gcsPath	string
dynamicTemplate.stagingLocation	string
gcsPath	string
location	string
validateOnly	boolean
view	string

SELECT examples

projects_locations_templates_get

Get the template associated with a template. To get the template, we recommend using projects.locations.templates.get with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.get is not recommended, because only templates that are running in us-central1 are retrieved. # IAM Permissions Requires the resourcemanager.projects.get permission on the project.

SELECT
metadata,
runtimeMetadata,
status,
templateType
FROM google.dataflow.templates
WHERE projectId = '{{ projectId }}' -- required
AND location = '{{ location }}' -- required
AND gcsPath = '{{ gcsPath }}'
AND view = '{{ view }}'
;

projects_templates_get

Get the template associated with a template. To get the template, we recommend using projects.locations.templates.get with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.get is not recommended, because only templates that are running in us-central1 are retrieved. # IAM Permissions Requires the resourcemanager.projects.get permission on the project.

SELECT
metadata,
runtimeMetadata,
status,
templateType
FROM google.dataflow.templates
WHERE projectId = '{{ projectId }}' -- required
AND gcsPath = '{{ gcsPath }}'
AND location = '{{ location }}'
AND view = '{{ view }}'
;

INSERT examples

projects_locations_templates_create

Creates a Cloud Dataflow job from a template. Do not enter confidential information when you supply string values using the API. To create a job, we recommend using projects.locations.templates.create with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.create is not recommended, because your job will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get

INSERT INTO google.dataflow.templates (
data__environment,
data__gcsPath,
data__jobName,
data__location,
data__parameters,
projectId,
location
)
SELECT 
'{{ environment }}',
'{{ gcsPath }}',
'{{ jobName }}',
'{{ location }}',
'{{ parameters }}',
'{{ projectId }}',
'{{ location }}'
RETURNING
id,
name,
clientRequestId,
createTime,
createdFromSnapshotId,
currentState,
currentStateTime,
environment,
executionInfo,
jobMetadata,
labels,
location,
pausable,
pipelineDescription,
projectId,
replaceJobId,
replacedByJobId,
requestedState,
runtimeUpdatableParams,
satisfiesPzi,
satisfiesPzs,
serviceResources,
stageStates,
startTime,
steps,
stepsLocation,
tempFiles,
transformNameMapping,
type
;

projects_templates_create

Creates a Cloud Dataflow job from a template. Do not enter confidential information when you supply string values using the API. To create a job, we recommend using projects.locations.templates.create with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.create is not recommended, because your job will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get

INSERT INTO google.dataflow.templates (
data__environment,
data__gcsPath,
data__jobName,
data__location,
data__parameters,
projectId
)
SELECT 
'{{ environment }}',
'{{ gcsPath }}',
'{{ jobName }}',
'{{ location }}',
'{{ parameters }}',
'{{ projectId }}'
RETURNING
id,
name,
clientRequestId,
createTime,
createdFromSnapshotId,
currentState,
currentStateTime,
environment,
executionInfo,
jobMetadata,
labels,
location,
pausable,
pipelineDescription,
projectId,
replaceJobId,
replacedByJobId,
requestedState,
runtimeUpdatableParams,
satisfiesPzi,
satisfiesPzs,
serviceResources,
stageStates,
startTime,
steps,
stepsLocation,
tempFiles,
transformNameMapping,
type
;

Manifest

# Description fields are for documentation purposes
- name: templates
props:
  - name: projectId
    value: "{{ projectId }}"
    description: Required parameter for the templates resource.
  - name: location
    value: "{{ location }}"
    description: Required parameter for the templates resource.
  - name: environment
    description: |
      The runtime environment for the job.
    value:
      additionalExperiments:
        - "{{ additionalExperiments }}"
      additionalPipelineOptions:
        - "{{ additionalPipelineOptions }}"
      additionalUserLabels: "{{ additionalUserLabels }}"
      bypassTempDirValidation: {{ bypassTempDirValidation }}
      diskSizeGb: {{ diskSizeGb }}
      enableStreamingEngine: {{ enableStreamingEngine }}
      ipConfiguration: "{{ ipConfiguration }}"
      kmsKeyName: "{{ kmsKeyName }}"
      machineType: "{{ machineType }}"
      maxWorkers: {{ maxWorkers }}
      network: "{{ network }}"
      numWorkers: {{ numWorkers }}
      serviceAccountEmail: "{{ serviceAccountEmail }}"
      streamingMode: "{{ streamingMode }}"
      subnetwork: "{{ subnetwork }}"
      tempLocation: "{{ tempLocation }}"
      workerRegion: "{{ workerRegion }}"
      workerZone: "{{ workerZone }}"
      zone: "{{ zone }}"
  - name: gcsPath
    value: "{{ gcsPath }}"
    description: |
      Required. A Cloud Storage path to the template from which to create the job. Must be a valid Cloud Storage URL, beginning with `gs://`.
  - name: jobName
    value: "{{ jobName }}"
    description: |
      Required. The job name to use for the created job.
  - name: location
    value: "{{ location }}"
    description: |
      The [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) to which to direct the request.
  - name: parameters
    value: "{{ parameters }}"
    description: |
      The runtime parameters to pass to the job.

Lifecycle Methods

projects_locations_templates_launch

Launches a template. To launch a template, we recommend using projects.locations.templates.launch with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.launch is not recommended, because jobs launched from the template will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get

EXEC google.dataflow.templates.projects_locations_templates_launch 
@projectId='{{ projectId }}' --required, 
@location='{{ location }}' --required, 
@dynamicTemplate.gcsPath='{{ dynamicTemplate.gcsPath }}', 
@dynamicTemplate.stagingLocation='{{ dynamicTemplate.stagingLocation }}', 
@gcsPath='{{ gcsPath }}', 
@validateOnly={{ validateOnly }} 
@@json=
'{
"environment": "{{ environment }}", 
"jobName": "{{ jobName }}", 
"parameters": "{{ parameters }}", 
"transformNameMapping": "{{ transformNameMapping }}", 
"update": {{ update }}
}'
;

projects_templates_launch

Launches a template. To launch a template, we recommend using projects.locations.templates.launch with a [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints). Using projects.templates.launch is not recommended, because jobs launched from the template will always start in us-central1. # IAM Permissions Requires the following IAM permission(s) on the project: - dataflow.jobs.create - resourcemanager.projects.get

EXEC google.dataflow.templates.projects_templates_launch 
@projectId='{{ projectId }}' --required, 
@dynamicTemplate.gcsPath='{{ dynamicTemplate.gcsPath }}', 
@dynamicTemplate.stagingLocation='{{ dynamicTemplate.stagingLocation }}', 
@gcsPath='{{ gcsPath }}', 
@location='{{ location }}', 
@validateOnly={{ validateOnly }} 
@@json=
'{
"environment": "{{ environment }}", 
"jobName": "{{ jobName }}", 
"parameters": "{{ parameters }}", 
"transformNameMapping": "{{ transformNameMapping }}", 
"update": {{ update }}
}'
;