batches

Creates, updates, deletes, gets or lists a batches resource.

Overview

Name	batches
Type	Resource
Id	google.dataproc.batches

Fields

The following fields are returned by SELECT queries:

projects_locations_batches_get

Name	Datatype	Description
name	string	Output only. The resource name of the batch.
createTime	string (google-datetime)	Output only. The time when the batch was created.
creator	string	Output only. The email address of the user who created the batch.
environmentConfig	object	Optional. Environment configuration for the batch execution. (id: EnvironmentConfig)
labels	object	Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a batch.
operation	string	Output only. The resource name of the operation associated with this batch.
pysparkBatch	object	Optional. PySpark batch config. (id: PySparkBatch)
pysparkNotebookBatch	object	Optional. PySpark notebook batch config. (id: PySparkNotebookBatch)
runtimeConfig	object	Optional. Runtime configuration for the batch execution. (id: RuntimeConfig)
runtimeInfo	object	Output only. Runtime information about batch execution. (id: RuntimeInfo)
sparkBatch	object	Optional. Spark batch config. (id: SparkBatch)
sparkRBatch	object	Optional. SparkR batch config. (id: SparkRBatch)
sparkSqlBatch	object	Optional. SparkSql batch config. (id: SparkSqlBatch)
state	string	Output only. The state of the batch. (STATE_UNSPECIFIED, PENDING, RUNNING, CANCELLING, CANCELLED, SUCCEEDED, FAILED)
stateHistory	array	Output only. Historical state information for the batch.
stateMessage	string	Output only. Batch state details, such as a failure description if the state is FAILED.
stateTime	string (google-datetime)	Output only. The time when the batch entered a current state.
uuid	string	Output only. A batch UUID (Unique Universal Identifier). The service generates this value when it creates the batch.

projects_locations_batches_list

Name	Datatype	Description

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
projects_locations_batches_get	select	projectsId, locationsId, batchesId		Gets the batch workload resource representation.
projects_locations_batches_list	select	projectsId, locationsId	pageSize, orderBy, pageToken, filter	Lists batch workloads.
projects_locations_batches_create	insert	projectsId, locationsId	requestId, batchId	Creates a batch workload that executes asynchronously.
projects_locations_batches_delete	delete	projectsId, locationsId, batchesId		Deletes the batch workload resource. If the batch is not in a CANCELLED, SUCCEEDED or FAILED State, the delete operation fails and the response returns FAILED_PRECONDITION.
projects_locations_batches_analyze	exec	projectsId, locationsId, batchesId		Analyze a Batch for possible recommendations and insights.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
batchesId	string
locationsId	string
projectsId	string
batchId	string
filter	string
orderBy	string
pageSize	integer (int32)
pageToken	string
requestId	string

SELECT examples

projects_locations_batches_get

Gets the batch workload resource representation.

SELECT
name,
createTime,
creator,
environmentConfig,
labels,
operation,
pysparkBatch,
pysparkNotebookBatch,
runtimeConfig,
runtimeInfo,
sparkBatch,
sparkRBatch,
sparkSqlBatch,
state,
stateHistory,
stateMessage,
stateTime,
uuid
FROM google.dataproc.batches
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND batchesId = '{{ batchesId }}' -- required
;

projects_locations_batches_list

Lists batch workloads.

SELECT
*
FROM google.dataproc.batches
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND pageSize = '{{ pageSize }}'
AND orderBy = '{{ orderBy }}'
AND pageToken = '{{ pageToken }}'
AND filter = '{{ filter }}'
;

INSERT examples

projects_locations_batches_create

Creates a batch workload that executes asynchronously.

INSERT INTO google.dataproc.batches (
data__pysparkBatch,
data__sparkRBatch,
data__environmentConfig,
data__runtimeConfig,
data__sparkBatch,
data__labels,
data__pysparkNotebookBatch,
data__sparkSqlBatch,
projectsId,
locationsId,
requestId,
batchId
)
SELECT 
'{{ pysparkBatch }}',
'{{ sparkRBatch }}',
'{{ environmentConfig }}',
'{{ runtimeConfig }}',
'{{ sparkBatch }}',
'{{ labels }}',
'{{ pysparkNotebookBatch }}',
'{{ sparkSqlBatch }}',
'{{ projectsId }}',
'{{ locationsId }}',
'{{ requestId }}',
'{{ batchId }}'
RETURNING
name,
done,
error,
metadata,
response
;

Manifest

# Description fields are for documentation purposes
- name: batches
props:
  - name: projectsId
    value: "{{ projectsId }}"
    description: Required parameter for the batches resource.
  - name: locationsId
    value: "{{ locationsId }}"
    description: Required parameter for the batches resource.
  - name: pysparkBatch
    description: |
      Optional. PySpark batch config.
    value:
      mainPythonFileUri: "{{ mainPythonFileUri }}"
      pythonFileUris:
        - "{{ pythonFileUris }}"
      jarFileUris:
        - "{{ jarFileUris }}"
      fileUris:
        - "{{ fileUris }}"
      args:
        - "{{ args }}"
      archiveUris:
        - "{{ archiveUris }}"
  - name: sparkRBatch
    description: |
      Optional. SparkR batch config.
    value:
      args:
        - "{{ args }}"
      archiveUris:
        - "{{ archiveUris }}"
      mainRFileUri: "{{ mainRFileUri }}"
      fileUris:
        - "{{ fileUris }}"
  - name: environmentConfig
    description: |
      Optional. Environment configuration for the batch execution.
    value:
      peripheralsConfig:
        metastoreService: "{{ metastoreService }}"
        sparkHistoryServerConfig:
          dataprocCluster: "{{ dataprocCluster }}"
      executionConfig:
        idleTtl: "{{ idleTtl }}"
        authenticationConfig:
          userWorkloadAuthenticationType: "{{ userWorkloadAuthenticationType }}"
        subnetworkUri: "{{ subnetworkUri }}"
        kmsKey: "{{ kmsKey }}"
        networkTags:
          - "{{ networkTags }}"
        ttl: "{{ ttl }}"
        serviceAccount: "{{ serviceAccount }}"
        networkUri: "{{ networkUri }}"
        resourceManagerTags: "{{ resourceManagerTags }}"
        stagingBucket: "{{ stagingBucket }}"
  - name: runtimeConfig
    description: |
      Optional. Runtime configuration for the batch execution.
    value:
      properties: "{{ properties }}"
      version: "{{ version }}"
      repositoryConfig:
        pypiRepositoryConfig:
          pypiRepository: "{{ pypiRepository }}"
      autotuningConfig:
        scenarios:
          - "{{ scenarios }}"
      containerImage: "{{ containerImage }}"
      cohort: "{{ cohort }}"
  - name: sparkBatch
    description: |
      Optional. Spark batch config.
    value:
      fileUris:
        - "{{ fileUris }}"
      args:
        - "{{ args }}"
      archiveUris:
        - "{{ archiveUris }}"
      jarFileUris:
        - "{{ jarFileUris }}"
      mainJarFileUri: "{{ mainJarFileUri }}"
      mainClass: "{{ mainClass }}"
  - name: labels
    value: "{{ labels }}"
    description: |
      Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a batch.
  - name: pysparkNotebookBatch
    description: |
      Optional. PySpark notebook batch config.
    value:
      params: "{{ params }}"
      notebookFileUri: "{{ notebookFileUri }}"
      pythonFileUris:
        - "{{ pythonFileUris }}"
      jarFileUris:
        - "{{ jarFileUris }}"
      archiveUris:
        - "{{ archiveUris }}"
      fileUris:
        - "{{ fileUris }}"
  - name: sparkSqlBatch
    description: |
      Optional. SparkSql batch config.
    value:
      queryVariables: "{{ queryVariables }}"
      jarFileUris:
        - "{{ jarFileUris }}"
      queryFileUri: "{{ queryFileUri }}"
  - name: requestId
    value: "{{ requestId }}"
  - name: batchId
    value: "{{ batchId }}"

DELETE examples

projects_locations_batches_delete

Deletes the batch workload resource. If the batch is not in a CANCELLED, SUCCEEDED or FAILED State, the delete operation fails and the response returns FAILED_PRECONDITION.

DELETE FROM google.dataproc.batches
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND batchesId = '{{ batchesId }}' --required
;

Lifecycle Methods

projects_locations_batches_analyze

Analyze a Batch for possible recommendations and insights.

EXEC google.dataproc.batches.projects_locations_batches_analyze 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@batchesId='{{ batchesId }}' --required 
@@json=
'{
"requestId": "{{ requestId }}", 
"requestorId": "{{ requestorId }}"
}'
;