batches

Creates, updates, deletes, gets or lists a batches resource.

Overview

Name	`batches`
Type	Resource
Id	`google.dataproc.batches`

Fields

The following fields are returned by SELECT queries:

projects_locations_batches_get
projects_locations_batches_list

Successful response

Name	Datatype	Description
`name`	`string`	Output only. The resource name of the batch.
`createTime`	`string (google-datetime)`	Output only. The time when the batch was created.
`creator`	`string`	Output only. The email address of the user who created the batch.
`environmentConfig`	`object`	Optional. Environment configuration for the batch execution. (id: EnvironmentConfig)
`labels`	`object`	Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a batch.
`operation`	`string`	Output only. The resource name of the operation associated with this batch.
`pysparkBatch`	`object`	Optional. PySpark batch config. (id: PySparkBatch)
`runtimeConfig`	`object`	Optional. Runtime configuration for the batch execution. (id: RuntimeConfig)
`runtimeInfo`	`object`	Output only. Runtime information about batch execution. (id: RuntimeInfo)
`sparkBatch`	`object`	Optional. Spark batch config. (id: SparkBatch)
`sparkRBatch`	`object`	Optional. SparkR batch config. (id: SparkRBatch)
`sparkSqlBatch`	`object`	Optional. SparkSql batch config. (id: SparkSqlBatch)
`state`	`string`	Output only. The state of the batch.
`stateHistory`	`array`	Output only. Historical state information for the batch.
`stateMessage`	`string`	Output only. Batch state details, such as a failure description if the state is FAILED.
`stateTime`	`string (google-datetime)`	Output only. The time when the batch entered a current state.
`uuid`	`string`	Output only. A batch UUID (Unique Universal Identifier). The service generates this value when it creates the batch.

Successful response

Name	Datatype	Description
`name`	`string`	Output only. The resource name of the batch.
`createTime`	`string (google-datetime)`	Output only. The time when the batch was created.
`creator`	`string`	Output only. The email address of the user who created the batch.
`environmentConfig`	`object`	Optional. Environment configuration for the batch execution. (id: EnvironmentConfig)
`labels`	`object`	Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a batch.
`operation`	`string`	Output only. The resource name of the operation associated with this batch.
`pysparkBatch`	`object`	Optional. PySpark batch config. (id: PySparkBatch)
`runtimeConfig`	`object`	Optional. Runtime configuration for the batch execution. (id: RuntimeConfig)
`runtimeInfo`	`object`	Output only. Runtime information about batch execution. (id: RuntimeInfo)
`sparkBatch`	`object`	Optional. Spark batch config. (id: SparkBatch)
`sparkRBatch`	`object`	Optional. SparkR batch config. (id: SparkRBatch)
`sparkSqlBatch`	`object`	Optional. SparkSql batch config. (id: SparkSqlBatch)
`state`	`string`	Output only. The state of the batch.
`stateHistory`	`array`	Output only. Historical state information for the batch.
`stateMessage`	`string`	Output only. Batch state details, such as a failure description if the state is FAILED.
`stateTime`	`string (google-datetime)`	Output only. The time when the batch entered a current state.
`uuid`	`string`	Output only. A batch UUID (Unique Universal Identifier). The service generates this value when it creates the batch.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
`projects_locations_batches_get`	`select`	`projectsId`, `locationsId`, `batchesId`		Gets the batch workload resource representation.
`projects_locations_batches_list`	`select`	`projectsId`, `locationsId`	`pageSize`, `pageToken`, `filter`, `orderBy`	Lists batch workloads.
`projects_locations_batches_create`	`insert`	`projectsId`, `locationsId`	`batchId`, `requestId`	Creates a batch workload that executes asynchronously.
`projects_locations_batches_delete`	`delete`	`projectsId`, `locationsId`, `batchesId`		Deletes the batch workload resource. If the batch is not in a CANCELLED, SUCCEEDED or FAILED State, the delete operation fails and the response returns FAILED_PRECONDITION.
`projects_locations_batches_analyze`	`exec`	`projectsId`, `locationsId`, `batchesId`		Analyze a Batch for possible recommendations and insights.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
`batchesId`	`string`
`locationsId`	`string`
`projectsId`	`string`
`batchId`	`string`
`filter`	`string`
`orderBy`	`string`
`pageSize`	`integer (int32)`
`pageToken`	`string`
`requestId`	`string`

`SELECT` examples

projects_locations_batches_get
projects_locations_batches_list

Gets the batch workload resource representation.

SELECT
name,
createTime,
creator,
environmentConfig,
labels,
operation,
pysparkBatch,
runtimeConfig,
runtimeInfo,
sparkBatch,
sparkRBatch,
sparkSqlBatch,
state,
stateHistory,
stateMessage,
stateTime,
uuid
FROM google.dataproc.batches
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND batchesId = '{{ batchesId }}' -- required;

Lists batch workloads.

SELECT
name,
createTime,
creator,
environmentConfig,
labels,
operation,
pysparkBatch,
runtimeConfig,
runtimeInfo,
sparkBatch,
sparkRBatch,
sparkSqlBatch,
state,
stateHistory,
stateMessage,
stateTime,
uuid
FROM google.dataproc.batches
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND pageSize = '{{ pageSize }}'
AND pageToken = '{{ pageToken }}'
AND filter = '{{ filter }}'
AND orderBy = '{{ orderBy }}';

`INSERT` examples

projects_locations_batches_create
Manifest

Creates a batch workload that executes asynchronously.

INSERT INTO google.dataproc.batches (
data__pysparkBatch,
data__sparkBatch,
data__sparkRBatch,
data__sparkSqlBatch,
data__labels,
data__runtimeConfig,
data__environmentConfig,
projectsId,
locationsId,
batchId,
requestId
)
SELECT 
'{{ pysparkBatch }}',
'{{ sparkBatch }}',
'{{ sparkRBatch }}',
'{{ sparkSqlBatch }}',
'{{ labels }}',
'{{ runtimeConfig }}',
'{{ environmentConfig }}',
'{{ projectsId }}',
'{{ locationsId }}',
'{{ batchId }}',
'{{ requestId }}'
RETURNING
name,
done,
error,
metadata,
response
;

# Description fields are for documentation purposes
- name: batches
  props:
    - name: projectsId
      value: string
      description: Required parameter for the batches resource.
    - name: locationsId
      value: string
      description: Required parameter for the batches resource.
    - name: pysparkBatch
      value: object
      description: >
        Optional. PySpark batch config.
        
    - name: sparkBatch
      value: object
      description: >
        Optional. Spark batch config.
        
    - name: sparkRBatch
      value: object
      description: >
        Optional. SparkR batch config.
        
    - name: sparkSqlBatch
      value: object
      description: >
        Optional. SparkSql batch config.
        
    - name: labels
      value: object
      description: >
        Optional. The labels to associate with this batch. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a batch.
        
    - name: runtimeConfig
      value: object
      description: >
        Optional. Runtime configuration for the batch execution.
        
    - name: environmentConfig
      value: object
      description: >
        Optional. Environment configuration for the batch execution.
        
    - name: batchId
      value: string
    - name: requestId
      value: string

`DELETE` examples

projects_locations_batches_delete

Deletes the batch workload resource. If the batch is not in a CANCELLED, SUCCEEDED or FAILED State, the delete operation fails and the response returns FAILED_PRECONDITION.

DELETE FROM google.dataproc.batches
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND batchesId = '{{ batchesId }}' --required;

Lifecycle Methods

projects_locations_batches_analyze

Analyze a Batch for possible recommendations and insights.

EXEC google.dataproc.batches.projects_locations_batches_analyze 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@batchesId='{{ batchesId }}' --required 
@@json=
'{
"requestId": "{{ requestId }}", 
"requestorId": "{{ requestorId }}"
}';

Overview​

Fields​

Methods​

Parameters​

SELECT examples​

INSERT examples​

DELETE examples​

Lifecycle Methods​