clusters

Creates, updates, deletes, gets or lists a clusters resource.

Overview

Name	clusters
Type	Resource
Id	google.dataproc.clusters

Fields

The following fields are returned by SELECT queries:

projects_regions_clusters_get

Successful response

Name	Datatype	Description
clusterName	string	Required. The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
clusterUuid	string	Output only. A cluster UUID (Unique Universal Identifier). Dataproc generates this value when it creates the cluster.
config	object	Optional. The cluster config for a cluster of Compute Engine Instances. Note that Dataproc may set default values, and values may change when clusters are updated.Exactly one of ClusterConfig or VirtualClusterConfig must be specified. (id: ClusterConfig)
labels	object	Optional. The labels to associate with this cluster. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a cluster.
metrics	object	Output only. Contains cluster daemon metrics such as HDFS and YARN stats.Beta Feature: This report is available for testing purposes only. It may be changed before final release. (id: ClusterMetrics)
projectId	string	Required. The Google Cloud Platform project ID that the cluster belongs to.
status	object	Output only. Cluster status. (id: ClusterStatus)
statusHistory	array	Output only. The previous cluster status.
virtualClusterConfig	object	Optional. The virtual cluster config is used when creating a Dataproc cluster that does not directly control the underlying compute resources, for example, when creating a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-overview). Dataproc may set default values, and values may change when clusters are updated. Exactly one of config or virtual_cluster_config must be specified. (id: VirtualClusterConfig)

projects_regions_clusters_list

Successful response

Name	Datatype	Description
clusterName	string	Required. The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
clusterUuid	string	Output only. A cluster UUID (Unique Universal Identifier). Dataproc generates this value when it creates the cluster.
config	object	Optional. The cluster config for a cluster of Compute Engine Instances. Note that Dataproc may set default values, and values may change when clusters are updated.Exactly one of ClusterConfig or VirtualClusterConfig must be specified. (id: ClusterConfig)
labels	object	Optional. The labels to associate with this cluster. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a cluster.
metrics	object	Output only. Contains cluster daemon metrics such as HDFS and YARN stats.Beta Feature: This report is available for testing purposes only. It may be changed before final release. (id: ClusterMetrics)
projectId	string	Required. The Google Cloud Platform project ID that the cluster belongs to.
status	object	Output only. Cluster status. (id: ClusterStatus)
statusHistory	array	Output only. The previous cluster status.
virtualClusterConfig	object	Optional. The virtual cluster config is used when creating a Dataproc cluster that does not directly control the underlying compute resources, for example, when creating a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-overview). Dataproc may set default values, and values may change when clusters are updated. Exactly one of config or virtual_cluster_config must be specified. (id: VirtualClusterConfig)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
projects_regions_clusters_get	select	projectId, region, clusterName		Gets the resource representation for a cluster in a project.
projects_regions_clusters_list	select	projectId, region	filter, pageSize, pageToken	Lists all regions/{region}/clusters in a project alphabetically.
projects_regions_clusters_create	insert	projectId, region	requestId, actionOnFailedPrimaryWorkers	Creates a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata).
projects_regions_clusters_patch	update	projectId, region, clusterName	gracefulDecommissionTimeout, updateMask, requestId	Updates a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata). The cluster must be in a RUNNING state or an error is returned.
projects_regions_clusters_delete	delete	projectId, region, clusterName	clusterUuid, requestId, gracefulTerminationTimeout	Deletes a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata).
projects_regions_clusters_stop	exec	projectId, region, clusterName		Stops a cluster in a project.
projects_regions_clusters_start	exec	projectId, region, clusterName		Starts a cluster in a project.
projects_regions_clusters_repair	exec	projectId, region, clusterName		Repairs a cluster.
projects_regions_clusters_diagnose	exec	projectId, region, clusterName		Gets cluster diagnostic information. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata). After the operation completes, Operation.response contains DiagnoseClusterResults (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#diagnoseclusterresults).
projects_regions_clusters_inject_credentials	exec	projectsId, regionsId, clustersId		Inject encrypted credentials into all of the VMs in a cluster.The target cluster must be a personal auth cluster assigned to the user who is issuing the RPC.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
clusterName	string
clustersId	string
projectId	string
projectsId	string
region	string
regionsId	string
actionOnFailedPrimaryWorkers	string
clusterUuid	string
filter	string
gracefulDecommissionTimeout	string (google-duration)
gracefulTerminationTimeout	string (google-duration)
pageSize	integer (int32)
pageToken	string
requestId	string
updateMask	string (google-fieldmask)

SELECT examples

projects_regions_clusters_get

Gets the resource representation for a cluster in a project.

SELECT
clusterName,
clusterUuid,
config,
labels,
metrics,
projectId,
status,
statusHistory,
virtualClusterConfig
FROM google.dataproc.clusters
WHERE projectId = '{{ projectId }}' -- required
AND region = '{{ region }}' -- required
AND clusterName = '{{ clusterName }}' -- required;

projects_regions_clusters_list

Lists all regions/{region}/clusters in a project alphabetically.

SELECT
clusterName,
clusterUuid,
config,
labels,
metrics,
projectId,
status,
statusHistory,
virtualClusterConfig
FROM google.dataproc.clusters
WHERE projectId = '{{ projectId }}' -- required
AND region = '{{ region }}' -- required
AND filter = '{{ filter }}'
AND pageSize = '{{ pageSize }}'
AND pageToken = '{{ pageToken }}';

INSERT examples

projects_regions_clusters_create

Creates a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata).

INSERT INTO google.dataproc.clusters (
data__projectId,
data__clusterName,
data__config,
data__virtualClusterConfig,
data__labels,
projectId,
region,
requestId,
actionOnFailedPrimaryWorkers
)
SELECT 
'{{ projectId }}',
'{{ clusterName }}',
'{{ config }}',
'{{ virtualClusterConfig }}',
'{{ labels }}',
'{{ projectId }}',
'{{ region }}',
'{{ requestId }}',
'{{ actionOnFailedPrimaryWorkers }}'
RETURNING
name,
done,
error,
metadata,
response
;

Manifest

# Description fields are for documentation purposes
- name: clusters
  props:
    - name: projectId
      value: string
      description: Required parameter for the clusters resource.
    - name: region
      value: string
      description: Required parameter for the clusters resource.
    - name: projectId
      value: string
      description: >
        Required. The Google Cloud Platform project ID that the cluster belongs to.
        
    - name: clusterName
      value: string
      description: >
        Required. The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
        
    - name: config
      value: object
      description: >
        Optional. The cluster config for a cluster of Compute Engine Instances. Note that Dataproc may set default values, and values may change when clusters are updated.Exactly one of ClusterConfig or VirtualClusterConfig must be specified.
        
    - name: virtualClusterConfig
      value: object
      description: >
        Optional. The virtual cluster config is used when creating a Dataproc cluster that does not directly control the underlying compute resources, for example, when creating a Dataproc-on-GKE cluster (https://cloud.google.com/dataproc/docs/guides/dpgke/dataproc-gke-overview). Dataproc may set default values, and values may change when clusters are updated. Exactly one of config or virtual_cluster_config must be specified.
        
    - name: labels
      value: object
      description: >
        Optional. The labels to associate with this cluster. Label keys must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). Label values may be empty, but, if present, must contain 1 to 63 characters, and must conform to RFC 1035 (https://www.ietf.org/rfc/rfc1035.txt). No more than 32 labels can be associated with a cluster.
        
    - name: requestId
      value: string
    - name: actionOnFailedPrimaryWorkers
      value: string

UPDATE examples

projects_regions_clusters_patch

Updates a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata). The cluster must be in a RUNNING state or an error is returned.

UPDATE google.dataproc.clusters
SET 
data__projectId = '{{ projectId }}',
data__clusterName = '{{ clusterName }}',
data__config = '{{ config }}',
data__virtualClusterConfig = '{{ virtualClusterConfig }}',
data__labels = '{{ labels }}'
WHERE 
projectId = '{{ projectId }}' --required
AND region = '{{ region }}' --required
AND clusterName = '{{ clusterName }}' --required
AND gracefulDecommissionTimeout = '{{ gracefulDecommissionTimeout}}'
AND updateMask = '{{ updateMask}}'
AND requestId = '{{ requestId}}'
RETURNING
name,
done,
error,
metadata,
response;

DELETE examples

projects_regions_clusters_delete

Deletes a cluster in a project. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata).

DELETE FROM google.dataproc.clusters
WHERE projectId = '{{ projectId }}' --required
AND region = '{{ region }}' --required
AND clusterName = '{{ clusterName }}' --required
AND clusterUuid = '{{ clusterUuid }}'
AND requestId = '{{ requestId }}'
AND gracefulTerminationTimeout = '{{ gracefulTerminationTimeout }}';

Lifecycle Methods

projects_regions_clusters_stop

Stops a cluster in a project.

EXEC google.dataproc.clusters.projects_regions_clusters_stop 
@projectId='{{ projectId }}' --required, 
@region='{{ region }}' --required, 
@clusterName='{{ clusterName }}' --required 
@@json=
'{
"clusterUuid": "{{ clusterUuid }}", 
"requestId": "{{ requestId }}"
}';

projects_regions_clusters_start

Starts a cluster in a project.

EXEC google.dataproc.clusters.projects_regions_clusters_start 
@projectId='{{ projectId }}' --required, 
@region='{{ region }}' --required, 
@clusterName='{{ clusterName }}' --required 
@@json=
'{
"clusterUuid": "{{ clusterUuid }}", 
"requestId": "{{ requestId }}"
}';

projects_regions_clusters_repair

Repairs a cluster.

EXEC google.dataproc.clusters.projects_regions_clusters_repair 
@projectId='{{ projectId }}' --required, 
@region='{{ region }}' --required, 
@clusterName='{{ clusterName }}' --required 
@@json=
'{
"clusterUuid": "{{ clusterUuid }}", 
"requestId": "{{ requestId }}", 
"nodePools": "{{ nodePools }}", 
"gracefulDecommissionTimeout": "{{ gracefulDecommissionTimeout }}", 
"parentOperationId": "{{ parentOperationId }}", 
"cluster": "{{ cluster }}"
}';

projects_regions_clusters_diagnose

Gets cluster diagnostic information. The returned Operation.metadata will be ClusterOperationMetadata (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#clusteroperationmetadata). After the operation completes, Operation.response contains DiagnoseClusterResults (https://cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#diagnoseclusterresults).

EXEC google.dataproc.clusters.projects_regions_clusters_diagnose 
@projectId='{{ projectId }}' --required, 
@region='{{ region }}' --required, 
@clusterName='{{ clusterName }}' --required 
@@json=
'{
"tarballGcsDir": "{{ tarballGcsDir }}", 
"tarballAccess": "{{ tarballAccess }}", 
"diagnosisInterval": "{{ diagnosisInterval }}", 
"job": "{{ job }}", 
"yarnApplicationId": "{{ yarnApplicationId }}", 
"jobs": "{{ jobs }}", 
"yarnApplicationIds": "{{ yarnApplicationIds }}"
}';

projects_regions_clusters_inject_credentials

Inject encrypted credentials into all of the VMs in a cluster.The target cluster must be a personal auth cluster assigned to the user who is issuing the RPC.

EXEC google.dataproc.clusters.projects_regions_clusters_inject_credentials 
@projectsId='{{ projectsId }}' --required, 
@regionsId='{{ regionsId }}' --required, 
@clustersId='{{ clustersId }}' --required 
@@json=
'{
"clusterUuid": "{{ clusterUuid }}", 
"credentialsCiphertext": "{{ credentialsCiphertext }}"
}';