training_pipelines

Creates, updates, deletes, gets or lists a training_pipelines resource.

Overview

Name	training_pipelines
Type	Resource
Id	google.aiplatform.training_pipelines

Fields

The following fields are returned by SELECT queries:

get

Name	Datatype	Description
name	string	Output only. Resource name of the TrainingPipeline.
createTime	string (google-datetime)	Output only. Time when the TrainingPipeline was created.
displayName	string	Required. The user-defined name of this TrainingPipeline.
encryptionSpec	object	Customer-managed encryption key spec for a TrainingPipeline. If set, this TrainingPipeline will be secured by this key. Note: Model trained by this TrainingPipeline is also secured by this key if model_to_upload is not set separately. (id: GoogleCloudAiplatformV1EncryptionSpec)
endTime	string (google-datetime)	Output only. Time when the TrainingPipeline entered any of the following states: PIPELINE_STATE_SUCCEEDED, PIPELINE_STATE_FAILED, PIPELINE_STATE_CANCELLED.
error	object	The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC. Each Status message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the API Design Guide. (id: GoogleRpcStatus)
inputDataConfig	object	Specifies Vertex AI owned input data that may be used for training the Model. The TrainingPipeline's training_task_definition should make clear whether this config is used and if there are any special requirements on how it should be filled. If nothing about this config is mentioned in the training_task_definition, then it should be assumed that the TrainingPipeline does not depend on this configuration. (id: GoogleCloudAiplatformV1InputDataConfig)
labels	object	The labels with user-defined metadata to organize TrainingPipelines. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
modelId	string	Optional. The ID to use for the uploaded Model, which will become the final component of the model resource name. This value may be up to 63 characters, and valid characters are [a-z0-9_-]. The first character cannot be a number or hyphen.
modelToUpload	object	Describes the Model that may be uploaded (via ModelService.UploadModel) by this TrainingPipeline. The TrainingPipeline's training_task_definition should make clear whether this Model description should be populated, and if there are any special requirements regarding how it should be filled. If nothing is mentioned in the training_task_definition, then it should be assumed that this field should not be filled and the training task either uploads the Model without a need of this information, or that training task does not support uploading a Model as part of the pipeline. When the Pipeline's state becomes PIPELINE_STATE_SUCCEEDED and the trained Model had been uploaded into Vertex AI, then the model_to_upload's resource name is populated. The Model is always uploaded into the Project and Location in which this pipeline is. (id: GoogleCloudAiplatformV1Model)
parentModel	string	Optional. When specify this field, the model_to_upload will not be uploaded as a new model, instead, it will become a new version of this parent_model.
startTime	string (google-datetime)	Output only. Time when the TrainingPipeline for the first time entered the PIPELINE_STATE_RUNNING state.
state	string	Output only. The detailed state of the pipeline. (PIPELINE_STATE_UNSPECIFIED, PIPELINE_STATE_QUEUED, PIPELINE_STATE_PENDING, PIPELINE_STATE_RUNNING, PIPELINE_STATE_SUCCEEDED, PIPELINE_STATE_FAILED, PIPELINE_STATE_CANCELLING, PIPELINE_STATE_CANCELLED, PIPELINE_STATE_PAUSED)
trainingTaskDefinition	string	Required. A Google Cloud Storage path to the YAML file that defines the training task which is responsible for producing the model artifact, and may also include additional auxiliary work. The definition files that can be used here are found in gs://google-cloud-aiplatform/schema/trainingjob/definition/. Note: The URI given on output will be immutable and probably different, including the URI scheme, than the one given on input. The output URI will point to a location where the user only has a read access.
trainingTaskInputs	any	Required. The training task's parameter(s), as specified in the training_task_definition's inputs.
trainingTaskMetadata	any	Output only. The metadata information as specified in the training_task_definition's metadata. This metadata is an auxiliary runtime and final information about the training task. While the pipeline is running this information is populated only at a best effort basis. Only present if the pipeline's training_task_definition contains metadata object.
updateTime	string (google-datetime)	Output only. Time when the TrainingPipeline was most recently updated.

list

Name	Datatype	Description
name	string	Output only. Resource name of the TrainingPipeline.
createTime	string (google-datetime)	Output only. Time when the TrainingPipeline was created.
displayName	string	Required. The user-defined name of this TrainingPipeline.
encryptionSpec	object	Customer-managed encryption key spec for a TrainingPipeline. If set, this TrainingPipeline will be secured by this key. Note: Model trained by this TrainingPipeline is also secured by this key if model_to_upload is not set separately. (id: GoogleCloudAiplatformV1EncryptionSpec)
endTime	string (google-datetime)	Output only. Time when the TrainingPipeline entered any of the following states: PIPELINE_STATE_SUCCEEDED, PIPELINE_STATE_FAILED, PIPELINE_STATE_CANCELLED.
error	object	The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC. Each Status message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the API Design Guide. (id: GoogleRpcStatus)
inputDataConfig	object	Specifies Vertex AI owned input data that may be used for training the Model. The TrainingPipeline's training_task_definition should make clear whether this config is used and if there are any special requirements on how it should be filled. If nothing about this config is mentioned in the training_task_definition, then it should be assumed that the TrainingPipeline does not depend on this configuration. (id: GoogleCloudAiplatformV1InputDataConfig)
labels	object	The labels with user-defined metadata to organize TrainingPipelines. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
modelId	string	Optional. The ID to use for the uploaded Model, which will become the final component of the model resource name. This value may be up to 63 characters, and valid characters are [a-z0-9_-]. The first character cannot be a number or hyphen.
modelToUpload	object	Describes the Model that may be uploaded (via ModelService.UploadModel) by this TrainingPipeline. The TrainingPipeline's training_task_definition should make clear whether this Model description should be populated, and if there are any special requirements regarding how it should be filled. If nothing is mentioned in the training_task_definition, then it should be assumed that this field should not be filled and the training task either uploads the Model without a need of this information, or that training task does not support uploading a Model as part of the pipeline. When the Pipeline's state becomes PIPELINE_STATE_SUCCEEDED and the trained Model had been uploaded into Vertex AI, then the model_to_upload's resource name is populated. The Model is always uploaded into the Project and Location in which this pipeline is. (id: GoogleCloudAiplatformV1Model)
parentModel	string	Optional. When specify this field, the model_to_upload will not be uploaded as a new model, instead, it will become a new version of this parent_model.
startTime	string (google-datetime)	Output only. Time when the TrainingPipeline for the first time entered the PIPELINE_STATE_RUNNING state.
state	string	Output only. The detailed state of the pipeline. (PIPELINE_STATE_UNSPECIFIED, PIPELINE_STATE_QUEUED, PIPELINE_STATE_PENDING, PIPELINE_STATE_RUNNING, PIPELINE_STATE_SUCCEEDED, PIPELINE_STATE_FAILED, PIPELINE_STATE_CANCELLING, PIPELINE_STATE_CANCELLED, PIPELINE_STATE_PAUSED)
trainingTaskDefinition	string	Required. A Google Cloud Storage path to the YAML file that defines the training task which is responsible for producing the model artifact, and may also include additional auxiliary work. The definition files that can be used here are found in gs://google-cloud-aiplatform/schema/trainingjob/definition/. Note: The URI given on output will be immutable and probably different, including the URI scheme, than the one given on input. The output URI will point to a location where the user only has a read access.
trainingTaskInputs	any	Required. The training task's parameter(s), as specified in the training_task_definition's inputs.
trainingTaskMetadata	any	Output only. The metadata information as specified in the training_task_definition's metadata. This metadata is an auxiliary runtime and final information about the training task. While the pipeline is running this information is populated only at a best effort basis. Only present if the pipeline's training_task_definition contains metadata object.
updateTime	string (google-datetime)	Output only. Time when the TrainingPipeline was most recently updated.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	projectsId, locationsId, trainingPipelinesId		Gets a TrainingPipeline.
list	select	projectsId, locationsId	pageSize, pageToken, filter, readMask	Lists TrainingPipelines in a Location.
create	insert	projectsId, locationsId		Creates a TrainingPipeline. A created TrainingPipeline right away will be attempted to be run.
delete	delete	projectsId, locationsId, trainingPipelinesId		Deletes a TrainingPipeline.
cancel	exec	projectsId, locationsId, trainingPipelinesId		Cancels a TrainingPipeline. Starts asynchronous cancellation on the TrainingPipeline. The server makes a best effort to cancel the pipeline, but success is not guaranteed. Clients can use PipelineService.GetTrainingPipeline or other methods to check whether the cancellation succeeded or whether the pipeline completed despite cancellation. On successful cancellation, the TrainingPipeline is not deleted; instead it becomes a pipeline with a TrainingPipeline.error value with a google.rpc.Status.code of 1, corresponding to Code.CANCELLED, and TrainingPipeline.state is set to CANCELLED.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
locationsId	string
projectsId	string
trainingPipelinesId	string
filter	string
pageSize	integer (int32)
pageToken	string
readMask	string (google-fieldmask)

SELECT examples

get

Gets a TrainingPipeline.

SELECT
name,
createTime,
displayName,
encryptionSpec,
endTime,
error,
inputDataConfig,
labels,
modelId,
modelToUpload,
parentModel,
startTime,
state,
trainingTaskDefinition,
trainingTaskInputs,
trainingTaskMetadata,
updateTime
FROM google.aiplatform.training_pipelines
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND trainingPipelinesId = '{{ trainingPipelinesId }}' -- required
;

list

Lists TrainingPipelines in a Location.

SELECT
name,
createTime,
displayName,
encryptionSpec,
endTime,
error,
inputDataConfig,
labels,
modelId,
modelToUpload,
parentModel,
startTime,
state,
trainingTaskDefinition,
trainingTaskInputs,
trainingTaskMetadata,
updateTime
FROM google.aiplatform.training_pipelines
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND pageSize = '{{ pageSize }}'
AND pageToken = '{{ pageToken }}'
AND filter = '{{ filter }}'
AND readMask = '{{ readMask }}'
;

INSERT examples

create

Creates a TrainingPipeline. A created TrainingPipeline right away will be attempted to be run.

INSERT INTO google.aiplatform.training_pipelines (
data__displayName,
data__modelId,
data__inputDataConfig,
data__modelToUpload,
data__parentModel,
data__trainingTaskInputs,
data__labels,
data__trainingTaskDefinition,
data__encryptionSpec,
projectsId,
locationsId
)
SELECT 
'{{ displayName }}',
'{{ modelId }}',
'{{ inputDataConfig }}',
'{{ modelToUpload }}',
'{{ parentModel }}',
'{{ trainingTaskInputs }}',
'{{ labels }}',
'{{ trainingTaskDefinition }}',
'{{ encryptionSpec }}',
'{{ projectsId }}',
'{{ locationsId }}'
RETURNING
name,
createTime,
displayName,
encryptionSpec,
endTime,
error,
inputDataConfig,
labels,
modelId,
modelToUpload,
parentModel,
startTime,
state,
trainingTaskDefinition,
trainingTaskInputs,
trainingTaskMetadata,
updateTime
;

Manifest

# Description fields are for documentation purposes
- name: training_pipelines
props:
  - name: projectsId
    value: "{{ projectsId }}"
    description: Required parameter for the training_pipelines resource.
  - name: locationsId
    value: "{{ locationsId }}"
    description: Required parameter for the training_pipelines resource.
  - name: displayName
    value: "{{ displayName }}"
    description: |
      Required. The user-defined name of this TrainingPipeline.
  - name: modelId
    value: "{{ modelId }}"
    description: |
      Optional. The ID to use for the uploaded Model, which will become the final component of the model resource name. This value may be up to 63 characters, and valid characters are `[a-z0-9_-]`. The first character cannot be a number or hyphen.
  - name: inputDataConfig
    description: |
      Specifies Vertex AI owned input data that may be used for training the Model. The TrainingPipeline's training_task_definition should make clear whether this config is used and if there are any special requirements on how it should be filled. If nothing about this config is mentioned in the training_task_definition, then it should be assumed that the TrainingPipeline does not depend on this configuration.
    value:
      savedQueryId: "{{ savedQueryId }}"
      datasetId: "{{ datasetId }}"
      timestampSplit:
        validationFraction: {{ validationFraction }}
        key: "{{ key }}"
        trainingFraction: {{ trainingFraction }}
        testFraction: {{ testFraction }}
      fractionSplit:
        validationFraction: {{ validationFraction }}
        trainingFraction: {{ trainingFraction }}
        testFraction: {{ testFraction }}
      bigqueryDestination:
        outputUri: "{{ outputUri }}"
      gcsDestination:
        outputUriPrefix: "{{ outputUriPrefix }}"
      annotationsFilter: "{{ annotationsFilter }}"
      stratifiedSplit:
        trainingFraction: {{ trainingFraction }}
        testFraction: {{ testFraction }}
        validationFraction: {{ validationFraction }}
        key: "{{ key }}"
      predefinedSplit:
        key: "{{ key }}"
      annotationSchemaUri: "{{ annotationSchemaUri }}"
      persistMlUseAssignment: {{ persistMlUseAssignment }}
      filterSplit:
        trainingFilter: "{{ trainingFilter }}"
        testFilter: "{{ testFilter }}"
        validationFilter: "{{ validationFilter }}"
  - name: modelToUpload
    description: |
      Describes the Model that may be uploaded (via ModelService.UploadModel) by this TrainingPipeline. The TrainingPipeline's training_task_definition should make clear whether this Model description should be populated, and if there are any special requirements regarding how it should be filled. If nothing is mentioned in the training_task_definition, then it should be assumed that this field should not be filled and the training task either uploads the Model without a need of this information, or that training task does not support uploading a Model as part of the pipeline. When the Pipeline's state becomes `PIPELINE_STATE_SUCCEEDED` and the trained Model had been uploaded into Vertex AI, then the model_to_upload's resource name is populated. The Model is always uploaded into the Project and Location in which this pipeline is.
    value:
      name: "{{ name }}"
      encryptionSpec:
        kmsKeyName: "{{ kmsKeyName }}"
      predictSchemata:
        parametersSchemaUri: "{{ parametersSchemaUri }}"
        predictionSchemaUri: "{{ predictionSchemaUri }}"
        instanceSchemaUri: "{{ instanceSchemaUri }}"
      containerSpec:
        grpcPorts:
          - containerPort: {{ containerPort }}
        predictRoute: "{{ predictRoute }}"
        livenessProbe:
          failureThreshold: {{ failureThreshold }}
          periodSeconds: {{ periodSeconds }}
          successThreshold: {{ successThreshold }}
          grpc:
            service: "{{ service }}"
            port: {{ port }}
          tcpSocket:
            port: {{ port }}
            host: "{{ host }}"
          exec:
            command:
              - "{{ command }}"
          initialDelaySeconds: {{ initialDelaySeconds }}
          httpGet:
            host: "{{ host }}"
            scheme: "{{ scheme }}"
            path: "{{ path }}"
            httpHeaders:
              - name: "{{ name }}"
                value: "{{ value }}"
            port: {{ port }}
          timeoutSeconds: {{ timeoutSeconds }}
        deploymentTimeout: "{{ deploymentTimeout }}"
        imageUri: "{{ imageUri }}"
        sharedMemorySizeMb: "{{ sharedMemorySizeMb }}"
        ports:
          - containerPort: {{ containerPort }}
        invokeRoutePrefix: "{{ invokeRoutePrefix }}"
        startupProbe:
          failureThreshold: {{ failureThreshold }}
          periodSeconds: {{ periodSeconds }}
          successThreshold: {{ successThreshold }}
          grpc:
            service: "{{ service }}"
            port: {{ port }}
          tcpSocket:
            port: {{ port }}
            host: "{{ host }}"
          exec:
            command:
              - "{{ command }}"
          initialDelaySeconds: {{ initialDelaySeconds }}
          httpGet:
            host: "{{ host }}"
            scheme: "{{ scheme }}"
            path: "{{ path }}"
            httpHeaders:
              - name: "{{ name }}"
                value: "{{ value }}"
            port: {{ port }}
          timeoutSeconds: {{ timeoutSeconds }}
        healthProbe:
          failureThreshold: {{ failureThreshold }}
          periodSeconds: {{ periodSeconds }}
          successThreshold: {{ successThreshold }}
          grpc:
            service: "{{ service }}"
            port: {{ port }}
          tcpSocket:
            port: {{ port }}
            host: "{{ host }}"
          exec:
            command:
              - "{{ command }}"
          initialDelaySeconds: {{ initialDelaySeconds }}
          httpGet:
            host: "{{ host }}"
            scheme: "{{ scheme }}"
            path: "{{ path }}"
            httpHeaders:
              - name: "{{ name }}"
                value: "{{ value }}"
            port: {{ port }}
          timeoutSeconds: {{ timeoutSeconds }}
        command:
          - "{{ command }}"
        healthRoute: "{{ healthRoute }}"
        args:
          - "{{ args }}"
        env:
          - name: "{{ name }}"
            value: "{{ value }}"
      supportedExportFormats:
        - id: "{{ id }}"
          exportableContents: "{{ exportableContents }}"
      versionId: "{{ versionId }}"
      labels: "{{ labels }}"
      baseModelSource:
        genieSource:
          baseModelUri: "{{ baseModelUri }}"
        modelGardenSource:
          publicModelName: "{{ publicModelName }}"
          skipHfModelCache: {{ skipHfModelCache }}
          versionId: "{{ versionId }}"
      supportedDeploymentResourcesTypes:
        - "{{ supportedDeploymentResourcesTypes }}"
      createTime: "{{ createTime }}"
      satisfiesPzi: {{ satisfiesPzi }}
      supportedInputStorageFormats:
        - "{{ supportedInputStorageFormats }}"
      versionUpdateTime: "{{ versionUpdateTime }}"
      defaultCheckpointId: "{{ defaultCheckpointId }}"
      metadataSchemaUri: "{{ metadataSchemaUri }}"
      description: "{{ description }}"
      versionAliases:
        - "{{ versionAliases }}"
      trainingPipeline: "{{ trainingPipeline }}"
      metadataArtifact: "{{ metadataArtifact }}"
      etag: "{{ etag }}"
      versionCreateTime: "{{ versionCreateTime }}"
      versionDescription: "{{ versionDescription }}"
      metadata: "{{ metadata }}"
      explanationSpec:
        metadata:
          featureAttributionsSchemaUri: "{{ featureAttributionsSchemaUri }}"
          outputs: "{{ outputs }}"
          inputs: "{{ inputs }}"
          latentSpaceSource: "{{ latentSpaceSource }}"
        parameters:
          integratedGradientsAttribution:
            smoothGradConfig:
              noiseSigma: {{ noiseSigma }}
              noisySampleCount: {{ noisySampleCount }}
              featureNoiseSigma: "{{ featureNoiseSigma }}"
            stepCount: {{ stepCount }}
            blurBaselineConfig:
              maxBlurSigma: {{ maxBlurSigma }}
          topK: {{ topK }}
          examples:
            neighborCount: {{ neighborCount }}
            exampleGcsSource:
              dataFormat: "{{ dataFormat }}"
              gcsSource: "{{ gcsSource }}"
            nearestNeighborSearchConfig: "{{ nearestNeighborSearchConfig }}"
            presets:
              query: "{{ query }}"
              modality: "{{ modality }}"
          outputIndices:
            - "{{ outputIndices }}"
          sampledShapleyAttribution:
            pathCount: {{ pathCount }}
          xraiAttribution:
            stepCount: {{ stepCount }}
            blurBaselineConfig:
              maxBlurSigma: {{ maxBlurSigma }}
            smoothGradConfig:
              noiseSigma: {{ noiseSigma }}
              noisySampleCount: {{ noisySampleCount }}
              featureNoiseSigma: "{{ featureNoiseSigma }}"
      pipelineJob: "{{ pipelineJob }}"
      modelSourceInfo:
        copy: {{ copy }}
        sourceType: "{{ sourceType }}"
      artifactUri: "{{ artifactUri }}"
      dataStats:
        validationDataItemsCount: "{{ validationDataItemsCount }}"
        trainingDataItemsCount: "{{ trainingDataItemsCount }}"
        trainingAnnotationsCount: "{{ trainingAnnotationsCount }}"
        testAnnotationsCount: "{{ testAnnotationsCount }}"
        validationAnnotationsCount: "{{ validationAnnotationsCount }}"
        testDataItemsCount: "{{ testDataItemsCount }}"
      originalModelInfo:
        model: "{{ model }}"
      deployedModels:
        - deployedModelId: "{{ deployedModelId }}"
          checkpointId: "{{ checkpointId }}"
          endpoint: "{{ endpoint }}"
      satisfiesPzs: {{ satisfiesPzs }}
      updateTime: "{{ updateTime }}"
      checkpoints:
        - checkpointId: "{{ checkpointId }}"
          epoch: "{{ epoch }}"
          step: "{{ step }}"
      displayName: "{{ displayName }}"
      supportedOutputStorageFormats:
        - "{{ supportedOutputStorageFormats }}"
  - name: parentModel
    value: "{{ parentModel }}"
    description: |
      Optional. When specify this field, the `model_to_upload` will not be uploaded as a new model, instead, it will become a new version of this `parent_model`.
  - name: trainingTaskInputs
    value: "{{ trainingTaskInputs }}"
    description: |
      Required. The training task's parameter(s), as specified in the training_task_definition's `inputs`.
  - name: labels
    value: "{{ labels }}"
    description: |
      The labels with user-defined metadata to organize TrainingPipelines. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
  - name: trainingTaskDefinition
    value: "{{ trainingTaskDefinition }}"
    description: |
      Required. A Google Cloud Storage path to the YAML file that defines the training task which is responsible for producing the model artifact, and may also include additional auxiliary work. The definition files that can be used here are found in gs://google-cloud-aiplatform/schema/trainingjob/definition/. Note: The URI given on output will be immutable and probably different, including the URI scheme, than the one given on input. The output URI will point to a location where the user only has a read access.
  - name: encryptionSpec
    description: |
      Customer-managed encryption key spec for a TrainingPipeline. If set, this TrainingPipeline will be secured by this key. Note: Model trained by this TrainingPipeline is also secured by this key if model_to_upload is not set separately.
    value:
      kmsKeyName: "{{ kmsKeyName }}"

DELETE examples

delete

Deletes a TrainingPipeline.

DELETE FROM google.aiplatform.training_pipelines
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND trainingPipelinesId = '{{ trainingPipelinesId }}' --required
;

Lifecycle Methods

cancel

Cancels a TrainingPipeline. Starts asynchronous cancellation on the TrainingPipeline. The server makes a best effort to cancel the pipeline, but success is not guaranteed. Clients can use PipelineService.GetTrainingPipeline or other methods to check whether the cancellation succeeded or whether the pipeline completed despite cancellation. On successful cancellation, the TrainingPipeline is not deleted; instead it becomes a pipeline with a TrainingPipeline.error value with a google.rpc.Status.code of 1, corresponding to Code.CANCELLED, and TrainingPipeline.state is set to CANCELLED.

EXEC google.aiplatform.training_pipelines.cancel 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@trainingPipelinesId='{{ trainingPipelinesId }}' --required
;