online_evaluators

Creates, updates, deletes, gets or lists an online_evaluators resource.

Overview

Name	online_evaluators
Type	Resource
Id	google.aiplatform.online_evaluators

Fields

The following fields are returned by SELECT queries:

get

Name	Datatype	Description
name	string	Identifier. The resource name of the OnlineEvaluator. Format: projects/{project}/locations/{location}/onlineEvaluators/{id}.
agentResource	string	Required. Immutable. The name of the agent that the OnlineEvaluator evaluates periodically. This value is used to filter the traces with a matching cloud.resource_id and link the evaluation results with relevant dashboards/UIs. This field is immutable. Once set, it cannot be changed.
cloudObservability	object	Data source for the OnlineEvaluator, based on Google Cloud Observability stack (Cloud Trace & Cloud Logging). (id: GoogleCloudAiplatformV1OnlineEvaluatorCloudObservability)
config	object	Required. Configuration for the OnlineEvaluator. (id: GoogleCloudAiplatformV1OnlineEvaluatorConfig)
createTime	string (google-datetime)	Output only. Timestamp when the OnlineEvaluator was created.
displayName	string	Optional. Human-readable name for the OnlineEvaluator. The name doesn't have to be unique. The name can consist of any UTF-8 characters. The maximum length is 63 characters. If the display name exceeds max characters, an INVALID_ARGUMENT error is returned.
metricSources	array	Required. A list of metric sources to be used for evaluating samples. At least one MetricSource must be provided. Right now, only predefined metrics and registered metrics are supported. Every registered metric must have display_name (or title) and score_range defined. Otherwise, the evaluations will fail. The maximum number of metric_sources is 25.
state	string	Output only. The state of the OnlineEvaluator. (STATE_UNSPECIFIED, ACTIVE, SUSPENDED, FAILED, WARNING)
stateDetails	array	Output only. Contains additional information about the state of the OnlineEvaluator. This is used to provide more details in the event of a failure.
updateTime	string (google-datetime)	Output only. Timestamp when the OnlineEvaluator was last updated.

list

Name	Datatype	Description
name	string	Identifier. The resource name of the OnlineEvaluator. Format: projects/{project}/locations/{location}/onlineEvaluators/{id}.
agentResource	string	Required. Immutable. The name of the agent that the OnlineEvaluator evaluates periodically. This value is used to filter the traces with a matching cloud.resource_id and link the evaluation results with relevant dashboards/UIs. This field is immutable. Once set, it cannot be changed.
cloudObservability	object	Data source for the OnlineEvaluator, based on Google Cloud Observability stack (Cloud Trace & Cloud Logging). (id: GoogleCloudAiplatformV1OnlineEvaluatorCloudObservability)
config	object	Required. Configuration for the OnlineEvaluator. (id: GoogleCloudAiplatformV1OnlineEvaluatorConfig)
createTime	string (google-datetime)	Output only. Timestamp when the OnlineEvaluator was created.
displayName	string	Optional. Human-readable name for the OnlineEvaluator. The name doesn't have to be unique. The name can consist of any UTF-8 characters. The maximum length is 63 characters. If the display name exceeds max characters, an INVALID_ARGUMENT error is returned.
metricSources	array	Required. A list of metric sources to be used for evaluating samples. At least one MetricSource must be provided. Right now, only predefined metrics and registered metrics are supported. Every registered metric must have display_name (or title) and score_range defined. Otherwise, the evaluations will fail. The maximum number of metric_sources is 25.
state	string	Output only. The state of the OnlineEvaluator. (STATE_UNSPECIFIED, ACTIVE, SUSPENDED, FAILED, WARNING)
stateDetails	array	Output only. Contains additional information about the state of the OnlineEvaluator. This is used to provide more details in the event of a failure.
updateTime	string (google-datetime)	Output only. Timestamp when the OnlineEvaluator was last updated.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	projectsId, locationsId, onlineEvaluatorsId		Gets details of an OnlineEvaluator.
list	select	projectsId, locationsId	pageSize, filter, pageToken, orderBy	Lists the OnlineEvaluators for the given project and location.
create	insert	projectsId, locationsId		Creates an OnlineEvaluator in the given project and location.
patch	update	projectsId, locationsId, onlineEvaluatorsId	updateMask	Updates the fields of an OnlineEvaluator.
delete	delete	projectsId, locationsId, onlineEvaluatorsId		Deletes an OnlineEvaluator.
suspend	exec	projectsId, locationsId, onlineEvaluatorsId		Suspends an OnlineEvaluator. When an OnlineEvaluator is suspended, it won't run any evaluations until it is activated again.
activate	exec	projectsId, locationsId, onlineEvaluatorsId		Activates an OnlineEvaluator.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
locationsId	string
onlineEvaluatorsId	string
projectsId	string
filter	string
orderBy	string
pageSize	integer (int32)
pageToken	string
updateMask	string (google-fieldmask)

SELECT examples

get

Gets details of an OnlineEvaluator.

SELECT
name,
agentResource,
cloudObservability,
config,
createTime,
displayName,
metricSources,
state,
stateDetails,
updateTime
FROM google.aiplatform.online_evaluators
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND onlineEvaluatorsId = '{{ onlineEvaluatorsId }}' -- required
;

list

Lists the OnlineEvaluators for the given project and location.

SELECT
name,
agentResource,
cloudObservability,
config,
createTime,
displayName,
metricSources,
state,
stateDetails,
updateTime
FROM google.aiplatform.online_evaluators
WHERE projectsId = '{{ projectsId }}' -- required
AND locationsId = '{{ locationsId }}' -- required
AND pageSize = '{{ pageSize }}'
AND filter = '{{ filter }}'
AND pageToken = '{{ pageToken }}'
AND orderBy = '{{ orderBy }}'
;

INSERT examples

create

Creates an OnlineEvaluator in the given project and location.

INSERT INTO google.aiplatform.online_evaluators (
data__metricSources,
data__name,
data__config,
data__cloudObservability,
data__displayName,
data__agentResource,
projectsId,
locationsId
)
SELECT 
'{{ metricSources }}',
'{{ name }}',
'{{ config }}',
'{{ cloudObservability }}',
'{{ displayName }}',
'{{ agentResource }}',
'{{ projectsId }}',
'{{ locationsId }}'
RETURNING
name,
done,
error,
metadata,
response
;

Manifest

# Description fields are for documentation purposes
- name: online_evaluators
props:
  - name: projectsId
    value: "{{ projectsId }}"
    description: Required parameter for the online_evaluators resource.
  - name: locationsId
    value: "{{ locationsId }}"
    description: Required parameter for the online_evaluators resource.
  - name: metricSources
    description: |
      Required. A list of metric sources to be used for evaluating samples. At least one MetricSource must be provided. Right now, only predefined metrics and registered metrics are supported. Every registered metric must have `display_name` (or `title`) and `score_range` defined. Otherwise, the evaluations will fail. The maximum number of `metric_sources` is 25.
    value:
      - metricResourceName: "{{ metricResourceName }}"
        metric:
          pointwiseMetricSpec:
            metricPromptTemplate: "{{ metricPromptTemplate }}"
            systemInstruction: "{{ systemInstruction }}"
            customOutputFormatConfig:
              returnRawOutput: {{ returnRawOutput }}
          bleuSpec:
            useEffectiveOrder: {{ useEffectiveOrder }}
          rougeSpec:
            useStemmer: {{ useStemmer }}
            rougeType: "{{ rougeType }}"
            splitSummaries: {{ splitSummaries }}
          exactMatchSpec: "{{ exactMatchSpec }}"
          computationBasedMetricSpec:
            type: "{{ type }}"
            parameters: "{{ parameters }}"
          predefinedMetricSpec:
            metricSpecName: "{{ metricSpecName }}"
            metricSpecParameters: "{{ metricSpecParameters }}"
          metadata:
            otherMetadata: "{{ otherMetadata }}"
            title: "{{ title }}"
            scoreRange:
              min: {{ min }}
              step: {{ step }}
              description: "{{ description }}"
              max: {{ max }}
          customCodeExecutionSpec:
            evaluationFunction: "{{ evaluationFunction }}"
          pairwiseMetricSpec:
            systemInstruction: "{{ systemInstruction }}"
            baselineResponseFieldName: "{{ baselineResponseFieldName }}"
            candidateResponseFieldName: "{{ candidateResponseFieldName }}"
            metricPromptTemplate: "{{ metricPromptTemplate }}"
            customOutputFormatConfig:
              returnRawOutput: {{ returnRawOutput }}
          aggregationMetrics:
            - "{{ aggregationMetrics }}"
          llmBasedMetricSpec:
            systemInstruction: "{{ systemInstruction }}"
            judgeAutoraterConfig:
              samplingCount: {{ samplingCount }}
              flipEnabled: {{ flipEnabled }}
              autoraterModel: "{{ autoraterModel }}"
              generationConfig:
                speechConfig: "{{ speechConfig }}"
                seed: {{ seed }}
                imageConfig: "{{ imageConfig }}"
                audioTranscriptionConfig: "{{ audioTranscriptionConfig }}"
                maxOutputTokens: {{ maxOutputTokens }}
                responseModalities: "{{ responseModalities }}"
                topP: {{ topP }}
                responseJsonSchema: "{{ responseJsonSchema }}"
                responseFormat: "{{ responseFormat }}"
                thinkingConfig: "{{ thinkingConfig }}"
                enableAffectiveDialog: {{ enableAffectiveDialog }}
                topK: {{ topK }}
                logprobs: {{ logprobs }}
                presencePenalty: {{ presencePenalty }}
                candidateCount: {{ candidateCount }}
                responseSchema: "{{ responseSchema }}"
                routingConfig: "{{ routingConfig }}"
                audioTimestamp: {{ audioTimestamp }}
                temperature: {{ temperature }}
                stopSequences: "{{ stopSequences }}"
                responseLogprobs: {{ responseLogprobs }}
                responseMimeType: "{{ responseMimeType }}"
                mediaResolution: "{{ mediaResolution }}"
                frequencyPenalty: {{ frequencyPenalty }}
            rubricGroupKey: "{{ rubricGroupKey }}"
            resultParserConfig:
              customCodeParserConfig:
                parsingFunction: "{{ parsingFunction }}"
            predefinedRubricGenerationSpec:
              metricSpecName: "{{ metricSpecName }}"
              metricSpecParameters: "{{ metricSpecParameters }}"
            metricPromptTemplate: "{{ metricPromptTemplate }}"
            rubricGenerationSpec:
              promptTemplate: "{{ promptTemplate }}"
              modelConfig:
                samplingCount: {{ samplingCount }}
                flipEnabled: {{ flipEnabled }}
                autoraterModel: "{{ autoraterModel }}"
                generationConfig: "{{ generationConfig }}"
              rubricTypeOntology:
                - "{{ rubricTypeOntology }}"
              rubricContentType: "{{ rubricContentType }}"
            additionalConfig: "{{ additionalConfig }}"
  - name: name
    value: "{{ name }}"
    description: |
      Identifier. The resource name of the OnlineEvaluator. Format: projects/{project}/locations/{location}/onlineEvaluators/{id}.
  - name: config
    description: |
      Required. Configuration for the OnlineEvaluator.
    value:
      randomSampling:
        percentage: {{ percentage }}
      maxEvaluatedSamplesPerRun: "{{ maxEvaluatedSamplesPerRun }}"
  - name: cloudObservability
    description: |
      Data source for the OnlineEvaluator, based on Google Cloud Observability stack (Cloud Trace & Cloud Logging).
    value:
      traceScope:
        filter:
          - duration:
              comparisonOperator: "{{ comparisonOperator }}"
              value: {{ value }}
            totalTokenUsage:
              comparisonOperator: "{{ comparisonOperator }}"
              value: {{ value }}
      logView: "{{ logView }}"
      openTelemetry:
        semconvVersion: "{{ semconvVersion }}"
      traceView: "{{ traceView }}"
  - name: displayName
    value: "{{ displayName }}"
    description: |
      Optional. Human-readable name for the OnlineEvaluator. The name doesn't have to be unique. The name can consist of any UTF-8 characters. The maximum length is `63` characters. If the display name exceeds max characters, an `INVALID_ARGUMENT` error is returned.
  - name: agentResource
    value: "{{ agentResource }}"
    description: |
      Required. Immutable. The name of the agent that the OnlineEvaluator evaluates periodically. This value is used to filter the traces with a matching cloud.resource_id and link the evaluation results with relevant dashboards/UIs. This field is immutable. Once set, it cannot be changed.

UPDATE examples

patch

Updates the fields of an OnlineEvaluator.

UPDATE google.aiplatform.online_evaluators
SET 
data__metricSources = '{{ metricSources }}',
data__name = '{{ name }}',
data__config = '{{ config }}',
data__cloudObservability = '{{ cloudObservability }}',
data__displayName = '{{ displayName }}',
data__agentResource = '{{ agentResource }}'
WHERE 
projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND onlineEvaluatorsId = '{{ onlineEvaluatorsId }}' --required
AND updateMask = '{{ updateMask}}'
RETURNING
name,
done,
error,
metadata,
response;

DELETE examples

delete

Deletes an OnlineEvaluator.

DELETE FROM google.aiplatform.online_evaluators
WHERE projectsId = '{{ projectsId }}' --required
AND locationsId = '{{ locationsId }}' --required
AND onlineEvaluatorsId = '{{ onlineEvaluatorsId }}' --required
;

Lifecycle Methods

suspend

Suspends an OnlineEvaluator. When an OnlineEvaluator is suspended, it won't run any evaluations until it is activated again.

EXEC google.aiplatform.online_evaluators.suspend 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@onlineEvaluatorsId='{{ onlineEvaluatorsId }}' --required
;

activate

Activates an OnlineEvaluator.

EXEC google.aiplatform.online_evaluators.activate 
@projectsId='{{ projectsId }}' --required, 
@locationsId='{{ locationsId }}' --required, 
@onlineEvaluatorsId='{{ onlineEvaluatorsId }}' --required
;