NEW! Advanced PDF + OCR Interface for Document AI

Add a custom agreement metric to Label Studio

Write a custom agreement metric to assess the quality of the predictions and annotations in your Label Studio Enterprise project.

Label Studio Enterprise contains a variety of agreement metrics for your project, but if you want to evaluate annotations using a custom metric or a standard metric not available in Label Studio, you can write your own.

note

This functionality is available out-of-the-box for Label Studio Enterprise Cloud users.

For Label Studio Enterprise on-prem environments, you must configure your cloud provider to allow Label Studio to deploy and invoke serverless functions. Label Studio supports AWS Lambda and Google Cloud Functions. For more information, see On-prem deployments.

Label Studio Enterprise includes various annotation and labeling statistics and the ability to add your own. The open source Community Edition of Label Studio does not contain these calculations. If you’re using Label Studio Community Edition, see Label Studio Features to learn more.

How to write your custom agreement metric

For more information about agreement metrics, see Task agreement.

You can use the agreement metric to compare two annotations, or one annotation with one prediction. Use the input parameters annotation_1 and annotation_2 to specify the annotations to compare, or annotation and prediction to compare.

Add your code to the following function defined in Label Studio:

def agreement(annotation_1, annotation_2, per_label=False) -> float:

This function takes the following arguments:

argument format description
annotation_1 JSON object The first annotation or prediction to compare when calculating agreement. Retrieved in Label Studio JSON format.
annotation_2 JSON object The second annotation or prediction to compare when calculating agreement. Retrieved in Label Studio JSON format.
per_label boolean Whether to perform an agreement calculation for each label in the annotation, or across the entire annotation result.
return float The agreement score to assign, as a float point number between 0 and 1.

For example, given the following labeling config:

<View>
  <Image name="image" value="$image"/>
  <Choices name="choice" toName="image" showInLine="true">
    <Choice value="Positive" />
    <Choice value="Negative" />
  <Choice value="Neutral" />
  </Choices>
</View>

The following agreement metric compares two annotations for a classification task with choice options of “Positive” and “Negative”:

def agreement(annotation_1, annotation_2, per_label=False) -> float:

    # Retrieve two annotations in the Label Studio JSON format
    r1 = annotation_1["result"][0]["value"]["choices"][0]
    r2 = annotation_2["result"][0]["value"]["choices"][0]
    
    # Determine annotation agreement based on specific choice values
    if r1 == r2:
        # If annotations match and include the choice "Positive", return an agreement score of 0.99
        if r1 == "Positive":
            return 0.99
        # If annotations match and include the choice "Negative", return an agreement score of 0.7
        if r1 == "Negative":
            return 0.7
    # If annotations do not match, return an agreement score of 0
    else:
        return 0

If you set per_label=True, you can define a separate method or agreement score for each label. If you do this, you must return a separate score for each label. For example, for a classification task, you could use the following function to assign a weight and return a specific agreement score for each label used in an annotation:

def agreement(annotation_1, annotation_2, per_label=False) -> float:

    label_1 = annotation_1["result"][0]["value"]["choices"][0]
    label_2 = annotation_2["result"][0]["value"]["choices"][0]
    weight = {"Positive": 0.99, "Negative": 0.01}
    
    if label_1 == label_2:
        if per_label:
            return {label_1: weight[label_1]}
        else:
            return weight[label_1]
    else:
        if per_label:
            return {label_1: 0, label_2: 0}
        else:
            return 0

Add your custom agreement metric to Label Studio

Custom agreement metrics are added at the project level under Settings > Quality > Agreement > Custom Agreement Metric Code.

note

You must configure the labeling interface before you can add your custom agreement metric.

On-prem deployment using AWS Lambda

Field Description
Lambda Tags Add tags to the AWS Lambda function using the syntax tag_name tag_value.
Lambda Prefix Select a prefix for the AWS Lambda function.

On-prem deployment using Google Cloud Functions

Unlike with AWS Lambda, you cannot set a prefix or tags for the Google Cloud Functions function. See Adding functions to projects in Label Studio below.

On-prem deployments

If you are deploying Label Studio Enterprise on-premises, you must set up permissions for the custom agreement metric functionality.

AWS Lambda

If you have Label Studio Enterprise deployed in a private cloud (self-managed) Amazon Web Services (AWS) Elastic Compute Cluster (EC2) instance or Amazon Elastic Kubernetes Service (EKS), you must grant additional permissions so that Label Studio Enterprise can run custom agreement metrics in AWS Lambda.

To set up the permissions, do the following:

  1. Create an AWS IAM role to be used by the custom metric Lambda functions to store logs in Cloudwatch
  2. Set up permissions that grant access to AWS Lambda. How you do this depends on your deployment scenario:

You must know the AWS account ID for the AWS account that you use to manage Label Studio Enterprise to perform these steps.

Create an AWS IAM role for logging

Using your preferred method, create an AWS IAM role.

  1. Create an AWS IAM role named LSE_CustomMetricsExecuteRole. Follow the steps to create a role to delegate permissions to an AWS service in the AWS Identity and Access Management documentation for Creating a role for an AWS service (console).
  2. Whether you create the role using the console or the AWS CLI, create or attach the following IAM policy to allow the role to store logs in Cloudwatch. Replace YOUR_AWS_ACCOUNT with your AWS account ID that has access to Label Studio Enterprise.
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*:YOUR_AWS_ACCOUNT:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy"
            ],
           "Resource": [
               "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
           ]
        }
    ]
}

Configure permissions between Label Studio Enterprise and AWS Lambda

After creating an IAM role to manage logs for the custom agreement metric, set up permissions to allow Label Studio Enterprise to interact with AWS Lambda.

How you set up permissions depends on how you deployed Label Studio Enterprise in your self-managed cloud infrastructure:

Docker Compose running in EC2

If you deployed Label Studio Enterprise using Docker Compose in an AWS EC2 instance, do the following to finish setting up permissions for the custom agreement metric functionality:

  1. Follow the AWS documentation steps for Creating an IAM user in your AWS account to create an IAM user with programmatic access. This type of user is granted an access key to access AWS services.
  2. While creating the IAM user, for the Set permissions option, choose to Attach existing policies directly.
  3. Select Create policy and attach the LSE_AllowInteractLambda policy.
  4. When you finish creating the user, save the username and access key somewhere secure.
  5. In the docker-compose.yaml file that you use to deploy Label Studio Enterprise, add the following environment variables in the app and rqworkers sections:

important

Update:

  • YOUR_AWS_ACCESS_KEY_ID, YOUR_AWS_SECRET_ACCESS_KEY and YOUR_AWS_ACCOUNT with the credentials for the account created in step 1.
  • YOUR_AWS_REGION with the AWS region that your EC2 instance exists in the following:
AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY
LS_LAMBDA_REGION_CUSTOM_METRICS=YOUR_AWS_REGION
LS_LAMBDA_ROLE_CUSTOM_METRICS=arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_CustomMetricsExecuteRole

After you set up these permissions in your environment, you’re ready to write your custom agreement metric and add it to Label Studio Enterprise:

  1. Write your custom agreement metric.
  2. Add your custom agreement metric to Label Studio Enterprise.
Deployed in EKS with an OIDC provider

If you deployed Label Studio Enterprise in Amazon Elastic Kubernetes Service (EKS) with OpenID Connect (OIDC) for identity and access management (IAM), do the following to finish setting up permissions for the custom agreement metric functionality:

  1. Create an AWS IAM role named LSE_ServiceAccountApp following the steps to create a role to delegate permissions to an AWS service in the AWS Identity and Access Management documentation for Creating a role for an AWS service (console).
  2. When relevant, attach the LSE_AllowInteractLambda policy to the LSE_ServiceAccountApp role.
  3. Update your helm values.yaml file to include the following map. Replace YOUR_AWS_ACCOUNT with your AWS account ID:
    app:
  serviceAccount:
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_ServiceAccountApp
  4. Restart your Helm release.

After you set up these permissions in your environment, you’re ready to write your custom agreement metric and add it to Label Studio Enterprise:

  1. Write your custom agreement metric.
  2. Add your custom agreement metric to Label Studio Enterprise.
Deployed in EKS without an OIDC provider

If you deployed Label Studio Enterprise in Amazon Elastic Kubernetes Service (EKS) and are not using OpenID Connect (OIDC) for identity and access management (IAM), do the following to finish setting up permissions for the custom agreement metric functionality:

  1. In the AWS console UI, go to EKS > Clusters > YOUR_CLUSTER_NAME > Node Group.
  2. Select the name of YOUR_NODE_GROUP with Label Studio Enterprise deployed.
  3. On the Details page, locate and select the option for Node IAM Role ARN.
  4. Create the AWS IAM policy LSE_AllowInteractLambda.
  5. Restart your Helm release.

After you set up these permissions in your environment, you’re ready to write your custom agreement metric and add it to Label Studio Enterprise:

  1. Write your custom agreement metric.
  2. Add your custom agreement metric to Label Studio Enterprise.

Create an IAM policy to grant AWS Lambda permissions

To grant permissions to a specific user, role, or EKS node group used to manage Label Studio Enterprise access to interact with AWS Lambda, use the following IAM policy. Create an IAM policy called LSE_AllowInteractLambda and replace YOUR_AWS_ACCOUNT with your AWS account ID:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": "iam:PassRole",
         "Resource": "arn:aws:iam::YOUR_AWS_ACCOUNT:role/LSE_CustomMetricsExecuteRole"
      },
      {
         "Sid": "VisualEditor1",
         "Effect": "Allow",
         "Action": [
            "lambda:CreateFunction",
            "lambda:UpdateFunctionCode",
            "lambda:InvokeFunction",
            "lambda:GetFunction",
            "lambda:DeleteFunction",
            "lambda:TagResource",
            "lambda:ListTags"
         ],
         "Resource": [
            "arn:aws:lambda:*:YOUR_AWS_ACCOUNT:function:custom-metric-*"
         ]
      },
      {
         "Sid": "VisualEditor2",
         "Effect": "Allow",
         "Action": "lambda:ListFunctions",
         "Resource": "*"
      },
      {
         "Action": [
            "logs:CreateLogGroup",
            "logs:PutRetentionPolicy",
            "logs:TagResource",
            "logs:StartQuery",
            "logs:GetQueryResults"
         ],
         "Effect": "Allow",
         "Resource": [
            "arn:aws:logs:*:YOUR_AWS_ACCOUNT:log-group:/aws/lambda/custom-metric-*"
         ]
      }
   ]
}

Google Cloud Functions

If you have Label Studio Enterprise deployed on Google Cloud Platform (GCP), you can use Google Cloud Functions (2nd gen) to run custom agreement metrics instead of AWS Lambda.

To set up the permissions, do the following:

  1. Enable the required GCP APIs.
  2. Create a runtime service account for the Cloud Functions.
  3. Grant permissions to the Label Studio Enterprise identity.
  4. Set up authentication that grants Label Studio access to manage and invoke Cloud Functions. How you do this depends on your deployment scenario:

You must know the GCP project ID for the project that you use to manage Label Studio Enterprise to perform these steps.

Enable the required GCP APIs

Label Studio Enterprise requires the following APIs to be enabled in your GCP project:

You can enable them using the gcloud CLI:

gcloud services enable \
  cloudfunctions.googleapis.com \
  cloudbuild.googleapis.com \
  storage.googleapis.com \
  logging.googleapis.com \
  artifactregistry.googleapis.com \
  run.googleapis.com \
  --project=YOUR_GCP_PROJECT

Create a runtime service account for Cloud Functions

Create a dedicated service account that the deployed Cloud Functions will run as. This account only needs permission to write logs.

  1. Create the service account:

    gcloud iam service-accounts create lse-custom-metrics-runtime \
  --display-name="LSE Custom Metrics Runtime" \
  --project=YOUR_GCP_PROJECT

  2. Grant the Logs Writer role so the function can emit logs:

    gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="roles/logging.logWriter"

Note the full service account email (lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com). You will use it when configuring the GCP_SERVICE_ACCOUNT_CUSTOM_METRICS environment variable.

Grant permissions to the Label Studio Enterprise identity

The identity that Label Studio Enterprise runs as (a service account, Compute Engine default service account, or GKE node service account) must be able to manage Cloud Functions and their source code.

Create a custom IAM role named LSE_CustomMetricsDeployer with the following permissions:

title: LSE Custom Metrics Deployer
description: Allows Label Studio Enterprise to deploy and invoke custom metric Cloud Functions
stage: GA
includedPermissions:
  # Cloud Functions lifecycle
  - cloudfunctions.functions.create
  - cloudfunctions.functions.update
  - cloudfunctions.functions.delete
  - cloudfunctions.functions.get
  - cloudfunctions.functions.list
  # Source upload via generateUploadUrl
  - cloudfunctions.functions.generateUploadUrl
  # Source code download from GCS staging bucket
  - storage.objects.get
  - storage.objects.create
  # Cloud Functions invocation (2nd gen uses Cloud Run)
  - run.routes.invoke
  # View function execution logs
  - logging.logEntries.list
  - logging.logs.list
  # Act as the runtime service account
  - iam.serviceAccounts.actAs

You can create this role using the gcloud CLI:

gcloud iam roles create LSE_CustomMetricsDeployer \
  --project=YOUR_GCP_PROJECT \
  --title="LSE Custom Metrics Deployer" \
  --description="Allows Label Studio Enterprise to deploy and invoke custom metric Cloud Functions" \
  --permissions=cloudfunctions.functions.create,cloudfunctions.functions.update,cloudfunctions.functions.delete,cloudfunctions.functions.get,cloudfunctions.functions.list,cloudfunctions.functions.generateUploadUrl,storage.objects.get,storage.objects.create,run.routes.invoke,logging.logEntries.list,logging.logs.list,iam.serviceAccounts.actAs

Then bind it to the Label Studio Enterprise identity. Replace LSE_IDENTITY with the appropriate member (see the deployment-specific sections below):

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="LSE_IDENTITY" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

note

Instead of a single custom role, you can also assign the following predefined roles: roles/cloudfunctions.developer, roles/run.invoker, roles/storage.objectAdmin, roles/logging.viewer, and roles/iam.serviceAccountUser. The custom role above follows the principle of least privilege.

Deployed with Docker Compose on a Compute Engine VM

If you deployed Label Studio Enterprise using Docker Compose on a GCP Compute Engine VM, do the following:

  1. Ensure the Compute Engine VM is running with a service account that has the LSE_CustomMetricsDeployer custom role (or equivalent predefined roles) bound at the project level. Alternatively, create a dedicated service account for Label Studio:

    gcloud iam service-accounts create lse-app \
  --display-name="LSE Application" \
  --project=YOUR_GCP_PROJECT

gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

    Then assign this service account to your Compute Engine instance.

  2. In the docker-compose.yaml file that you use to deploy Label Studio Enterprise, add the following environment variables in the app and rqworkers sections:

important

Replace YOUR_GCP_PROJECT with your GCP project ID and YOUR_GCP_REGION with your preferred region (e.g. us-central1).

CUSTOM_METRIC_PROVIDER=gcp
GCP_PROJECT_CUSTOM_METRICS=YOUR_GCP_PROJECT
GCP_REGION_CUSTOM_METRICS=YOUR_GCP_REGION
GCP_SERVICE_ACCOUNT_CUSTOM_METRICS=lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

note

When running on Compute Engine with a properly scoped service account, Label Studio uses Application Default Credentials (ADC) automatically. No key file is required.

After you set up these permissions in your environment, you’re ready to write your custom agreement metric and add it to Label Studio Enterprise:

  1. Write your custom agreement metric.
  2. Add your custom agreement metric to Label Studio Enterprise.

Deployed in GKE with Workload Identity

If you deployed Label Studio Enterprise in Google Kubernetes Engine (GKE) with Workload Identity, do the following:

  1. Create a GCP service account for Label Studio Enterprise (if you haven’t already):

    gcloud iam service-accounts create lse-app \
  --display-name="LSE Application" \
  --project=YOUR_GCP_PROJECT

  2. Bind the LSE_CustomMetricsDeployer custom role to the service account:

    gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

  3. Allow the Kubernetes service account to impersonate the GCP service account:

    gcloud iam service-accounts add-iam-policy-binding \
  lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:YOUR_GCP_PROJECT.svc.id.goog[YOUR_NAMESPACE/YOUR_K8S_SERVICE_ACCOUNT]"

  4. Update your Helm values.yaml to annotate the Kubernetes service account and set the required environment variables:

    app:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: lse-app@YOUR_GCP_PROJECT.iam.gserviceaccount.com
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

rqworker:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

  5. Restart your Helm release.

After you set up these permissions in your environment, you’re ready to write your custom agreement metric and add it to Label Studio Enterprise:

  1. Write your custom agreement metric.
  2. Add your custom agreement metric to Label Studio Enterprise.

Deployed in GKE without Workload Identity

If you deployed Label Studio Enterprise in GKE and are not using Workload Identity, the pods authenticate using the GKE node’s service account.

  1. Identify the service account used by your node pool. In the GCP console, go to Kubernetes Engine > Clusters > YOUR_CLUSTER > Node Pools > YOUR_NODE_POOL > Security and note the service account.

  2. Bind the LSE_CustomMetricsDeployer custom role to the node pool’s service account:

    gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT \
  --member="serviceAccount:YOUR_NODE_SERVICE_ACCOUNT@YOUR_GCP_PROJECT.iam.gserviceaccount.com" \
  --role="projects/YOUR_GCP_PROJECT/roles/LSE_CustomMetricsDeployer"

  3. Update your Helm values.yaml to set the required environment variables:

    app:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

rqworker:
  extraEnvironmentVars:
    CUSTOM_METRIC_PROVIDER: gcp
    GCP_PROJECT_CUSTOM_METRICS: YOUR_GCP_PROJECT
    GCP_REGION_CUSTOM_METRICS: YOUR_GCP_REGION
    GCP_SERVICE_ACCOUNT_CUSTOM_METRICS: lse-custom-metrics-runtime@YOUR_GCP_PROJECT.iam.gserviceaccount.com

  4. Restart your Helm release.

After you set up these permissions in your environment, you’re ready to write your custom agreement metric and add it to Label Studio Enterprise:

  1. Write your custom agreement metric.
  2. Add your custom agreement metric to Label Studio Enterprise.

Environment variable reference for GCP

Variable Description Default
CUSTOM_METRIC_PROVIDER Set to gcp to use Google Cloud Functions instead of AWS Lambda. aws
GCP_PROJECT_CUSTOM_METRICS The GCP project ID where Cloud Functions are deployed. (none, required)
GCP_REGION_CUSTOM_METRICS The GCP region for Cloud Functions (e.g. us-central1, europe-west1). us-central1
GCP_SERVICE_ACCOUNT_CUSTOM_METRICS The email of the runtime service account that deployed Cloud Functions execute as. (none, optional)

Adding functions to projects in Label Studio

When using Google Cloud Functions, Label Studio stores and reuses the function identity per project, then resolves the function URL automatically during scoring.

To discover this in Google Cloud:

  1. Open Cloud Functions in the configured GCP project/region.
  2. Locate functions created for your project using the auto-generated name pattern: custom-metric-<project_id>-<timestamp>.
  3. Open the function details to view logs, deployment state, and trigger URL.
Function lifecycle behavior
Behavior Description
Deploy Creates a new function if no matching existing function is found.
Update Updates the existing function when possible.
Provider/metric change If you switch away from this metric/provider, Label Studio attempts to delete the previously managed GCP function.
Runtime status in app The app reports states such as NotDeployed, Pending, Active, Failed, NotFound, Error.
Discoverability and metadata Label Studio applies internal labels (metadata) to functions for project/org/host tracking. These labels are primarily for internal management and cloud-side filtering. The function HTTPS URL is resolved and cached automatically by Label Studio.

important

Prefix and tags/labels are not currently exposed as user-facing fields in the UI for GCP.

Advanced users can only set such values through API-level metric parameters (if your organization exposes that workflow), not via standard settings screens.

In this article

  1. How to write your custom agreement metric
  2. Add your custom agreement metric to Label Studio
  3. On-prem deployment using AWS Lambda
  4. On-prem deployment using Google Cloud Functions
  5. On-prem deployments
  6. AWS Lambda
  7. Google Cloud Functions
Designed for teams of all sizes Contact Sales