Clarifai Home

Annotations

Label your data.

Annotations

Annotations (also known as labels) describe your inputs. When you add inputs to your app, we will create an input level annotation for each input. This input level annotation contains any data you provided in POST /inputs call. Models in your default workflow can also write annotations.

Once your input is successfully indexed, you can add additional annotations such as concepts and bounding boxes.

Add Annotations

You can label your inputs by calling the POST /annotations endpoint. For example, you can add concept(s) to an image, draw a bounding box, or label concept(s) in a video frame.

When you add an annotation, the app's default workflow will not run by default. This means that the annotation will not be immediately available for training of your custom model or for visual search. To make the annotation available for AI based search and training, you need to provide embed_model_version_id field. This field specifies how to associate the annotation for your input to one of the embedding models in your default workflow. When associated during patching then we know how to index it for training and visual search, therefore if your use case includes those features it is recommended to provide this field on each add annotation call.

You can add from 1 up to 128 annotations in a single API call.

Each annotation should contain at most one region. If it is a video, each annotation should contain 1 frame. If there are multiple regions in a frame you want to label, you can add multiple annotations for each regoin and each annotation will be contained within the same frame but a different region.

Annotate images with concepts

To annotate a concept present anywhere in an image:

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

post_annotations_response = stub.PostAnnotations(
    service_pb2.PostAnnotationsRequest(
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                data=resources_pb2.Data(
                    concepts=[
                        resources_pb2.Concept(id="tree", value=1.),  # 1 means true, this concept is present.
                        resources_pb2.Concept(id="water", value=0.)  # 0 means false, this concept is not present.
                    ]
                ),
                embed_model_version_id="{EMBED_MODEL_VERSION_ID}"
            )
        ]
    ),
    metadata=metadata
)

if post_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Post annotations failed, status: " + post_annotations_response.status.description)

Annotate New Bounding Boxes in an Image

You can label a new bounding box by providing bounding box coordinates.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

post_annotations_response = stub.PostAnnotations(
    service_pb2.PostAnnotationsRequest(
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                data=resources_pb2.Data(
                    regions=[
                        resources_pb2.Region(
                            region_info=resources_pb2.RegionInfo(
                                bounding_box=resources_pb2.BoundingBox(       # draw a bounding box
                                    top_row=0,
                                    left_col=0,
                                    bottom_row=0.5,
                                    right_col=0.5
                                )
                            ),
                            data=resources_pb2.Data(
                                concepts=[
                                    resources_pb2.Concept(id="tree", value=1.),  # 1 means true, this concept is present.
                                    resources_pb2.Concept(id="water", value=0.)  # 0 means false, this concept is not present.
                                ]
                            )
                        )
                    ]
                ),
                embed_model_version_id="{EMBED_MODEL_VERSION_ID}"
            ),
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                data=resources_pb2.Data(
                    regions=[
                        resources_pb2.Region(
                            region_info=resources_pb2.RegionInfo(
                                bounding_box=resources_pb2.BoundingBox(        # draw another bounding box
                                    top_row=0.6,
                                    left_col=0.6,
                                    bottom_row=0.8,
                                    right_col=0.8
                                )
                            ),
                            data=resources_pb2.Data(
                                concepts=[
                                    resources_pb2.Concept(id="bike", value=1.),  # 1 means true, this concept is present.
                                ]
                            )
                        )
                    ]
                ),
                embed_model_version_id="{EMBED_MODEL_VERSION_ID}"
            )
        ]
    ),
    metadata=metadata
)

if post_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Post annotations failed, status: " + post_annotations_response.status.description)

Annotate Existing Regions in an Image

When you add an input, detection models (such as Face Detection or General Detection) will detect regions in your image where there appear to be relevant objects. You can check these detected regions by listing model's annotations. Your labels should be contained within Region.Data. Each annotation can have only 1 region. If you want to label multiple regions, it is possible to label multiple annotations in a single API call.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

post_annotations_response = stub.PostAnnotations(
    service_pb2.PostAnnotationsRequest(
        annotations=[
            resources_pb2.Annotation(                # label a region in this image
                input_id="{YOUR_INPUT_ID}",
                data=resources_pb2.Data(
                    regions=[
                        resources_pb2.Region(
                            id="{REGION_ID_1}" ,  # this should be a region id returned from list annotations call
                            data=resources_pb2.Data(
                                concepts=[
                                    resources_pb2.Concept(id="tree", value=1.),  # 1 means true, this concept is present.
                                    resources_pb2.Concept(id="water", value=0.)  # 0 means false, this concept is not present.
                                ]
                            )
                        )
                    ]
                ),
                embed_model_version_id="{EMBED_MODEL_VERSION_ID}"
            ),
            resources_pb2.Annotation(                # label another region in this image
                input_id="{YOUR_INPUT_ID}",
                data=resources_pb2.Data(
                    regions=[
                        resources_pb2.Region(
                            id="{REGION_ID_2}" ,  # this should be a region id returned from list annotations call
                            data=resources_pb2.Data(
                                concepts=[
                                    resources_pb2.Concept(id="bike", value=1.),  # 1 means true, this concept is present.
                                ]
                            )
                        )
                    ]
                ),
                embed_model_version_id="{EMBED_MODEL_VERSION_ID}"
            ),
        ]
    ),
    metadata=metadata
)

if post_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Post annotations failed, status: " + post_annotations_response.status.description)

Annotate Images with Different user_id and status.

Each annotation is tied to a user or a model in your workflow. By default, when a user posts an annotation, this user is the owner of the annotation. Sometimes however, you might want to post an annotation as other user, for example, when assigning an image to another user, an annotation can be created with another user_id (and status PENDING).

Note: only the app owner can post an annotation with other user's user_id, collaborators cannot.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

post_annotations_response = stub.PostAnnotations(
    service_pb2.PostAnnotationsRequest(
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                user_id="{USER_ID}",    # If empty, it is the user who posts this annotation
                data=status_pb2.Status(
                    code=status_code_pb2.ANNOTATION_PENDING  # annotation pending status. By default success.
                ),
            )
        ]
    ),
    metadata=metadata
)

if post_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Post annotations failed, status: " + post_annotations_response.status.description)

List Annotations

You can get a list of annotations within your app with a GET call. Annotations will be returned from oldest to newest.

These requests are paginated. By default each page will return 20 annotations.

List All User Created Annotations in Your App

To list all your user labelled annotations.

Note this will not show annotations by models in your worfklow. To include model created annotations, you need to set list_all_annotations to true.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

list_annotations_response = stub.ListAnnotations(
    service_pb2.ListAnnotationsRequest(per_page=10),
    metadata=metadata
)

if list_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("List annotations failed, status: " + list_annotations_response.status.description)

for annotation_object in list_annotations_response.annotations:
    print(annotation_object)

List All Annotations in Your App

List all annotations, including models created.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

list_annotations_response = stub.ListAnnotations(
    service_pb2.ListAnnotationsRequest(per_page=10, list_all_annotations=True),
    metadata=metadata
)

if list_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("List annotations failed, status: " + list_annotations_response.status.description)

for annotation_object in list_annotations_response.annotations:
    print(annotation_object)

List User Created Annotations by Input IDs

To list all user created annotations for certain input (one or several), provide a list of input IDs.

Note: this will not show annotations by models in your worfklow. To include model created annotations, you need to set list_all_annotations to true.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

list_annotations_response = stub.ListAnnotations(
    service_pb2.ListAnnotationsRequest(input_ids=["{YOUR_INPUT_ID_1}". "{YOUR_INPUT_ID_2}"], per_page=10),
    metadta=metadata
)

if list_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("List annotations failed, status: " + list_annotations_response.status.description)

for annotation_object in list_annotations_response.annotations:
    print(annotation_object)

List Annotations by Input IDs and Annotation IDs

You can list annotations by both input IDs and annotation IDs. Number of input IDs and annotation IDs should be the same. Since we are finding annotatieon by IDs this will match any user or model created annotations.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

list_annotations_response = stub.ListAnnotations(
    service_pb2.ListAnnotationsRequest(
       input_ids=["{YOUR_INPUT_ID_1}". "{YOUR_INPUT_ID_2}"]
        ids=["{YOUR_ANNOTATION_ID_1}", "{YOUR_ANNOTATION_ID_2}"],
        per_page=10
    ),
    metadata=metadata
)

if list_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("List annotations failed, status: " + list_annotations_response.status.description)

for annotation_object in list_annotations_response.annotations:
    print(annotation_object)

List Annotations by User IDs

An annotation is created by either a user or a model. You can list annotations created by specific user(s) by provider their user IDs.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

list_annotations_response = stub.ListAnnotations(
    service_pb2.ListAnnotationsRequest(user_ids=["{USER_ID_1}", "{USER_ID_2}"], per_page=10),
    metadata=metadata
)

if list_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("List annotations failed, status: " + list_annotations_response.status.description)

for annotation_object in list_annotations_response.annotations:
    print(annotation_object)

List Annotations by Model Version IDs

An annotation is created by either a user or a model. For example if your workflow has a detection model, when you add input, the model will detect objects in your input. You can see these detected objects by listing the annotations created detection model. You can also label these regions by using Post annotation with the region id returned from this call.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

list_annotations_response = stub.ListAnnotations(
    service_pb2.ListAnnotationsRequest(
        model_version_ids=["{MODEL_VERSION_ID_1}", "{MODEL_VERSION_ID_2}"],
        per_page=10
    ),
    metadata=metadata
)

if list_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("List annotations failed, status: " + list_annotations_response.status.description)

for annotation_object in list_annotations_response.annotations:
    print(annotation_object)

Update Annotations

Changing annotation data is possible by PATCHing exisitng annotations. The application owner can change any user-created annotations. Collaborators are not allowed to change annotations made by other collaborators.

Generally speaking, you should send PATCH when you want to change the data you have posted, for example, changing the concept from positive to negative or adjusting the bbounding box cooridnates. If you want to adding more tags, you can always POST new annotations. There is no limit on how many annotations an input can have.

Update supports overwrite, merge, remove actions. You can update from 1 up to 128 annotations in a single API call.

Update Annotation with Concepts

Update an annotation of a image with a new concept, or to change a concept value from true to false (or vice versa).

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

patch_annotations_response = stub.PatchAnnotations(
    service_pb2.PatchAnnotationsRequest(
        action="merge",  # Supported actions: overwrite, merge, remove.
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                id="{YOUR_ANNOTATION_ID}",
                data=resources_pb2.Data(
                    concepts=[
                        resources_pb2.Concept(id="apple", value=1.)  # 1 means true, this concept is present.
                    ]
                )
            )
        ]
    ),
    metadata=metadata
)

if patch_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Patch annotations failed, status: " + patch_annotations_response.status.description)

Update Annotation with Concepts in a Region

When you update region data, you must nest this new data within region.data. Set the region_id to the current region_id if you do not want to change or remove this region.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

patch_annotations_response = stub.PatcchAnnotations(
    service_pb2.PatchAnnotationsRequest(
        action="merge",  # Supported actions: overwrite, merge, remove.
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                id="{YOUR_ANNOTATION_ID}",
                data=resources_pb2.Data(
                    regions=[
                        resources_pb2.Region(
                            id="{REGION_ID}" ,  # this should be the region id of this annotation before patch
                            data=resources_pb2.Data(
                                concepts=[
                                    resources_pb2.Concept(id="tree", value=1.),  # 1 means true, this concept is present.
                                ]
                            )
                        )
                    ]
                )
            )
        ]
    ),
    metadata=metadata
)

if patch_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Patch annotations failed, status: " + patch_annotations_response.status.description)

Update Annotation Region Coordinates

You can update region bounding boxes coordinates. When changing the region, you should use overwrite action. With overwrite action, you need to provide any data you want to keep in this annotation.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

patch_annotations_response = stub.PatcchAnnotations(
    service_pb2.PatchAnnotationsRequest(
        action="overwrite",
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                id="{YOUR_ANNOTATION_ID}",
                data=resources_pb2.Data(
                    regions=[
                        resources_pb2.Region(
                            region_info=resources_pb2.RegionInfo(
                                bounding_box=resources_pb2.BoundingBox(        # move bounding bbox to a new cooridnates
                                    top_row=0.5,
                                    left_col=0.5,
                                    bottom_row=0.8,
                                    right_col=0.8
                                )
                            ),
                            data=resources_pb2.Data(    # need to provde tags you previously labeled since this is overwrite action
                                concepts=[
                                    resources_pb2.Concept(id="bike", value=1.),  # 1 means true, this concept is present.
                                ]
                            )
                        )
                    ]
                )
            )
        ]
    ),
    metadata=metadata
)

if patch_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Patch annotations failed, status: " + patch_annotations_response.status.description)

Update Annotation Status

You can update an annotation status.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

patch_annotations_response = stub.PatchAnnotations(
    service_pb2.PatchAnnotationsRequest(
        action="merge",  # Supported actions: overwrite, merge, remove.
        annotations=[
            resources_pb2.Annotation(
                input_id="{YOUR_INPUT_ID}",
                id="{YOUR_ANNOTATION_ID}",
                status=status_pb2.Status(
                    code=status_code_pb2.ANNOTATION_SUCCESS
                )
            )
        ]
    ),
    metadata=metadata
)

if patch_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Patch annotations failed, status: " + patch_annotations_response.status.description)

Delete Annotations

Delete Annotation by Input ID and Annotation ID

You can delete a single annotation by input ID and annotation ID.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

delete_annotation_response = stub.DeleteAnnotation(
    service_pb2.DeleteAnnotationRequest(
        input_id="{YOUR_INPUT_ID}",
        annotation_id="{YOUR_ANNOTATION_ID}"
    ),
    metadata=metadata
)

if delete_annotation_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Delete annotation failed, status: " + delete_annotation_response.status.description)

Bulk Delete Annotations by Input Ids and Annotation IDs

You can delete multiple annotations in one API call. You need to provide a list of input IDs and a list of annotation IDs. The number of input IDs has to match number of annotation IDs.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

delete_annotations_response = stub.DeleteAnnotations(
    service_pb2.DeleteAnnotationsRequest(
        input_ids=["{YOUR_INPUT_ID_1}", "{YOUR_INPUT_ID_2}"],
        annotation_id=["{YOUR_ANNOTATION_ID_1}", "{YOUR_ANNOTATION_ID_2}"]
    ),
    metadata=metadata
)

if delete_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Delete annotations failed, status: " + delete_annotations_response.status.description)
import com.clarifai.grpc.api.*;
import com.clarifai.grpc.api.status.*;

// Insert here the initialization code as outlined on this page:
// https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

BaseResponse deleteAnnotationsResponse = stub.deleteAnnotations(
    DeleteAnnotationsRequest.newBuilder()
        .addInputIds("{YOUR_INPUT_ID_1}")
        .addInputIds("{YOUR_INPUT_ID_2}")
        .addIds("{YOUR_ANNOTATION_ID_1}")
        .addIds("{YOUR_ANNOTATION_ID_2}")
        .build()
);

if (deleteAnnotationsResponse.getStatus().getCode() != StatusCode.SUCCESS) {
    throw new RuntimeException("Delete annotations failed, status: " + deleteAnnotationsResponse.getStatus());
}
// Insert here the initialization code as outlined on this page:
// https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

stub.DeleteAnnotations(
    {
        input_ids: ["{YOUR_INPUT_ID_1}", "{YOUR_INPUT_ID_2}"],
        annotation_ids: ["{YOUR_ANNOTATION_ID_1}", "{YOUR_ANNOTATION_ID_2}"]
    },
    metadata,
    (err, response) => {
        if (err) {
            throw new Error(err);
        }

        if (response.status.code !== 10000) {
            throw new Error("Delete annotations failed, status: " + response.status.description);
        }
    }
);
curl -X DELETE \
  -H "Authorization: Key YOUR_API_KEY" \
  -d '
  {
    "input_ids":["{YOUR_INPUT_ID_1}","{YOUR_INPUT_ID_2}"],
    "ids":["{YOUR_ANNOTATION_ID_1}", "{YOUR_ANNOTATION_ID_2}"]
  }'\
  https://api.clarifai.com/v2/annotations

Bulk Delete All Annotations by Input IDs

To delete all annotations of a given input, you just need to set input ID(s). This will delete all annotations for these input(s) EXCEPT input level annotations which only get deleted if you delete the inputs themselves.

# Insert here the initialization code as outlined on this page:
# https://docs.clarifai.com/api-guide/api-overview/api-clients#client-installation-instructions

delete_annotations_response = stub.DeleteAnnotations(
    service_pb2.DeleteAnnotationsRequest(
        input_ids=["{YOUR_INPUT_ID_1}", "{YOUR_INPUT_ID_2}"]
    ),
    metadata=metadata
)

if delete_annotations_response.status.code != status_code_pb2.SUCCESS:
    raise Exception("Delete annotations failed, status: " + delete_annotations_response.status.description)
PreviousScribe LabelNextTraining Data

Last updated