Bifrost metadata

Attention

This is an experimental feature and is not enabled by default. Please contact us if you require access for your use case.

When downloading a collection, a COCO JSON file is included in the ZIP archive. In addition to the standard COCO fields, the file would also contain metadata specific to Bifrost renders.

This custom metadata is located under the images and annotations sections of the JSON, within a field named bifrost_metadata.

Example format

coco.json

{
    "images": [
        {
            "id": 1,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_0.jpg",
            "bifrost_metadata": {
                "bifrost_camera_position": {
                    "x": 0,
                    "y": 0,
                    "z": 1000
                },
                "bifrost_camera_rotation": {
                    "roll": 0,
                    "pitch": -30,
                    "yaw": 0
                },
                "bifrost_camera_heading": 0.0,
                "bifrost_camera_fov": 90.0
            }
        }
    ],
    "annotations": [
        {
            "id": 1,
            "image_id": 1,
            "bifrost_metadata": {
                "bifrost_object_id": "boat_1",
                "bifrost_heading_absolute_degrees": 90.02760314941406,
                "bifrost_distance_to_camera_m": 34.114948730468754,
                "bifrost_absolute_position_m": {
                    "x": 32.7561962890625,
                    "y": 0.5690269851684571,
                    "z": 0.4848830032348633
                },
                "bifrost_absolute_rotation_degrees": {
                    "roll": -0.8629000186920166,
                    "pitch": 90.02760314941406,
                    "yaw": -1.8286000490188599
                },
                "bifrost_scale": {
                    "x": 1,
                    "y": 1,
                    "z": 1
                }
            }
        }
    ]
}

Image level metadata

By default, the following information is exported under the images.bifrost_metadata field:

Field Name	Unit	Description
bifrost_camera_position	Meters	Camera position in Bifrost world coordinates.
bifrost_camera_rotation	Degrees	Camera rotation relative to Bifrost world.
bifrost_camera_heading	Degrees	Camera heading relative to Bifrost world.
bifrost_camera_fov	Degrees	Horizontal field of view of the camera.

Object level metadata

Note

Object level metadata is only available for assets added to the World via world.add.

Objects such as Sky, Land, and Water would not have any associated metadata.

By default, the following information is exported under the annotations.bifrost_metadata field:

Field Name	Unit	Description
bifrost_object_id		The ID of the object. This can be used for tracking tasks.
bifrost_heading_absolute_degrees	Degrees	Object’s heading relative to Bifrost world.
bifrost_distance_to_camera_m*	Meters	Distance from the camera to the center of the object’s base.
bifrost_absolute_position_m	Meters	Object position in Bifrost world coordinates.
bifrost_absolute_rotation_degrees	Degrees	Object rotation relative to Bifrost world.
bifrost_scale		Scale (multiplier) applied to the object in each dimension.

*If you require a different method of distance calculation, please let us know!

Adding custom metadata

Note

Custom metadata can currently only be added at the image level.

Annotation-level custom metadata is not supported at this time.

To enable custom metadata, update the imports

Import the necessary modules

from bbi.experimental import World, METADATA_UNSET_KEY
from bbi import *

With the experimental module imported, add custom metadata for each frame using scenario.set_metadata_keyframe.

Add custom metadata

scenario.set_metadata_keyframe(<frame_index>, <key>, <value>)

An important behavior to note is that metadata keys set in a previous frame will automatically persist into subsequent frames unless explicitly updated. To remove a key from a specific frame, set its value to METADATA_UNSET_KEY.

Here is a full example of how to add custom metadata

Full example

from bbi.experimental import World, METADATA_UNSET_KEY
from bbi import *

world = World(...)
scenario = world.new_scenario(3)
# Frame 0
scenario.set_metadata_keyframe(0, "weather", "rainy")
scenario.set_metadata_keyframe(0, "sun", {"elevation": 30, "azimuth": 180})
scenario.set_metadata_keyframe(0, "special_key", 1)
# Frame 1
scenario.set_metadata_keyframe(1, "weather", "sunny")
scenario.set_metadata_keyframe(1, "special_key", METADATA_UNSET_KEY)
# Frame 2
scenario.set_metadata_keyframe(2, "weather", "cloudy")

scenario.render()

Example output

coco.json

{
    "images": [
        {
            "id": 1,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_0.jpg",
            "bifrost_metadata": {
                "weather": "rainy",
                "sun": {
                  "elevation": 30,
                  "azimuth": 180
                },
                "special_key": 1,
            }
        },
        {
            "id": 2,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_1.jpg",
            "bifrost_metadata": {
                "weather": "sunny",
                "sun": {
                  "elevation": 30,
                  "azimuth": 180
                },
            }
        },
        {
            "id": 3,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_2.jpg",
            "bifrost_metadata": {
                "weather": "sunny",
                "sun": {
                  "elevation": 30,
                  "azimuth": 180
                },
            }
        }
    ],
}