Bifrost metadata

When downloading a collection, a COCO JSON file is included in the ZIP archive. In addition to the standard COCO fields, the file would also contain metadata specific to Bifrost renders.

This custom metadata is located under the images and annotations sections of the JSON, within a field named bifrost_metadata.

Example format

coco.json

{
    "images": [
        {
            "id": 1,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_0.jpg",
            "bifrost_metadata": {
                "bifrost_camera_position": {
                    "x": 0,
                    "y": 0,
                    "z": 1000
                },
                "bifrost_camera_rotation": {
                    "roll": 0,
                    "pitch": -30,
                    "yaw": 0
                },
                "bifrost_camera_heading": 0.0,
                "bifrost_camera_fov": 90.0
            }
        }
    ],
    "annotations": [
        {
            "id": 1,
            "image_id": 1,
            "bifrost_metadata": {
                "bifrost_object_id": "boat_1",
                "bifrost_heading_absolute_degrees": 90.02760314941406,
                "bifrost_distance_to_camera_m": 34.114948730468754,
                "bifrost_absolute_position_m": {
                    "x": 32.7561962890625,
                    "y": 0.5690269851684571,
                    "z": 0.4848830032348633
                },
                "bifrost_absolute_rotation_degrees": {
                    "roll": -0.8629000186920166,
                    "pitch": 90.02760314941406,
                    "yaw": -1.8286000490188599
                },
                "bifrost_absolute_centroid_position_m": {
                    "x": 33.5,
                    "y": 1.2,
                    "z": 5.8
                },
                "bifrost_absolute_size_m": {
                    "x": 20.299549560546875,
                    "y": 21.7643212890625,
                    "z": 65.81230957031251
                },
                "bifrost_absolute_cuboid_positions_m": [
                    {"x": 23.15, "y": -9.68, "z": -27.12},
                    {"x": 43.85, "y": -9.68, "z": -27.12},
                    {"x": 23.15, "y": 12.08, "z": -27.12},
                    {"x": 43.85, "y": 12.08, "z": -27.12},
                    {"x": 23.15, "y": -9.68, "z": 38.69},
                    {"x": 43.85, "y": -9.68, "z": 38.69},
                    {"x": 23.15, "y": 12.08, "z": 38.69},
                    {"x": 43.85, "y": 12.08, "z": 38.69}
                ],
                "bifrost_scale": {
                    "x": 1,
                    "y": 1,
                    "z": 1
                }
            }
        }
    ]
}

Image level metadata

By default, the following information is exported under the images.bifrost_metadata field:

Field Name	Unit	Description
bifrost_camera_position	Meters	Camera position in Bifrost world coordinates.
bifrost_camera_rotation	Degrees	Camera rotation relative to Bifrost world.
bifrost_camera_heading	Degrees	Camera heading relative to Bifrost world.
bifrost_camera_fov	Degrees	Horizontal field of view of the camera.

Object level metadata

Note

Object level metadata is only available for assets added to the World via world.add.

Objects such as Sky, Land, and Water would not have any associated metadata.

By default, the following information is exported under the annotations.bifrost_metadata field:

Field Name	Unit	Description
bifrost_object_id		The ID of the object. This can be used for tracking tasks.
bifrost_heading_absolute_degrees	Degrees	Object’s heading relative to Bifrost world.
bifrost_distance_to_camera_m*	Meters	Distance from the camera to the center of the object’s base.
bifrost_absolute_position_m	Meters	Object position in Bifrost world coordinates.
bifrost_absolute_rotation_degrees	Degrees	Object rotation relative to Bifrost world.
bifrost_absolute_centroid_position_m	Meters	Object’s centroid position in Bifrost world coordinates.
bifrost_absolute_size_m	Meters	Object size in x, y, and z dimensions, which corresponds to depth, width, and height.
bifrost_absolute_cuboid_positions_m	Meters	Array of 8 corner positions of the object’s 3D bounding cuboid in Bifrost world coordinates.
bifrost_scale		Scale (multiplier) applied to the object in each dimension.
bifrost_in_frame	Boolean	Whether the object is in the frame. An object is considered to be in frame if at least one pixel of the object is visible in the image.

*If you require a different method of distance calculation, please let us know!

Adding custom metadata

Note

Custom metadata can currently only be added at the image level.

Annotation-level custom metadata is not supported at this time.

Add custom metadata for each frame using scenario.set_metadata_keyframe.

Add custom metadata

scenario.set_metadata_keyframe(<frame_index>, <key>, <value>)

An important behavior to note is that metadata keys set in a previous frame will automatically persist into subsequent frames unless explicitly updated. To remove a key from a specific frame, set its value to METADATA_UNSET_KEY.

Here is a full example of how to add custom metadata

Full example

from bbi import *

world = World(...)
scenario = world.new_scenario(3)
# Frame 0
scenario.set_metadata_keyframe(0, "weather", "rainy")
scenario.set_metadata_keyframe(0, "sun", {"elevation": 30, "azimuth": 180})
scenario.set_metadata_keyframe(0, "special_key", 1)
# Frame 1
scenario.set_metadata_keyframe(1, "weather", "sunny")
scenario.set_metadata_keyframe(1, "special_key", METADATA_UNSET_KEY)
# Frame 2
scenario.set_metadata_keyframe(2, "weather", "cloudy")

scenario.render()

Example output

coco.json

{
    "images": [
        {
            "id": 1,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_0.jpg",
            "bifrost_metadata": {
                "weather": "rainy",
                "sun": {
                  "elevation": 30,
                  "azimuth": 180
                },
                "special_key": 1,
            }
        },
        {
            "id": 2,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_1.jpg",
            "bifrost_metadata": {
                "weather": "sunny",
                "sun": {
                  "elevation": 30,
                  "azimuth": 180
                },
            }
        },
        {
            "id": 3,
            "file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_2.jpg",
            "bifrost_metadata": {
                "weather": "cloudy",
                "sun": {
                  "elevation": 30,
                  "azimuth": 180
                },
            }
        }
    ],
}

Frequently asked questions

You can download the helper functions used in this section here.

Note

These helper functions depend on numpy and scipy, which are not included in the Python standard library. Install them with:

pip install numpy scipy

All examples below refer to the following image and its associated COCO annotation file:

• COCO annotation file

• Image:

  

---

Are the object level metadata relative to the camera or the world?

All object-level metadata are absolute, meaning they are defined in the world coordinate system, not relative to the camera.

For more information on the coordinate system, see Coordinate system.

---

Can I convert the absolute coordinates to be relative to the camera?

Yes! Absolute coordinates can be converted to be relative to any object in the world, including the camera.

To do this:

1. Retrieve the object’s bifrost_absolute_position_m and bifrost_absolute_rotation_degrees.

2. Create a Transform object (from the helper functions file) using those values.

3. Use the Transform.relative_to() method to convert its absolute transform into the camera’s local coordinate space.

Example: get relative transform and plot relative yaw with respect to camera

Get relative yaw of an object with respect to the camera

import json
from PIL import Image, ImageDraw, ImageFont

coco = json.load(open("coco.json"))

# get image info
image_info = coco["images"][0]
img_id = image_info["id"]
img_width, img_height = image_info["width"], image_info["height"]
position_data = image_info["bifrost_metadata"]["bifrost_camera_position_m"]
cam_position = Vector3(position_data["x"], position_data["y"], position_data["z"])
rotation_data = image_info["bifrost_metadata"]["bifrost_camera_rotation_degrees"]
cam_rotation = Vector3(rotation_data["roll"], rotation_data["pitch"], rotation_data["yaw"])
cam_transform = Transform(cam_position, cam_rotation)

# get annotations of objects that are in frame
annotations = [
    ann for ann in coco["annotations"] if ann["image_id"] == img_id
    and ann.get("bifrost_metadata", {}).get("bifrost_in_frame")
]

img = Image.open("RGB_camera_0000.jpg")
draw = ImageDraw.Draw(img)
for ann in annotations:
    # get relative transform
    ann_transform = Transform(
            position=Vector3(**ann["bifrost_metadata"]["bifrost_absolute_position_m"]),
            rotation=Vector3(*ann["bifrost_metadata"]["bifrost_absolute_rotation_degrees"].values()),
    )
    relative_transform = ann_transform.relative_to(cam_transform)

    # write relative yaw
    bbox = ann["bbox"]
    center = (bbox[0] + bbox[2] / 2, bbox[1] + bbox[3] / 2)
    font = ImageFont.truetype("arial.ttf", 24)
    text = f"{relative_transform.rotation.z:.2f}°"
    text_bbox = draw.textbbox(center, text, font=font)
    bg_box = (text_bbox[0]-5, text_bbox[1]-5, text_bbox[2]+5, text_bbox[3]+5)
    draw.rectangle(bg_box, fill="white")
    draw.text(center, text, fill=(200, 0, 0), font=font)

---

How do I convert from world coordinates to screen coordinates?

Use the world_to_screen function from the helper functions file to convert a 3D world-space point into 2D screen-space coordinates.

There are two important cases to be aware of:

• The function returns None when the point lies behind the camera.

• The function returns coordinates outside the image bounds when the point is in front of the camera but outside the camera’s field of view.

Example: convert world coordinates to screen coordinates and plot objects’ centroids

Convert world coordinates to screen coordinates and plot objects’ centroids

import json
from PIL import Image, ImageDraw

coco = json.load(open("coco.json"))

# get image info
image_info = coco["images"][0]
img_id = image_info["id"]
img_width, img_height = image_info["width"], image_info["height"]
position_data = image_info["bifrost_metadata"]["bifrost_camera_position_m"]
cam_position = Vector3(position_data["x"], position_data["y"], position_data["z"])
rotation_data = image_info["bifrost_metadata"]["bifrost_camera_rotation_degrees"]
cam_rotation = Vector3(rotation_data["roll"], rotation_data["pitch"], rotation_data["yaw"])
cam_hfov = image_info["bifrost_metadata"]["bifrost_camera_fov_degrees"]
cam_vfov = calculate_vfov(cam_hfov, (img_width, img_height))

# get annotations of objects that are in frame
annotations = [
    ann for ann in coco["annotations"] if ann["image_id"] == img_id
    and ann.get("bifrost_metadata", {}).get("bifrost_in_frame")
]

img = Image.open("RGB_camera_0000.jpg")
draw = ImageDraw.Draw(img)
for ann in annotations:
    centroid_data = ann["bifrost_metadata"]["bifrost_absolute_centroid_position_m"]
    centroid = Vector3(centroid_data["x"], centroid_data["y"], centroid_data["z"])
    screen_coords = world_to_screen(
        centroid, cam_position, cam_rotation, cam_hfov, cam_vfov, (img_width, img_height)
    )
    if screen_coords is None:
        continue
    draw.ellipse(
        (screen_coords[0] - 5, screen_coords[1] - 5, screen_coords[0] + 5, screen_coords[1] + 5),
        fill="red",
        outline="black"
    )