Bifrost metadata¶
When downloading a collection, a COCO JSON file is included in the ZIP archive. In addition to the standard COCO fields, the file would also contain metadata specific to Bifrost renders.
This custom metadata is located under the images and annotations sections of the JSON, within a field named bifrost_metadata.
Example format
{
"images": [
{
"id": 1,
"file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_0.jpg",
"bifrost_metadata": {
"bifrost_camera_position": {
"x": 0,
"y": 0,
"z": 1000
},
"bifrost_camera_rotation": {
"roll": 0,
"pitch": -30,
"yaw": 0
},
"bifrost_camera_heading": 0.0,
"bifrost_camera_fov": 90.0
}
}
],
"annotations": [
{
"id": 1,
"image_id": 1,
"bifrost_metadata": {
"bifrost_object_id": "boat_1",
"bifrost_heading_absolute_degrees": 90.02760314941406,
"bifrost_distance_to_camera_m": 34.114948730468754,
"bifrost_absolute_position_m": {
"x": 32.7561962890625,
"y": 0.5690269851684571,
"z": 0.4848830032348633
},
"bifrost_absolute_rotation_degrees": {
"roll": -0.8629000186920166,
"pitch": 90.02760314941406,
"yaw": -1.8286000490188599
},
"bifrost_absolute_centroid_position_m": {
"x": 33.5,
"y": 1.2,
"z": 5.8
},
"bifrost_absolute_size_m": {
"x": 20.299549560546875,
"y": 21.7643212890625,
"z": 65.81230957031251
},
"bifrost_absolute_cuboid_positions_m": [
{"x": 23.15, "y": -9.68, "z": -27.12},
{"x": 43.85, "y": -9.68, "z": -27.12},
{"x": 23.15, "y": 12.08, "z": -27.12},
{"x": 43.85, "y": 12.08, "z": -27.12},
{"x": 23.15, "y": -9.68, "z": 38.69},
{"x": 43.85, "y": -9.68, "z": 38.69},
{"x": 23.15, "y": 12.08, "z": 38.69},
{"x": 43.85, "y": 12.08, "z": 38.69}
],
"bifrost_scale": {
"x": 1,
"y": 1,
"z": 1
}
}
}
]
}
Image level metadata¶
By default, the following information is exported under the images.bifrost_metadata field:
Field Name |
Unit |
Description |
|---|---|---|
|
Meters |
Camera position in Bifrost world coordinates. |
|
Degrees |
Camera rotation relative to Bifrost world. |
|
Degrees |
Camera heading relative to Bifrost world. |
|
Degrees |
Horizontal field of view of the camera. |
Object level metadata¶
Note
Object level metadata is only available for assets added to the World via
world.add.
Objects such as Sky, Land, and Water would not have any associated metadata.
By default, the following information is exported under the annotations.bifrost_metadata field:
*If you require a different method of distance calculation, please let us know!
Adding custom metadata¶
Note
Custom metadata can currently only be added at the image level.
Annotation-level custom metadata is not supported at this time.
Add custom metadata for each frame using
scenario.set_metadata_keyframe.
scenario.set_metadata_keyframe(<frame_index>, <key>, <value>)
An important behavior to note is that metadata keys set in a previous frame will automatically persist
into subsequent frames unless explicitly updated. To remove a key from a specific frame, set its value
to METADATA_UNSET_KEY.
Here is a full example of how to add custom metadata
from bbi import *
world = World(...)
scenario = world.new_scenario(3)
# Frame 0
scenario.set_metadata_keyframe(0, "weather", "rainy")
scenario.set_metadata_keyframe(0, "sun", {"elevation": 30, "azimuth": 180})
scenario.set_metadata_keyframe(0, "special_key", 1)
# Frame 1
scenario.set_metadata_keyframe(1, "weather", "sunny")
scenario.set_metadata_keyframe(1, "special_key", METADATA_UNSET_KEY)
# Frame 2
scenario.set_metadata_keyframe(2, "weather", "cloudy")
scenario.render()
Example output
{
"images": [
{
"id": 1,
"file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_0.jpg",
"bifrost_metadata": {
"weather": "rainy",
"sun": {
"elevation": 30,
"azimuth": 180
},
"special_key": 1,
}
},
{
"id": 2,
"file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_1.jpg",
"bifrost_metadata": {
"weather": "sunny",
"sun": {
"elevation": 30,
"azimuth": 180
},
}
},
{
"id": 3,
"file_name": "e8c419d6-e9d8-49e0-ad43-161aadc4b9c9/RGB_camera_2.jpg",
"bifrost_metadata": {
"weather": "cloudy",
"sun": {
"elevation": 30,
"azimuth": 180
},
}
}
],
}
Frequently asked questions¶
You can download the helper functions used in this section here.
Note
These helper functions depend on numpy and scipy, which are not included in the Python standard library. Install them with:
pip install numpy scipy
All examples below refer to the following image and its associated COCO annotation file:
Image:
Are the object level metadata relative to the camera or the world?¶
All object-level metadata are absolute, meaning they are defined in the world coordinate system, not relative to the camera.
For more information on the coordinate system, see Coordinate system.
Can I convert the absolute coordinates to be relative to the camera?¶
Yes! Absolute coordinates can be converted to be relative to any object in the world, including the camera.
To do this:
Retrieve the object’s
bifrost_absolute_position_mandbifrost_absolute_rotation_degrees.Create a
Transformobject (from the helper functions file) using those values.Use the
Transform.relative_to()method to convert its absolute transform into the camera’s local coordinate space.
Example: get relative transform and plot relative yaw with respect to camera
import json
from PIL import Image, ImageDraw, ImageFont
coco = json.load(open("coco.json"))
# get image info
image_info = coco["images"][0]
img_id = image_info["id"]
img_width, img_height = image_info["width"], image_info["height"]
position_data = image_info["bifrost_metadata"]["bifrost_camera_position_m"]
cam_position = Vector3(position_data["x"], position_data["y"], position_data["z"])
rotation_data = image_info["bifrost_metadata"]["bifrost_camera_rotation_degrees"]
cam_rotation = Vector3(rotation_data["roll"], rotation_data["pitch"], rotation_data["yaw"])
cam_transform = Transform(cam_position, cam_rotation)
# get annotations of objects that are in frame
annotations = [
ann for ann in coco["annotations"] if ann["image_id"] == img_id
and ann.get("bifrost_metadata", {}).get("bifrost_in_frame")
]
img = Image.open("RGB_camera_0000.jpg")
draw = ImageDraw.Draw(img)
for ann in annotations:
# get relative transform
ann_transform = Transform(
position=Vector3(**ann["bifrost_metadata"]["bifrost_absolute_position_m"]),
rotation=Vector3(*ann["bifrost_metadata"]["bifrost_absolute_rotation_degrees"].values()),
)
relative_transform = ann_transform.relative_to(cam_transform)
# write relative yaw
bbox = ann["bbox"]
center = (bbox[0] + bbox[2] / 2, bbox[1] + bbox[3] / 2)
font = ImageFont.truetype("arial.ttf", 24)
text = f"{relative_transform.rotation.z:.2f}°"
text_bbox = draw.textbbox(center, text, font=font)
bg_box = (text_bbox[0]-5, text_bbox[1]-5, text_bbox[2]+5, text_bbox[3]+5)
draw.rectangle(bg_box, fill="white")
draw.text(center, text, fill=(200, 0, 0), font=font)
How do I convert from world coordinates to screen coordinates?¶
Use the world_to_screen function from the helper functions file to convert a 3D world-space point into 2D screen-space coordinates.
There are two important cases to be aware of:
The function returns
Nonewhen the point lies behind the camera.The function returns coordinates outside the image bounds when the point is in front of the camera but outside the camera’s field of view.
Example: convert world coordinates to screen coordinates and plot objects’ centroids
import json
from PIL import Image, ImageDraw
coco = json.load(open("coco.json"))
# get image info
image_info = coco["images"][0]
img_id = image_info["id"]
img_width, img_height = image_info["width"], image_info["height"]
position_data = image_info["bifrost_metadata"]["bifrost_camera_position_m"]
cam_position = Vector3(position_data["x"], position_data["y"], position_data["z"])
rotation_data = image_info["bifrost_metadata"]["bifrost_camera_rotation_degrees"]
cam_rotation = Vector3(rotation_data["roll"], rotation_data["pitch"], rotation_data["yaw"])
cam_hfov = image_info["bifrost_metadata"]["bifrost_camera_fov_degrees"]
cam_vfov = calculate_vfov(cam_hfov, (img_width, img_height))
# get annotations of objects that are in frame
annotations = [
ann for ann in coco["annotations"] if ann["image_id"] == img_id
and ann.get("bifrost_metadata", {}).get("bifrost_in_frame")
]
img = Image.open("RGB_camera_0000.jpg")
draw = ImageDraw.Draw(img)
for ann in annotations:
centroid_data = ann["bifrost_metadata"]["bifrost_absolute_centroid_position_m"]
centroid = Vector3(centroid_data["x"], centroid_data["y"], centroid_data["z"])
screen_coords = world_to_screen(
centroid, cam_position, cam_rotation, cam_hfov, cam_vfov, (img_width, img_height)
)
if screen_coords is None:
continue
draw.ellipse(
(screen_coords[0] - 5, screen_coords[1] - 5, screen_coords[0] + 5, screen_coords[1] + 5),
fill="red",
outline="black"
)

