Toby McGuire
Resume
Projects/Underwater computer vision

Case study · ML / AI · Simulation

Teaching a robot to see in murky water.

I worked on autonomy and perception for OSU Underwater Robotics, where the biggest challenge was that our cameras can barely see underwater. So I built a Blender project that would generate synthetic datapoints of our target objects in obscured/blurry situations to train our models on.

Team
OSU Underwater Robotics
Role
Perception & Data
Stack
Blender · YOLOv5 · Python
When
2020 – 2024

Finding training data on short notice

Each new competition had new goal objects to detect, meaning we had a limited amount of time to gather data and train out YOLOv5 perception model on them. When we could, we would miss rainy day cases where the objects are obscured or the visibility is bad.

This lead to our vision system working really well in clear pools and above water, but heavily lacking in real world bodies of water with sand and silt.

So I went to work building a system to generate fake datapoints that simulated these conditions using the following systems:

  • Fog/Haze Using depth nodes and tweaking a threshold value, I could adjust the post processing of an image to add more or less haze to an image to obscure an object
  • Debris I randomly scattered objects around the scene to partially obscure the goal object, verifying that at least 20% of the object was visible.
  • Collecting the datapoint With blender, I was able to seperatly output a black/white mask of the key object ontop of the scene, and get bounding box data writted to a log file.

This system was randomized for each datapoint generated, and cound generate about 5 new datapoints a second on a medium-high end PC in 2022.

Blender to Perception

This breaks down the workflow of generating new datapoints for an object

Select Choose a goal object to generate data for
Randomize Randomly choose goal object pos, hazard objects, and level of fog.
Record Export raw image, object mask, and bounding box
Train Use collection of datapoints to train YOLOv5 perception model

Generating a datapoint

Here's a simplified version of the generator. Each click places the goal object at a random position and rotation, scatters debris around it, and picks a random haze level. Next to the image you get the object mask and the bounding box that the training labels come from.

Debris in front of the object can hide parts of it, so the mask only covers what's still visible and the bounding box is fit to that. If less than 20% of the object would be visible, the debris get randomized again. Best practice in real life is to avoid using too many large debris to make this unlikely.

Synthetic datapoint generator
Image
A generated underwater image containing the goal object.
MaskObject only
The black and white mask of the goal object.
Haze—
Object visible—

Taking the same idea into industry

At Nauticus Robotics I took this solution further and had more compute at my disposal to generate these datapoints even faster, with more variations on the goal object to increase variability in the data, and higher quality renders to be more true to life.

At this time, YOLOv8 came out with the ability to add orientation to detections, which only required small changes to my generator to get even higher quality datapoints.

3,000+ synthetic labeled training images per hour
+30% object confidence in low visibility environments