Ding-dong, there’s something fishy about this AI
N.B. For an interactive environment to play around, scroll way down.
“Hey Daryl, any recommendations on an easy-to-train computer vision model?”
It was Saturday afternoon when I received above message from Gijs, a fellow startup founder. Seems out-of-the-blue, but actually not an uncommon Saturday afternoon question working in AI. On autopilot, I immediately replied: “Teachable Machine”. This is the most user-friendly interface to train computer vision models. Seconds later I scrolled up a bit in my messages and saw the reason for the question: visdeurbel.nl (fish doorbell)
Visdeurbel.nl is a recently launched livestream that shows whether there’s fish waiting in front of a canal lock (sluis in Dutch) in the City of Utrecht. When enough people watching the livestream ring the doorbell on behalf of the fish (cause, no fish fingers), the person in charge of the lock gets notified. This person can then doublecheck and open the lock, letting the fish go with the flow to their mating grounds.
Ok, you have my attention
Like Gijs realized, this seems like a perfect use case for AI, specifically a deep learning algorithm that detects ‘fish-shaped objects’. The website gained quite a bit of Dutch media attention after the launch last week, so they have plenty of human intelligence for this object detection task. But artificial intelligence doesn’t get fed up with fads and nowadays is quite capable of object detection. So let’s see how we can automate fish detection!
Let’s not reinvent the wheel
AI object detection has been around for a while, and has been used to research a plethora of different use cases. The best part? Since the problem had become object detection, I figured: skip the training of a custom model and go straight to implementing a pre-trained model. (So no more Teachable Machine required.) The go-to method for object detection is still YOLO. It’s pre-trained on many classes (e.g. person, car, etc.).
Judging from glances at the livestream, there didn’t seem to be a whole lot happening below the surface. It consisted mostly of still images and sometimes fish. So if any object would be detected, I was pretty sure it’ll be a fish. I wasn’t expecting a whole lot else to be swimming by. So: 1. download YOLO, 2. apply to the watery dataset and 3. Bob’s your uncle.
Wrong. Should’ve known, since I also do not have any uncles named Bob. From the livestream data that Gijs retrieved, it turned out there was something fishy in the water. Or rather, there were some not-so-fishy objects floating by in the form of leaves, branches and air bubbles.
I love papers with code
We needed a purpose-trained model that detects fish specifically. Luckily, since YOLO is so widespread and re-trained to use in many different applications, I figured: Somewhere out there, someone at some point in time, had wanted to detect fish with YOLO as well. And indeed there was! I ran into this paper on fish detection with YOLO from Xu and Matzner (2018). In the paper, they did the awesome thing and linked their Github code. Their results on their data looked quite promising.
This saved hours/days otherwise spent on labeling and retraining a YOLO model on our own collected fish data. The next step was getting their library to work. For this I created a Google Colab notebook, which together with a paper and Github code is the trifecta of reproducible AI research. Google Colab offers a quick, easy and completely free way to try out deep learning models. These would otherwise require days of setting up a GPU python, TensorFlow and PyTorch environment. See below for link to the interactive Colab.
The result
In only an afternoon of tweaking, it’s already quite good at detecting fishy business. There’s also room for improvement, especially in low-light conditions. But with a few simple tweaks, it could already be implemented for visdeurbel.nl’s specific use case. If you were to combine this AI with a simple business rule like:
“If there’s more than 3 fish detected for longer than 5 seconds, then notify person in charge of watching the canal lock.”
This simple addition would immediately:
- limit the number of times the AI would cry wolf-fish
- save the lock watcher from being crowd source pranked by a horde of internet trolls (remember Boaty McBoatface?),
- help out even when humans are not watching the livestream.
The interactive Colab runs on my Github fork and can be tried out here. Give it a try even if you’re not that technical, just follow the described steps, or ask in the comments.
P.S. All fish related puns in this blog are intended. Sorry, not sorry.
