Project Snake Eyes: Automatic Feature Extraction Based on Thermal Vision
I am pleased to announce Project Snake Eyes. This ambitious project aims to combine image analysis and infrared imaging to create biomimetic thermal vision — computer vision that simulates the ability of some animals such as snakes to see in darkness. One of the goals of Project Snake Eyes is to create probably the world’s first robotic snake that can “hunt” through thermal sensing. Project Snake Eyes not only can detect heat, but it can also estimate the size and proximity of the source, giving the robotic snake the artificial intelligence to figure out whether or not it should strike the target. To accomplish this goal, I have devised some algorithms that allow robust automatic feature extraction in real time from thermal images through a FLIR ONE camera. This article reports my progress thus far. The algorithms have been implemented in the SmartIR app that I am developing.
Figure 1a shows the detection of windows at night from outside a house. Half of the window was open at the time of observation. The lower temperature of the upper pane was partially due to the reflection of infrared light from the cooler environment such as the sky by the glass. The lower pane, which was the open part of the window, was due to the warmer inside of the house. As you can see, my algorithm approximately identified the shapes and sizes of the two panes (represented by the translucent overlays), which stood out from the background because of their distinct temperatures. This ability will be used to develop advanced algorithms for automatically analyzing the thermal signature of a building, paving the road to large-scale building thermal analyses through automation. Figure 1b shows a more complex scenario using my computer desk as an example. As you can see, the algorithm automatically detected most of the hot spots (or cold spots, but I use hot spots to represent points of interest at temperatures higher or lower than the ambient temperature). Feature extraction such as blob detection through thermal vision could result in enhanced computer vision for technologies such as autonomous vehicles. Figure 1c shows object reconstruction based on residue heat using my hand as an example. The residue heat that my hand left on the wall revealed its shape as six separate polygons (the largest one corresponds to the palm and the five smaller ones to the fingertips). Object reconstruction through residue heat could find its applications in certain industry monitoring and control. Figure 1d shows object detection and tracking using a colleague as an example. As you can see, the algorithm basically captured his body shape. For real-time tracking, I still need to improve the algorithm to reduce the lag, but the result looks promising so far. Object tracking will be used to realize animal and human tracking in dark conditions for many science and engineering applications.
There are three different criteria for objection detection in thermal computer vision. The screenshot on the left of Figure 2 shows that it is able to approximately recognize the shapes of a dish of warm water and a dish of cool water and then calculate the average surface temperatures of the identified objects (which are presented by the translucent polygons). I call this surface detection. The screenshot in the middle of Figure 2 shows that SmartIR can detect the areas at the highest or lowest temperatures of an object as well. I call this spot detection. The screenshot on the right of Figure 2 shows that SmartIR is able to single out a human body in darkness and reconstruct the silhouette. I call this shape detection.
The algorithm tends to work better when there is a large temperature difference between the subject and the surround (I call this factor thermal contrast), as shown in Figure 3.
I expect to carry on the research and development of Project Snake Eyes in the coming years. Stay tuned!
