Peter Beerel & Gourav Datta
In the year 2023, cameras have become an integral part of our lives – whether it’s the camera on your smartphone used for a social media selfie or a camera employed in a facial recognition system that allows you to enter a secure building.
Researchers at the USC Viterbi School of Engineering are now taking camera advancements a step further and pushing the limits on what they can do by integrating machine learning.
“We are developing smarter cameras by putting machine learning intelligence right next to the camera pixels,” said Peter Beerel, USC Viterbi professor in the Ming Hsieh Department of Electrical and Computer Engineering – Systems.
“This intelligence enables the cameras to more efficiently perform computer vision tasks such as recognizing and tracking objects for a wide variety of applications including face detection, autonomous driving and wildfire prevention.”
Peter Beerel, Gourav Datta, a USC Viterbi Ph.D. candidate in electrical engineering, and Souvik Kundu, a former USC Viterbi Ph.D. student have partnered with the USC Information Science Institute’s Ajey P. Jacob and Akhilesh R. Jaiswal to conduct this research funded by a $1 million DARPA (Defense Advanced Research Projects Agency) grant. While their main goal is to advance smart camera technology, they are also doing it in an energy-efficient way.
By taking the machine learning circuitry and embedding it inside the camera itself, the camera consumes less power.
“Otherwise, you’re going to basically take the entire image and send it somewhere else to a different chip just to process it,” Beerel explained. “The images are quite big these days — we are talking about megapixels. Our method is more energy efficient because it is processed locally, so less data must be sent elsewhere. Our research is aimed at doing more and more on the device itself; the more you can do closer to the sensor, the more energy efficient it can be.”
Datta says as machine learning becomes more common across the world, the need to start utilizing power efficient solutions increases. “If we consume less power to achieve the same goals, we are saving energy that otherwise affects, say, climate change. It may be small, but these add up when you are talking about millions of devices,” said Datta.
Many smart camera devices currently rely on batteries, but the aim of this new research is to reduce energy while extending battery life. Another function of this smart camera technology is to ensure safety and security.
“It captures the image of the person, and this machine learning system will detect if this is the correct person, or an intruder, and know whether to unlock a door or not,” said Datta.
Datta led the publication of four peer-reviewed articles on smart cameras in prestigious venues, such as Nature Scientific Reports, Winter Conference on Applications of Computer Vision (WACV), and European Conference on Computer Vision (ECCV).
Beerel and his team are now extending this research by working with a world leader in camera technology. They were recently selected to receive a $150,000 grant from Samsung University to expand the function of cameras.
Their work with Samsung will further the advancement of “self-healing cameras.”
For example, let’s say you take a picture, and a portion of the image is distorted. Machine learning can fill in gaps with the correct information to make the picture appear normal.
“Our idea is that we’ll use machine learning, where it’s able to gather more information about the objects, then it should be able to do a better job of filling in these gaps,” said Beerel. “The camera pixels can go bad, and if enough goes bad, that chip can’t be sold. Machine learning can be used to compensate for the bad pixels.”
Beerel explained how they used hardware and software and played in the boundary between the two. “We codesign the machine learning model and the circuitry and this gives us a whole new angle.”
“This solution requires a deep understanding of the hardware from manufacturing to transistors, as well as algorithms and such. As electrical engineers, we understand this, and it’s been nice to play that middle role because it makes more of a significant impact,” said Beerel.
Beerel encourages students to continue to uncover new truths about their studies. “I want to empower my students to find new research areas, solve new problems, and figure out how to fund it.”
Published on March 28th, 2023
Last updated on March 28th, 2023
