This essay appeared in the 2021 Fueling Discovery special section of the Wisconsin State Journal on 10/17/21. View the full 2021 Fueling Discovery PDF here.
About the author
Mohit Gupta is an assistant professor of computer sciences. He directs the WISIONLab with research interests in computer vision and computational imaging. His recent work in computational single-photon imaging is opening up new possibilities for computer vision systems that work in extremely low light, with applications in robotics, scientific imaging and human-computer interfaces.
We have witnessed a remarkable evolution of camera technology over the past century. Perhaps one of the most profound shifts is that these days, photographs are increasingly created by algorithms, with the role of cameras being merely to capture light. The “raw light capture” is algorithmically and computationally transformed in multiple ways before a photograph is created and consumed. Most modern cameras can thus be thought of as “computational cameras” — they are routinely able to create high-quality photographs even with low-cost devices including cell phones, thus commoditizing a powerful form of visual expression.
Beyond capturing aesthetically pleasing images, such cameras are playing an instrumental role in augmented reality, digital communication, medicine, scientific imaging, and robotics. Robots enabled by such computational cameras are now beginning to autonomously drive cars, explore space, and manage our factories, marking a radical shift in how we go about our daily lives.
With the most common cameras today, such as the one in your cell phone, one can think of each pixel as a light bucket which collects photons. For such cameras, each pixel typically requires capturing hundreds to thousands of photons per pixel to create a reasonable image.
But what if there are cameras that could record individual photons?
There is an emerging class of single-photon cameras that promise such single-photon sensitivity. Single-photon computational cameras capture images that are algorithmically manipulated at the granularity of individual photons (the minimum amount of measurable light).
For such single-photon cameras, each pixel is like a teaspoon that fills up as soon as it detects a photon. Due to their high sensitivity and (relatively) low costs, single-photon cameras are driving an imaging revolution.
A new generation of devices is emerging, with novel functionalities that were hitherto considered impossible: imaging at a trillion frames-per-second, being able to image around corners, and capturing high-quality images in almost complete darkness. Imagine a camera mounted on a telescope being able to locate a dim astronomical object even in a distant galaxy, or a high-performance scientific camera tracking high-speed cell deformation for facilitating cancer research. Although these single-photon cameras were long relegated to niche scientific applications, recently they have started making their way to consumer domains (including the new Apple iPhone), placing this exciting technology at the fingertips of billions of people worldwide. Werner Heisenberg, the noted physicist, once said: “What we observe is not nature itself, but nature exposed to our method of questioning.”
By zooming in to individual photons, single-photon cameras allow us to ask some of the most fundamental visual questions from nature, thus getting us closer to its most intimate truths.
On the one hand, the realization that next-generation cameras and algorithms can work synergistically to generate new kinds of photographs, functionalities and visual experiences has profound positive implications — including for photography, robotics, computer vision, medical imaging and AI. On the other hand, this also raises questions about the nature of reality. Can we trust what we see?
It is becoming easier to create photographs and videos of people and events that never were. These include fabricated videos of political speeches, falsified photographic evidence in legal cases, pornography, and manipulated news reel images. Such photos and videos can potentially be weaponized, posing a serious societal risk.
Following these examples, it is easy to picture computational cameras as dangerous devices. However, as scholar Issac Asimov said, “I could not bring myself to believe that if knowledge presented danger, the solution was ignorance. To me, it always seemed that the solution had to be wisdom. You did not refuse to look at danger, rather you learned how to handle it safely. Any technological advance can be dangerous.
Fire was dangerous from the start, and so was speech —and both are still dangerous to this day —but human beings would not be human without them.”
As with any technology, we need the necessary checks and balances in place so that cameras are used responsibly. If we can do this successfully, this new breed of computationally enhanced super-cameras will enable us to see the wonderful world around us in a new light.