Picture Imperfect

Giryes’s work is broadly applicable to all manner of circumstances where datasets are sketchy or incomplete, solving problems not only for still images but also for video and even medical systems that determine which human cells are cancerous and which are healthy. In one side project, he and his colleagues trained a neural network to accurately assign archaeological artifacts to the correct location and period. Using a publicly available photo database from the Israel Antiquities Authority, the AI was fed a massive amount of information on everything from the Lower Paleolithic period (which started about 1.4 million years ago) to the Late Islamic period (14th century). Using a neural network, the AI extracted features known as “embeddings” from the images of the artifacts. A computer then translated the colours, shapes and other aspects of the artifacts’ appearence into a series of numbers that can be compared from artifact to artifact, eliminating the subjectivity that human brains may be prone to when analyzing a piece of pottery.

Archaeology is a particularly difficult field because two temporally or spatially adjacent societies might have tools or weapons that look remarkably similar. In tests, Giryes’s algorithm handily beat two trained archaeologists, correctly identifying artifacts with much greater accuracy. This technique is valuable in illuminating new avenues of study for finding meaningful relationships between ancient sites — and much more.

“Getting lots of examples requires lots of effort,” says Giryes. “The question for us is do we need to collect all the data every time, or can we develop a technique that would achieve the same result in a faster way?”

Although Giryes’s work with large datasets and deep learning has a wide range of applications, his main focus is computational imaging and designing new optical systems using artificial intelligence. The practical outcomes of this work include everything from improving picture quality to image recognition, training a computer to identify real-world images from tiny fragments.

Representation of three-dimensional data presents a particularly thorny problem that his lab is tackling. “You take a photo with your phone,” Giryes says, “and you want it to be a 3D photo, but you’re only using a single camera. We’re figuring out how to design the optics of the camera to get better depth estimation.” The easiest way to make a three-dimensional image is by combining regular images from different angles. But computers building models of lifelike objects don’t always have the luxury of data inputs from all directions. Instead, Giryes developed an AI technology that teaches a camera (by automatically designing its optical element) how to get 3D information from a single direction. In addition to capturing 3D data, Giryes also studies how to better represent it. His team uses “implicit representation” to accurately depict or manipulate objects with realistic depth.

We’re all familiar with pixels, the smallest element in a digital display. An alternative approach uses “voxels,” the 3D equivalent. Think of a three-dimensional picture of a stack of blocks; each voxel is one block. (Etymology is helpful here: the word pixel derives from “picture” plus “element,” whereas voxel comes from “volume” plus “element.”) Using neural networks, computers assemble these voxels into images of, say, an airplane and have an understanding not just of what an airplane would look like head-on but from all sides. Although voxels are useful, they are of limited resolution, says Giryes, “but with implicit representation, you can simulate a function with any resolution you want.”

In the paper he wrote with colleagues on the subject, the technique is described as allowing for “manipulation of implicit shapes by means of transforming, interpolating and combining shape segments together without requiring explicit part supervision.” In lay terms, “imagine that you have two chairs,” says Giryes. “I can take the back of one chair and the legs of another and edit them together, mix up the parts and generate new types of shapes you’ve never seen before.”

The neural network learns to separate out one piece of the model from another, modifying individual parts without disrupting the rest of the image. By taking a three-dimensional representation of a common object and disentangling its component parts into essential geometric data, the model allows users to more easily edit these implicit shapes in high resolution. An image of a chair then ceases to be a chair, but is reduced to its most basic shape segments, which can be reassembled in multiple configurations without requiring direct supervisions of each discrete part. This technique holds promise for designers and engineers trying to imagine new products and test detailed representations of how they might perform under different stressors.

Shady Abu-Hussein, one of Giryes’s PhD students, describes how the research team has also worked to combat low resolution in images, a significant factor that reduces the usefulness of many pictures. The goal is to create a “super resolution network” that can decode the fuzz and automatically enhance the quality of what lies beneath. “These networks are usually trained for a specific lens structure,” says Abu-Hussein. “Our work means that you don’t need to train a neural network for each new camera lens from scratch. You only need to model the blurriness of the lens and then modify the already trained neural network inference steps and get great results.”

One of the most exciting real-world applications for Giryes’s brand of deep learning is in the field of self-driving cars, which must use computer vision to assess the world around them with critical levels of accuracy. Giryes consults with the Israeli company Innoviz to develop guidance and obstacle-detection systems for autonomous vehicles. Innoviz — which recently signed a $4 billion USD deal with Volkswagen — develops LIDAR (laser imaging, detection and ranging), a technology that employs laser-light rebounds to measure distance and create 3D images of objects ahead. But even the best computerized eyes are only as good as the knowledge base they use to process their inputs.

“It costs enormous amounts of money and time to collect 10,000 hours of driving data and then have a person tell you what’s in each frame,” says Giryes. So, the goal is to leapfrog over reprogramming every possible situation a car might encounter and get it to learn from similar experiences. For example, a human driver has probably never seen a five-metre cube painted gold with blue polka dots, but we’d still automatically know to swerve if it fell off the truck in front of us. Getting autonomous systems to extrapolate from the data they already have with minimal or no adaptation is the ultimate goal.

“Making decisions using multiple sensors such as both cameras and LIDAR is important as we need to learn how to use data to be adaptive to different environments,” Giryes says. “We have a car that works well in the day, but we also need it to drive at night. It has to work as well in the city as it does in the country. It has to drive well in Europe and also drive well in India.”

In both academia and industry, Israel has a been a crucial hub in the development of deep learning techniques for processing all manner of inputs. The country is home not only to dozens of successful high-tech start-ups but also Tel Aviv University and Technion’s ground-breaking electrical engineering and computer science research groups. Giryes was in one of the latter labs while conducting his PhD research and has always tried to pursue interdisciplinarity in his work, a collaborative attitude he now extends to the many graduate and postdoctoral students he supervises.

“Raja isn’t just a great mentor and teacher,” says Abu-Hussein, “but he also likes to help out his students with the smallest details, even down to looking at the lines of code, which a lot of professors would just assign another more experienced student to do.”

That drive for interdisciplinarity carries over into Giryes’s work, as he combines deep learning techniques with advances in optical hardware. “Right now, you have engineers who design optics and engineers who design sensors, and then you have the people who program the post-processing algorithms that give you better results,” he says. “But the ultimate goal is to use both deep learning and new optics to design a totally new camera that is innovative and revolutionary.”

With so much of our lives recorded on screen, we’re going to need exactly that kind of leading-edge thinking to make the images do justice to the magnificent complexity of the real world surrounding us.