TOP SCEINCE
Researchers leverage shadows to model 3D scenes, including objects blocked from view
Imagine driving through a tunnel in an autonomous vehicle, but unbeknownst to you, a crash has stopped traffic up ahead. Normally, you’d need to rely on the car in front of you to know you should start braking. But what if your vehicle could see around the car ahead and apply the brakes even sooner?
They have introduced a method that creates physically accurate, 3D models of an entire scene, including areas blocked from view, using images from a single camera position. Their technique uses shadows to determine what lies in obstructed portions of the scene.
They call their approach PlatoNeRF, based on Plato’s allegory of the cave, a passage from the Greek philosopher’s “Republic”in which prisoners chained in a cave discern the reality of the outside world based on shadows cast on the cave wall.
By combining lidar (light detection and ranging) technology with machine learning, PlatoNeRF can generate more accurate reconstructions of 3D geometry than some existing AI techniques. Additionally, PlatoNeRF is better at smoothly reconstructing scenes where shadows are hard to see, such as those with high ambient light or dark backgrounds.
In addition to improving the safety of autonomous vehicles, PlatoNeRF could make AR/VR headsets more efficient by enabling a user to model the geometry of a room without the need to walk around taking measurements. It could also help warehouse robots find items in cluttered environments faster.
“Our key idea was taking these two things that have been done in different disciplines before and pulling them together — multibounce lidar and machine learning. It turns out that when you bring these two together, that is when you find a lot of new opportunities to explore and get the best of both worlds,” says Tzofi Klinghoffer, an MIT graduate student in media arts and sciences, affiliate of the MIT Media Lab, and lead author of a paper on PlatoNeRF.
Klinghoffer wrote the paper with his advisor, Ramesh Raskar, associate professor of media arts and sciences and leader of the Camera Culture Group at MIT; senior author Rakesh Ranjan, a director of AI research at Meta Reality Labs; as well as Siddharth Somasundaram at MIT, and Xiaoyu Xiang, Yuchen Fan, and Christian Richardt at Meta. The research will be presented at the Conference on Computer Vision and Pattern Recognition.
Shedding light on the problem
Reconstructing a full 3D scene from one camera viewpoint is a complex problem.
Some machine-learning approaches employ generative AI models that try to guess what lies in the occluded regions, but these models can hallucinate objects that aren’t really there. Other approaches attempt to infer the shapes of hidden objects using shadows in a color image, but these methods can struggle when shadows are hard to see.
For PlatoNeRF, the MIT researchers built off these approaches using a new sensing modality called single-photon lidar. Lidars map a 3D scene by emitting pulses of light and measuring the time it takes that light to bounce back to the sensor. Because single-photon lidars can detect individual photons, they provide higher-resolution data.
The researchers use a single-photon lidar to illuminate a target point in the scene. Some light bounces off that point and returns directly to the sensor. However, most of the light scatters and bounces off other objects before returning to the sensor. PlatoNeRF relies on these second bounces of light.
By calculating how long it takes light to bounce twice and then return to the lidar sensor, PlatoNeRF captures additional information about the scene, including depth. The second bounce of light also contains information about shadows.
The system traces the secondary rays of light — those that bounce off the target point to other points in the scene — to determine which points lie in shadow (due to an absence of light). Based on the location of these shadows, PlatoNeRF can infer the geometry of hidden objects.
The lidar sequentially illuminates 16 points, capturing multiple images that are used to reconstruct the entire 3D scene.
“Every time we illuminate a point in the scene, we are creating new shadows. Because we have all these different illumination sources, we have a lot of light rays shooting around, so we are carving out the region that is occluded and lies beyond the visible eye,” Klinghoffer says.
A winning combination
Key to PlatoNeRF is the combination of multibounce lidar with a special type of machine-learning model known as a neural radiance field (NeRF). A NeRF encodes the geometry of a scene into the weights of a neural network, which gives the model a strong ability to interpolate, or estimate, novel views of a scene.
This ability to interpolate also leads to highly accurate scene reconstructions when combined with multibounce lidar, Klinghoffer says.
“The biggest challenge was figuring out how to combine these two things. We really had to think about the physics of how light is transporting with multibounce lidar and how to model that with machine learning,” he says.
They compared PlatoNeRF to two common alternative methods, one that only uses lidar and the other that only uses a NeRF with a color image.
They found that their method was able to outperform both techniques, especially when the lidar sensor had lower resolution. This would make their approach more practical to deploy in the real world, where lower resolution sensors are common in commercial devices.
“About 15 years ago, our group invented the first camera to ‘see’ around corners, that works by exploiting multiple bounces of light, or ‘echoes of light.’ Those techniques used special lasers and sensors, and used three bounces of light. Since then, lidar technology has become more mainstream, that led to our research on cameras that can see through fog. This new work uses only two bounces of light, which means the signal to noise ratio is very high, and 3D reconstruction quality is impressive,” Raskar says.
In the future, the researchers want to try tracking more than two bounces of light to see how that could improve scene reconstructions. In addition, they are interested in applying more deep learning techniques and combining PlatoNeRF with color image measurements to capture texture information.
Further information: https://openaccess.thecvf.com/content/CVPR2024/html/Klinghoffer_PlatoNeRF_3D_Reconstruction_in_Platos_Cave_via_Single-View_Two-Bounce_Lidar_CVPR_2024_paper.html
TOP SCEINCE
Early dark energy could resolve cosmology’s two biggest puzzles
A new study by MIT physicists proposes that a mysterious force known as early dark energy could solve two of the biggest puzzles in cosmology and fill in some major gaps in our understanding of how the early universe evolved.
Now, the MIT team has found that both puzzles could be resolved if the early universe had one extra, fleeting ingredient: early dark energy. Dark energy is an unknown form of energy that physicists suspect is driving the expansion of the universe today. Early dark energy is a similar, hypothetical phenomenon that may have made only a brief appearance, influencing the expansion of the universe in its first moments before disappearing entirely.
Some physicists have suspected that early dark energy could be the key to solving the Hubble tension, as the mysterious force could accelerate the early expansion of the universe by an amount that would resolve the measurement mismatch.
The MIT researchers have now found that early dark energy could also explain the baffling number of bright galaxies that astronomers have observed in the early universe. In their new study, reported in the Monthly Notices of the Royal Astronomical Society, the team modeled the formation of galaxies in the universe’s first few hundred million years. When they incorporated a dark energy component only in that earliest sliver of time, they found the number of galaxies that arose from the primordial environment bloomed to fit astronomers’ observations.
“You have these two looming open-ended puzzles,” says study co-author Rohan Naidu, a postdoc in MIT’s Kavli Institute for Astrophysics and Space Research. “We find that in fact, early dark energy is a very elegant and sparse solution to two of the most pressing problems in cosmology.”
The study’s co-authors include lead author and Kavli postdoc Xuejian (Jacob) Shen, and MIT professor of physics Mark Vogelsberger, along with Michael Boylan-Kolchin at the University of Texas at Austin, and Sandro Tacchella at the University of Cambridge.
Big city lights
Based on standard cosmological and galaxy formation models, the universe should have taken its time spinning up the first galaxies. It would have taken billions of years for primordial gas to coalesce into galaxies as large and bright as the Milky Way.
But in 2023, NASA’s James Webb Space Telescope (JWST) made a startling observation. With an ability to peer farther back in time than any observatory to date, the telescope uncovered a surprising number of bright galaxies as large as the modern Milky Way within the first 500 million years, when the universe was just 3 percent of its current age.
“The bright galaxies that JWST saw would be like seeing a clustering of lights around big cities, whereas theory predicts something like the light around more rural settings like Yellowstone National Park,” Shen says. “And we don’t expect that clustering of light so early on.”
For physicists, the observations imply that there is either something fundamentally wrong with the physics underlying the models or a missing ingredient in the early universe that scientists have not accounted for. The MIT team explored the possibility of the latter, and whether the missing ingredient might be early dark energy.
Physicists have proposed that early dark energy is a sort of antigravitational force that is turned on only at very early times. This force would counteract gravity’s inward pull and accelerate the early expansion of the universe, in a way that would resolve the mismatch in measurements. Early dark energy, therefore, is considered the most likely solution to the Hubble tension.
Galaxy skeleton
The MIT team explored whether early dark energy could also be the key to explaining the unexpected population of large, bright galaxies detected by JWST. In their new study, the physicists considered how early dark energy might affect the early structure of the universe that gave rise to the first galaxies. They focused on the formation of dark matter halos — regions of space where gravity happens to be stronger, and where matter begins to accumulate.
“We believe that dark matter halos are the invisible skeleton of the universe,” Shen explains. “Dark matter structures form first, and then galaxies form within these structures. So, we expect the number of bright galaxies should be proportional to the number of big dark matter halos.”
The team developed an empirical framework for early galaxy formation, which predicts the number, luminosity, and size of galaxies that should form in the early universe, given some measures of “cosmological parameters.” Cosmological parameters are the basic ingredients, or mathematical terms, that describe the evolution of the universe.
Physicists have determined that there are at least six main cosmological parameters, one of which is the Hubble constant — a term that describes the universe’s rate of expansion. Other parameters describe density fluctuations in the primordial soup, immediately after the Big Bang, from which dark matter halos eventually form.
The MIT team reasoned that if early dark energy affects the universe’s early expansion rate, in a way that resolves the Hubble tension, then it could affect the balance of the other cosmological parameters, in a way that might increase the number of bright galaxies that appear at early times. To test their theory, they incorporated a model of early dark energy (the same one that happens to resolve the Hubble tension) into an empirical galaxy formation framework to see how the earliest dark matter structures evolve and give rise to the first galaxies.
“What we show is, the skeletal structure of the early universe is altered in a subtle way where the amplitude of fluctuations goes up, and you get bigger halos, and brighter galaxies that are in place at earlier times, more so than in our more vanilla models,” Naidu says. “It means things were more abundant, and more clustered in the early universe.”
“A priori, I would not have expected the abundance of JWST’s early bright galaxies to have anything to do with early dark energy, but their observation that EDE pushes cosmological parameters in a direction that boosts the early-galaxy abundance is interesting,” says Marc Kamionkowski, professor of theoretical physics at Johns Hopkins University, who was not involved with the study. “I think more work will need to be done to establish a link between early galaxies and EDE, but regardless of how things turn out, it’s a clever — and hopefully ultimately fruitful — thing to try.”
“We demonstrated the potential of early dark energy as a unified solution to the two major issues faced by cosmology. This might be an evidence for its existence if the observational findings of JWST get further consolidated,” Vogelsberger concludes. “In the future, we can incorporate this into large cosmological simulations to see what detailed predictions we get.”
This research was supported, in part, by NASA and the National Science Foundation.
TOP SCEINCE
Plant-derived secondary organic aerosols can act as mediators of plant-plant interactions
A new study published in Science reveals that plant-derived secondary organic aerosols (SOAs) can act as mediators of plant-plant interactions. This research was conducted through the cooperation of chemical ecologists, plant ecophysiologists and atmospheric physicists at the University of Eastern Finland.
The study showed that Scots pine seedlings, when damaged by large pine weevils, release VOCs that activate defences in nearby plants of the same species. Interestingly, the biological activity persisted after VOCs were oxidized to form SOAs. The results indicated that the elemental composition and quantity of SOAs likely determines their biological functions.
“A key novelty of the study is the finding that plants adopt subtly different defence strategies when receiving signals as VOCs or as SOAs, yet they exhibit similar degrees of resistance to herbivore feeding,” said Professor James Blande, head of the Environmental Ecology Research Group. This observation opens up the possibility that plants have sophisticated sensing systems that enable them to tailor their defences to information derived from different types of chemical cue.
“Considering the formation rate of SOAs from their precursor VOCs, their longer lifetime compared to VOCs, and the atmospheric air mass transport, we expect that the ecologically effective distance for interactions mediated by SOAs is longer than that for plant interactions mediated by VOCs,” said Professor Annele Virtanen, head of the Aerosol Physics Research Group. This could be interpreted as plants being able to detect cues representing close versus distant threats from herbivores.
The study is expected to open up a whole new complex research area to environmental ecologists and their collaborators, which could lead to new insights on the chemical cues structuring interactions between plants.
TOP SCEINCE
Folded or cut, this lithium-sulfur battery keeps going
Most rechargeable batteries that power portable devices, such as toys, handheld vacuums and e-bikes, use lithium-ion technology. But these batteries can have short lifetimes and may catch fire when damaged. To address stability and safety issues, researchers reporting in ACS Energy Letters have designed a lithium-sulfur (Li-S) battery that features an improved iron sulfide cathode. One prototype remains highly stable over 300 charge-discharge cycles, and another provides power even after being folded or cut.
The team coated iron sulfide cathodes in different polymers and found in initial electrochemical performance tests that polyacrylic acid (PAA) performed best, retaining the electrode’s discharge capacity after 300 charge-discharge cycles. Next, the researchers incorporated a PAA-coated iron sulfide cathode into a prototype battery design, which also included a carbonate-based electrolyte, a lithium metal foil as an ion source, and a graphite-based anode. They produced and then tested both pouch cell and coin cell battery prototypes.
After more than 100 charge-discharge cycles, Wang and colleagues observed no substantial capacity decay in the pouch cell. Additional experiments showed that the pouch cell still worked after being folded and cut in half. The coin cell retained 72% of its capacity after 300 charge-discharge cycles. They next applied the polymer coating to cathodes made from other metals, creating lithium-molybdenum and lithium-vanadium batteries. These cells also had stable capacity over 300 charge-discharge cycles. Overall, the results indicate that coated cathodes could produce not only safer Li-S batteries with long lifespans, but also efficient batteries with other metal sulfides, according to Wang’s team.
The authors acknowledge funding from the National Natural Science Foundation of China; the Natural Science Foundation of Sichuan, China; and the Beijing National Laboratory for Condensed Matter Physics.
-
Solar Energy3 years ago
DLR testing the use of molten salt in a solar power plant in Portugal
-
Camera1 year ago
DJI Air 3 vs. Mini 4 Pro: which compact drone is best?
-
world news1 year ago
Gulf, France aid Gaza, Russia evacuates citizens
-
Camera1 year ago
Sony a9 III: what you need to know
-
world news1 year ago
Strong majority of Americans support Israel-Hamas hostage deal
-
Camera4 years ago
Charles ‘Chuck’ Geschke, co-founder of Adobe and inventor of the PDF, dies at 81
-
Solar Energy1 year ago
Glencore eyes options on battery recycling project
-
Indian Defense3 years ago
Israeli Radar Company Signs MoU To Cooperate With India’s Alpha Design Technologies