D²iCE at the Electronic Imaging conference 2025
Posted on April 7, 2025
Posted on April 7, 2025
San Francisco Airport Hyatt Regency Hotel (top left) was the main location of Electronic Imaging 2025, Alcatraz island (top right), Golden Gate Bridge (bottom left) and the California Academy of Sciences (bottom right).
On the 1st of February 2025, five members of D²iCE Research Centre, Prof. Patrick Denny, Seamie Hayes, Aryan Singh, Sanjay Kumar and Daniel Jakab attended the 37th Electronic Imaging conference organized by the Society for Imaging Science and Technology (IS&T). We are very grateful for the D²iCE Research Centre, Science Foundation Ireland, Lero, the ECE Department and the Faculty of Science and Engineering, University of Limerick for making this trip possible.
Daniel Jakab had the fantastic opportunity of presenting a paper entitled, “SOLAS: Superpositioning an Optical Lens in Automotive Simulation”. This work demonstrates the use of ray-tracing as a method for optical simulations in the context of automotive simulation utilizing the Python-based ray-tracing library, KrakenOS a relatively new tool developed by Joel Herrera Vázquez and his research team at the Institute of Astronomy, National Autonomous University of Mexico (UNAM). Please see the following link to the published paper here. Special thanks to Julian Barthel and Alexander Braun from Düsseldorf University of Applied Sciences for making the optical simulations from KrakenOS visually possible by applying them directly on automotive simulations. Special shout out to Daniel Jakab’s supportive supervisors, fellow researchers and colleagues from D²iCE Research Centre, University of Galway and Valeo Vision Systems: Ciarán Eising, Reenu Mohandas, Dara Molloy, Mahendar Kumbham, PhD, Brian Deegan, Fiachra Collins and Tony Scanlan who have all helped to make this publication possible.
Automotive Simulation is a potentially cost-effective strategy to identify and test corner case scenarios in automotive perception. Recent work has shown a significant shift in creating realistic synthetic data for road traffic scenarios using a video graphics engine. However, a gap exists in modeling realistic optical aberrations associated with cameras in automotive simulation. This paper builds on the concept from existing literature to model optical degradations in simulated environments using the Python-based ray-tracing library KrakenOS. As a novel pipeline, we degrade automotive fisheye simulation using an optical doublet with +/-2◦ Field of View(FOV), introducing realistic optical artifacts into two simulation images from SynWoodscape and Parallel Domain Woodscape. We evaluate KrakenOS by calculating the Root Mean Square Error (RMSE), which averaged around 0.023 across the RGB light spectrum compared to Ansys Zemax OpticStudio, an industrial benchmark for optical design and simulation. Lastly, we measure the image sharpness of the degraded simulation using the ISO12233:2023 Slanted Edge Method and show how both qualitative and measured results indicate the extent of the spatial variation in image sharpness from the periphery to the center of the degradations.
Daniel Jakab, member of D2iCE presenting on Monday 3rd February.
During this conference, on the 5th of February, Daniel Jakab had an incredible opportunity to attend two Short Courses offered by Electronic Imaging. The first course was called “Image Quality Foundations: Standards for Mobile, Automotive, and Machine Vision Applications” presented by Dr. Peter D. Burns, Managing Director of Burns Digital Imaging (formed in 2011), and Don Williams, Consultant of Image Science Associates. This was the second time Daniel Jakab was able to attend this session following on from last year’s Electronic Imaging 2024 and as a returning scholar, Daniel Jakab highly recommends this course where each year it is different and just as good (if not better!) than the last. This course is extremely beneficial for both researchers and industrial experts alike interested in the field of Image Quality. The course material is designed to be inclusive where discussion is encouraged, providing an excellent learning environment for all. Please do see a previous blog post here by Daniel Jakab who recounts a day-to-day diary at Electronic Imaging.
The second course was called “Information Metrics for Optimizing Machine Vision Systems” presented by Norman Koren (the Founder of Imatest LLC, a leader in image quality testing since 2004). This short course is designed to be a perfect follow-on course to the previous where Information Metrics linking Image Quality and Computer Vision performance is the end goal. A question that is perhaps on everyone’s mind, is how do we design our cameras and imaging systems such that we know how our Computer Vision and Artificial Intelligence algorithms behave? It is a compelling thought, and the truth is we don’t know until we do the measurements. But even before this, we must first devise the metrics to take credible measurements. And this is exactly what this course delivers. Not only are the mathematical foundations of new metrics presented but, a live demonstration is provided to all using new Imatest software called Simatest, an Image Signal Processing (ISP), and camera simulator. With a course as advanced and forward-thinking as this, you would not want to miss this next year.
Overall, if Electronic Imaging were to be described in a particular way, the phrase ‘once-in-a-lifetime experience’ would come to mind. Any imaging or data scientist would easily find their place here no matter what their background is whether it is medical imaging, automotive, astronomy, mobile, or any other visual domain for that matter. At this stage, if Electronic Imaging isn’t already on your calendar for next year, then…make sure it is!
Seamie Hayes presenting his work on Tuesday 4th February.
Title: Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2
Abstract: Birds Eye View perception models require extensive data to perform and generalize effectively. While traditional datasets often provide abundant driving scenes from diverse locations, this is not always the case. It is crucial to maximize the utility of the available training data. With the advent of large foundation models such as DINOv2 and Metric3Dv2, a pertinent question arises: can these models be integrated into existing model architectures to not only reduce the required training data but surpass the performance of current models? We choose two model architectures in the vehicle segmentation domain to alter: Lift-Splat-Shoot, and Simple-BEV. For Lift-Splat-Shoot, we explore the implementation of frozen DINOv2 for feature extraction and Metric3Dv2 for depth estimation, where we greatly exceed the baseline results by 7.4 IoU while utilizing only half the training data and iterations. Furthermore, we introduce an innovative application of Metric3Dv2's depth information as a PseudoLiDAR point cloud incorporated into the Simple-BEV architecture, replacing traditional LiDAR. This integration results in a +3 IoU improvement compared to the Camera-only model.
On 4th February 2025, Seamie Hayes on behalf of me, presented my paper.
Title: Minimizing Occlusion Effect on Multi-view Camera Perception in BEV with Multi-sensor Fusion
Abstract: Autonomous driving technology is rapidly evolving, offering the potential for safer and more efficient transportation. However, the performance of these systems can be significantly compromised by the occlusion on sensors due to environmental factors like dirt, dust, rain, and fog. These occlusions severely affect vision-based tasks such as object detection, vehicle segmentation, and lane recognition. In this paper, we investigate the impact of various kinds of occlusions on camera sensor by projecting their effects from multi-view camera images of the nuScenes dataset into the Bird’s-Eye View (BEV) domain. This approach allows us to analyze how occlusions spatially distribute and influence vehicle segmentation accuracy within the BEV domain. Despite significant advances in sensor technology and multi-sensor fusion, a gap remains in the existing literature regarding the specific effects of camera occlusions on BEV-based perception systems. To address this gap, we use a multi-sensor fusion technique that integrates LiDAR and radar sensor data to mitigate the performance degradation caused by occluded cameras. Our findings demonstrate that this approach significantly enhances the accuracy and robustness of vehicle segmentation tasks, leading to more reliable autonomous driving systems.
Seamie Hayes presenting on behalf of Sanjay Kumar on Tuesday 4th February.
Title: Image Segmentation: Inducing graph-based learning
EI conference was a great experience, I had the opportunity to present my work and interact with some very interesting people at the venue, explored the city, and it was very much an enjoyable experience. Our presented study explores the potential of graph neural networks (GNNs) to enhance semantic segmentation across diverse image modalities. We evaluate the effectiveness of a novel GNN based U-Net architecture on three distinct datasets: PascalVOC, a standard benchmark for natural image segmentation, WoodScape, a challenging dataset of fisheye images commonly used in autonomous driving, introducing significant geometric distortions; and ISIC2016, a dataset of dermoscopic images for skin lesion segmentation. We compare our proposed UNet-GNN model against established convolutional neural networks (CNNs) based segmentation models, including U-Net and U-Net++, as well as the transformer-based SwinUNet. Unlike these methods, which primarily rely on local convolutional operations or global selfattention, GNNs explicitly model relationships between image regions by constructing and operating on a graph representation of the image features. This approach allows the model to capture long-range dependencies and complex spatial relationships, which we hypothesize will be particularly beneficial for handling geometric distortions present in fisheye imagery and capturing intricate boundaries in medical images. Our analysis demonstrates the versatility of GNNs in addressing diverse segmentation challenges and highlights their potential to improve segmentation accuracy in various applications, including autonomous driving and medical image analysis.