Singapore Management University's School of Computing and Information Systems (SCIS) has made significant strides in the field of artificial intelligence and computer vision with a groundbreaking publication titled 'Portrait Shadow Removal via Self-Exemplar Illumination Equalization'. This innovative work, presented at the prestigious 32nd ACM International Conference on Multimedia (ACM MM '24) in Melbourne, Australia, introduces a novel method that tackles one of the most persistent challenges in image processing: removing shadows from portrait photographs. Shadows in portraits, often cast by uneven lighting or facial features, can distort facial details, making post-processing essential for photographers, social media users, and professionals in fields like forensics and e-commerce.
The paper, authored by Qian Huang and Cheng Xu from SMU alongside collaborators from South China University of Technology, highlights the university's growing prowess in developing practical AI solutions for real-world problems. By leveraging self-exemplars—well-lit regions within the same image as reference points—the method achieves superior illumination equalization, producing natural-looking results without artifacts common in traditional techniques.
The Challenge of Shadows in Portrait Photography
Shadows in portraits arise from complex interactions between light sources, subject pose, and environment. Traditional shadow removal methods rely on paired shadow-shadow-free datasets or physical priors like lighting models, but these often fail on portraits due to intricate facial structures and soft shadows. Self-shadows from noses or chins are particularly tricky, as they blend seamlessly with skin tones.
Prior approaches, such as generative adversarial networks (GANs) or attention-based models, struggle with color consistency and texture preservation. SMU's contribution addresses this by focusing exclusively on portraits, recognizing their unique characteristics like high-frequency details in hair and eyes.
Self-Exemplar Illumination Equalization: The Core Innovation
At the heart of the breakthrough is the Self-Exemplar Illumination Equalization Network (SEIEN). This deep learning model operates in a bilinear framework, decomposing the image into shadow and non-shadow components while using the image itself as an exemplar for relighting.
- Shadow Detection Module: Identifies shadowed regions using multi-scale feature fusion, capturing both local edges and global context.
- Exemplar Mining: Automatically selects well-lit facial parts (e.g., forehead, cheeks) as self-exemplars, avoiding external data dependency.
- Illumination Equalization: Transfers illumination statistics from exemplars to shadowed areas via adaptive normalization, preserving identity and texture.
- Refinement Network: Post-processes for seamless blending, minimizing halo effects.
This step-by-step process ensures photorealistic outputs, outperforming baselines by significant margins in metrics like PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index).
Performance on Benchmark Datasets
The method was rigorously tested on two public portrait shadow datasets: ShadowPortrait and PortraitShadow. Quantitative results show improvements of up to 2.5 dB in PSNR and 0.05 in SSIM compared to state-of-the-art models like DC-ShadowNet and ShadowFormer. Qualitatively, portraits exhibit natural skin tones and sharp features without over-smoothing or color shifts.
| Method | PSNR (ShadowPortrait) | SSIM (ShadowPortrait) |
|---|---|---|
| Baseline (e.g., ShadowFormer) | 28.5 | 0.92 |
| SEIEN (SMU) | 31.2 | 0.97 |
Ablation studies confirm the self-exemplar component's critical role, boosting performance by 15% when integrated.
The Research Team Behind the Breakthrough
Lead authors Qian Huang and Cheng Xu, PhD candidates at SMU SCIS, worked under Associate Professor Shengfeng He, a renowned expert in computer vision. He, who joined SMU in 2023, has a track record of pioneering shadow removal techniques, including Mask-ShadowNet (IEEE SPL 2021) and DeshadowNet (CVPR 2017). His Google Scholar h-index reflects over 300 publications and collaborations with top labs.
Collaborators from South China University of Technology brought expertise in graphics and optimization, blending Singapore's AI ecosystem with mainland China's computational resources.
Photo by Tarik Haiga on Unsplash
Shengfeng He's Vision for Image Processing at SMU
Assoc Prof Shengfeng He's career trajectory—from PhD at City University of Hong Kong to faculty at SMU—positions him as a bridge between academia and industry. Awarded Stanford's Top 2% Scientists (2024), Google's Research Award, and multiple CCF-Tencent honors, his focus on low-level vision tasks like de-raining, denoising, and now de-shadowing addresses practical pain points in photography and augmented reality.
At SMU, He mentors students on deployable AI, emphasizing Singapore's role as an AI hub. His lab's work aligns with national initiatives like AI Singapore, fostering talent for tech giants like Grab and Sea.
SMU SCIS: Pioneering AI and Vision Research in Singapore
SMU's SCIS, established to drive digital transformation, excels in AI, data science, and cybersecurity. Ranked highly in QS for business analytics, SCIS produces graduates for Singapore's Smart Nation vision. Recent feats include AI for cybersecurity and multimodal learning, with the shadow removal paper exemplifying interdisciplinary impact.
The school's emphasis on real-world projects, partnerships with Alibaba Cloud and NVIDIA, equips researchers for global challenges. Shadow removal enhances applications in portrait enhancement apps, vital for Singapore's vibrant social media culture and e-commerce sector.
Real-World Applications and Industry Relevance
Beyond academia, the technique promises transformations in:
- Consumer apps like photo editors (e.g., Lightroom plugins).
- Professional photography and videography.
- Forensics for accurate facial recognition.
- AR/VR for seamless virtual makeup trials.
- E-commerce product shots with consistent lighting.
In Singapore, where AI adoption is high, this could boost startups in beauty tech and media, aligning with the $1B AI investment plan.
Implications for Singapore's AI Landscape
Singapore aims to be a global AI leader, with SMU contributing through talent and IP. This publication underscores SCIS's role in low-level vision, complementing high-level models like GPT. It attracts collaborations, funding from NRF, and positions graduates for roles at GovTech or private firms.
Challenges like dataset scarcity are met with self-supervised learning, reducing reliance on labeled data—a boon for resource-constrained environments.
Future Directions and Open Challenges
The authors plan extensions to video portraits, multi-light sources, and integration with diffusion models for generative editing. Real-time mobile deployment via lightweight networks is next. Broader impacts include ethical AI for bias-free facial processing.
SMU envisions scaling to national datasets, partnering with A*STAR for commercialization.
Photo by Woliul Hasan on Unsplash
Why This Matters for Higher Education in Singapore
In Singapore's competitive higher ed landscape, SMU SCIS exemplifies research-driven education. Publications like this enhance rankings, attract top students, and drive economic value. For aspiring researchers, it highlights interdisciplinary paths in AI vision.
As Singapore invests in AI, breakthroughs like this solidify its position, inspiring the next generation.