Photogrammetry in VFX: How to Create Hyper-Realistic 3D Models
Share
Photogrammetry in VFX: How to Create Hyper-Realistic 3D Models
Introduction
Photogrammetry is revolutionizing VFX and CGI production, allowing artists to create highly detailed 3D models from real-world objects, environments, and textures. Used in films, video games, and virtual production, photogrammetry provides a fast, accurate, and cost-effective way to generate photo-realistic assets.
From hyper-detailed digital humans in The Matrix Resurrections to scanned environments in The Mandalorian, photogrammetry is now a key tool in modern VFX workflows. In this article, we’ll explore how photogrammetry works, the best tools available, and how VFX artists use it to create hyper-realistic assets.
What Is Photogrammetry in VFX?
Photogrammetry is the process of capturing real-world objects and environments using multiple photographs, which are then processed into a high-resolution 3D model with accurate textures.
Why Is Photogrammetry Important in VFX?
✅ Saves Time – Automates 3D modeling by generating geometry from real-world photos.
✅ Increases Realism – Captures real surface imperfections, lighting, and fine details.
✅ Enhances CGI Integration – Ensures photo-realistic digital assets for films, games, and VR.
✅ Scales for Large Environments – Can reconstruct entire landscapes, buildings, and cities.
🔹 Example:
In Blade Runner 2049, photogrammetry was used to digitally recreate city environments, ensuring that every building and surface had a realistic level of detail.
How Photogrammetry Works - Step by Step
1. Capturing High-Resolution Photos
To create a high-quality 3D scan, artists take hundreds of overlapping photos of an object from multiple angles.
🔹 Best Practices for Photogrammetry Capturing:
✅ Use a DSLR or high-resolution camera for detailed texture capture.
✅ Ensure consistent lighting to avoid shadows or reflections.
✅ Capture at least 70-80% overlap between images for better alignment.
✅ Shoot in RAW format for maximum image fidelity.
🔹 Example:
In The Mandalorian, photogrammetry was used to capture real-world desert landscapes, which were then turned into digital environments for virtual production.
2. Image Processing & 3D Reconstruction
Once photos are captured, they are processed in photogrammetry software, which aligns the images, reconstructs depth, and generates a high-resolution 3D mesh.
🔹 Key Steps in Image Processing:
✅ Feature Matching – Software detects common points between images.
✅ Depth Calculation – Uses stereo vision to estimate 3D depth and geometry.
✅ Mesh Generation – Converts point cloud data into a 3D surface.
✅ Texture Projection – High-resolution textures are applied to the mesh.
🔹 Example:
For The Batman (2022), photogrammetry was used to capture Gotham City’s realistic streets, props, and architectural details, making digital scenes look indistinguishable from reality.
3. Cleanup & Optimization for VFX Pipelines
After processing, the raw 3D model needs to be optimized for VFX or real-time rendering.
🔹 Best Practices for 3D Optimization:
✅ Retopology – Convert high-poly scans into clean, animation-ready geometry.
✅ Texture Baking – Transfer high-detail textures to low-poly models for efficiency.
✅ Normal & Displacement Maps – Preserve fine details without increasing geometry.
✅ Export in VFX-Friendly Formats – Use OBJ, FBX, or USD for compatibility with Maya, Houdini, Unreal Engine, and Nuke.
🔹 Example:
In Jurassic World, photogrammetry was used to scan real dinosaur fossils, helping VFX artists create scientifically accurate CGI dinosaurs.
Best Photogrammetry Software for VFX Artists
| Software | Purpose |
|---|---|
| RealityCapture | Fast, high-quality 3D scanning for VFX and game development. |
| Agisoft Metashape | Professional photogrammetry tool with deep texture processing. |
| Meshroom (AliceVision) | Free, open-source photogrammetry software. |
| 3DF Zephyr | User-friendly 3D scanning with AI-assisted reconstruction. |
| Photoshop & Substance Painter | Used for texture cleanup and enhancement. |
🔹 Example:
For The Matrix Resurrections, RealityCapture was used to scan real actors’ faces and costumes, ensuring hyper-realistic digital doubles.
Photogrammetry in Virtual Production & Game Engines
With real-time rendering becoming more popular, photogrammetry is now used in virtual production and game engines like Unreal Engine and Unity.
🔹 How It’s Used in Virtual Production:
✅ Scanning real environments to create photo-realistic digital sets.
✅ Integrating scanned objects into LED Volume workflows (The Mandalorian).
✅ Creating game-ready assets for real-time VFX and animation.
🔹 Example:
In Ghost of Tsushima, photogrammetry was used to scan real Japanese landscapes and historical locations, creating an authentic open-world game environment.
Challenges in Photogrammetry for VFX
1. Handling Large Data Sets
📂 Problem: High-resolution scans create massive file sizes.
✅ Solution: Use LOD (Level of Detail) optimization and texture compression techniques.
2. Dealing with Reflective & Transparent Surfaces
🔍 Problem: Glass, water, and shiny surfaces don’t scan properly.
✅ Solution: Apply matte spray coatings before scanning or use LiDAR scanning as an alternative.
3. Lighting Consistency Issues
💡 Problem: Inconsistent shadows and exposure create scanning errors.
✅ Solution: Capture diffuse lighting conditions or shoot indoors with controlled lights.
🔹 Example:
For Star Wars: The Rise of Skywalker, photogrammetry was combined with LiDAR scanning to accurately reconstruct complex set pieces and alien environments.
The Future of Photogrammetry in VFX
With AI-driven scanning, real-time photogrammetry, and LiDAR integration, the future of 3D asset creation is becoming faster, more efficient, and highly detailed.
🔹 Upcoming Trends in Photogrammetry:
✅ AI-Powered Reconstruction – Uses deep learning for faster, more accurate 3D scans.
✅ Real-Time Photogrammetry for Virtual Production – Allows instant environment capture for LED walls.
✅ Hybrid LiDAR & Photogrammetry Workflows – Combines laser scanning for precision and photogrammetry for textures.
🔹 Example:
NASA is developing AI-enhanced photogrammetry techniques for scanning Mars terrains, which could one day be used in sci-fi films and space simulations.
Conclusion
Photogrammetry is a game-changing tool in VFX, allowing artists to create hyper-realistic 3D models from real-world references. Whether used for environment scanning, character modeling, or virtual production, it is redefining how CGI assets are created.
With AI-powered processing, real-time scanning, and LiDAR integration, photogrammetry is becoming faster, more accessible, and more powerful than ever.
🚀 Want to explore photogrammetry for VFX? Start experimenting with RealityCapture, Agisoft Metashape, and Unreal Engine today!
References
- The VES Handbook of Visual Effects – Jeffrey A. Okun & Susan Zwerman.
- RealityCapture Documentation – High-End 3D Scanning for Film & Games.
- SIGGRAPH 2023 – AI-Powered Photogrammetry & Real-Time Environment Capture.
- Behind the Scenes – The Making of The Mandalorian: Photogrammetry & Virtual Production.
- Pixar’s RenderMan Blog – Photogrammetry in High-End Animation & VFX.