headshot of Matthias Nießner from the shoulders up, wearing a light indigo collared shirt and smiling

Speaker: Matthias Nießner

Date: Monday, July 11th, 2022

Time: 11:00 am - 12:00 pm

Location: Zoom only

Zoom link is on our Events Calendar

Host: Tobias Höllerer

Title: Neural Surface Reconstruction: To Learn or Not to Learn?


In the recent years, we have seen tremendous progress on leveraging neural networks – in particular, for reconstructing 3D surfaces, neural networks have become a necessary tool in order to achieve state-of-the-art performance. While we often see networks used to learn generalizable discriminative features (e.g., establishing feature matches for structure-from-motion), MLP-based representations have recently gained significant popularity as data structures substituting explicit representations such as voxel grids. Importantly, these representations can also be leveraged as parametric models, for instance, to regularize underconstrained reconstruction problems. Overall, the question is: when do we need to use networks for learning, and when should they be used for storing data and surface fitting?

In this talk, I will give an overview of our latest works in this area, starting with neural networks as feed forward predictors and generative models for surface reconstruction. I discuss how to learn parametric models that can model dynamic scene reconstructions, e.g., for humans or arbitrary deformable surfaces. Finally, I will discuss when to use networks as model representations -- when to simply leverage them for fitting problems, and when generalization and feature learning is required to achieve cutting-edge reconstruction results.


Dr. Matthias Nießner is a Professor at the Technical University of Munich, where he leads the Visual Computing Lab. Before, he was a Visiting Assistant Professor at Stanford University. Prof. Nießner’s research lies at the intersection of computer vision, graphics, and machine learning, where he is particularly interested in cutting-edge techniques for 3D reconstruction, semantic 3D scene understanding, video editing, and AI-driven video synthesis. In total, he has published over 150 academic publications, including 25 papers at the prestigious ACM Transactions on Graphics (SIGGRAPH / SIGGRAPH Asia) journal and 55 works at the leading vision conferences (CVPR, ECCV, ICCV); several of these works won best paper awards, including at SIGCHI’14, HPG’15, SPG’18, and the SIGGRAPH’16 Emerging Technologies Award for the best Live Demo. Prof. Nießner’s work enjoys wide media coverage, with many articles featured in main-stream media including the New York Times, Wall Street Journal, Spiegel, MIT Technological Review, and many more, and his was work led to several TV appearances such as on Jimmy Kimmel Live, where Prof. Nießner demonstrated the popular Face2Face technique; Prof. Nießner’s academic Youtube channel currently has over 5 million views. For his work, Prof. Nießner received several awards: he is a TUM-IAS Rudolph Moessbauer Fellow (2017 – ongoing), he won the Google Faculty Award for Machine Perception (2017), the Nvidia Professor Partnership Award (2018), as well as the prestigious ERC Starting Grant 2018 which comes with 1.5 million Euro in research funding; in 2019, he received the Eurographics Young Researcher Award honoring the best upcoming graphics researcher in Europe. In addition to his academic impact, Prof. Nießner is a co-founder and director of Synthesia Inc., a startup dedicated to democratizing synthetic media generation with cutting-edge AI-driven video synthesis technology.