PointSemSeg+ Semantic segmentation of point clouds using Bayesian neural networks for uncertainty quantifi-cation.

With the increasing use of digital building information modeling (BIM) throughout the entire construction life cycle, the need for BIM-compliant documentation of existing structures (as-is modeling) is growing at the same time. The rapid technological developments of recent years in the field of surveying systems (e.g. laser scanners) now allow efficient and spatially high-resolution building capturing in the form of 3D point clouds. However, the derivation of geometric and semantic as-is models from the 3D point clouds (scan to BIM) is a complex, multi-step process (cf. Fig. 1) that is characterized by a high level of effort and numerous manual work steps. To increase the degree of automation in scan to BIM, machine learning techniques appear extremely promising, especially for the important step of semantic segmentation. Deep neural networks (DNN) in particular are capable of learning complex relationships from large amounts of training data and thus already achieve good results in the semantic segmentation of structural components.

However, an important information for the subsequent steps (e.g. geometry fitting) or even for the later use of the automatically derived models, is the knowledge of the quality, since the automated process never succeeds without errors. Despite great progress in recent years, modern DNN architectures are not yet able to quantify the uncertainty of their outputs with sufficient accuracy, which is a significant limitation to their usability in real-world applications. In order to increase the robustness and reliability of semantic segmentation of 3D point clouds in the field of infrastructure construction, the project specifically investigates Bayesian neural networks for uncertainty estimation.

The project is divided into 5 work packages:

[AP1] Requirements analysis, data collection and preprocessing

• Selection of representative components based on the Nibelungen Bridge Worms
• Data collection with geodetic laser scanner (terrestrial, UAV), camera
• Manual segmentation and classification of data (ground truth)

[AP2] Sensitivity analysis for the investigation of point cloud features.

• Analysis of acquired features (including coordinates, color values and radiometric information)
• Analysis of derived features (e.g., local covariance, height, texture features, etc.).

[AP3] Pre-segmentation

• Pre-segmentation with classical methods (e.g. Support Vector Machines, Random Forests etc.)

[AP4] Bayesian neural networks for semantic segmentation of point clouds.

• Extension of known network architectures with Bayesian layers
• Investigation of the influence of the different characteristics from AP2
• Reliability investigation of the estimated uncertainty

[AP5] Evaluation

• Creation of a demonstrator using only Deep Learning techniques to evaluate its suitability considering uncertainty in practice.