Semantic segmentation is an important task in computer vision to help machines gain a high-level understanding of the environment, similar to the human vision system. For example it is used in self-driving cars which are equipped with various sensors such as cameras and 3D laser scanners to gain a complete understanding of their environment. In recent years the field has been dominated by Deep Neural Networks (DNNs), which are notorious for requiring large amounts of training data. Creating these datasets is very time consuming and costly. Moreover, the datasets can only be applied to a specific type of sensor. The present work addresses this problem. It will be shown that knowledge from publicly available image datasets can be reused to minimize the labeling costs for 3D point clouds. For this purpose, the labels from classified images are transferred to 3D point clouds. To bridge the gap between sensor modalities, the geometric relationship of the sensors in a fully calibrated system is used. Due to various errors the naive label transfer can lead to a significant amount of incorrect class label assignments in 3D. Within the work the different reasons and possible solutions are shown in order to improve the label transfer. First, Scanstrip Network (SNet) is presented. The network learns to correct wrong class assignments in 3D point clouds and implicitly considers different sources of errors. It is trained in a supervised manner and only on a small amount of data. The simple but effective network design achieves an mean Intersection over Union (mIoU) of 0.67 as opposed to the baseline value of 0.48, outperforming similar and even state-of-the-art networks. These results are further improved by training SNet in a semi-supervised manner. For this, large amounts of automatically generated labels are used for pretraining, allowing the network to achieve a mIoU of 0.71. One problem at the beginning of the label transfer is classification errors in images and wrong 2D pixels to 3D point assignments. To address this, Multi-View Network (MVNet) is introduced. This network learns to relate multi-view 2D predictions for single 3D points. The network is able to reduce classification errors in 2D with very little training data and outperforms other semi-supervised methods. By combining SNet and MVNet into Label Transfer Network (LTNet), the complete label transfer from 2D to 3D can be learned. LTNet works in both domains simultaneously and achieves a mIoU of 0.75 in 3D, which outperforms all previous models. Moreover it is shown that it is possible to handle dynamic occlusions and self-occlusions in 3D through a self-supervised manner, i.e. without ground truth. Dynamic occlusions occur when moving objects appear in one domain but not in the other leading to incorrect object assignments. Here a Conditional Generative Adversarial Network (CGAN) is introduced that learns to map from 3D point clouds to 2D photorealistic images. Since the synthesized images and the point clouds match very well, this approach leads to much better results when mapping image labels belonging to dynamic classes such as cars to 3D point clouds. For self occlusions a GAN is introduced that learns to complete a range of 3D objects from incomplete observations only. The results show that the GAN performs almost similar to semi-supervised or fully-supervised methods, which helps in identifying occupied regions in 3D and could potentially lead to fewer errors in the label transfer process.
展开▼
