Joint learning method for semantic segmentation and depth estimation in driving scenes based on Transformer cross-modal feature alignment

Vehicle-mounted visual perception requires simultaneous acquisition of road target category, boundary structure and spatial distance information. To improve the semantic segmentation accuracy, depth estimation quality and on-vehicle reasoning efficiency in complex driving scenarios, a hardware-aware hierarchical window encoder, a dynamic road-region sparse attention and a boundary-constrained multi-scale decoder are constructed, combined with a cross-modal semantic-depth alignment and a depth-range adaptive refinement, to ultimately form a driving scene joint learning model based on Transformer cross-modal feature alignment. Experimental results show that the precision rate, recall rate and F1 value of the proposed model reach 92.71%, 91.64% and 92.17%, and the semantic-depth alignment similarity of vehicles, pedestrians and road areas reaches 89.7%, 94.2% and 89.3% respectively. In addition, the depth estimation accuracy of this model under medium-close, medium-range and long-range conditions is 97.63%, 95.28% and 92.47%, corresponding to an execution time of 29.36-31.27 ms. The proposed model can effectively improve the performance of semantic segmentation, depth estimation and real-time warning of driving scenes, and provide new technical ideas for car following control, collision warning and path planning.