Introduction

Dense depth prediction is widely utilized in such areas as scene understanding [1, 2], autonomous driving [3, 4], robotics [5], AR/VR [6], IoT [7]. Usually, applications in these areas have strong limitations on hardware capabilities. However, the impressive progress made so far in the domain of monocular [8-11] and binocular [12, 13] depth estimation comes at the expense of a significant increase in the complexity, which typically demands considerable computational and memory resources.

The efficiency of Deep Neural Networks (DNNs) inference on low-end devices is achieved by using low-precision computations [14, 15] and low-bit representation of DNN weights and activations. Considerable research has been devoted towards reducing quantization error. However, the problem of data precision loss is largely unexplored, presumably because it does not pose a problem for models predicting RGB images, which are naturally presented as three eight-bit channels. In contrast, depth maps require at least ten or eleven bit for accurate representation (representation of range 0.5…10 m with 1cm accuracy requires ten bits). Consequently, low-end devices with low-precision computations necessarily lead to on-device depth quality loss. Solution to this problem can be in usage on mixed-precision quantization. Mixed-precision quantization is less power efficient as compared to uniform quantization and not always supported on low-end devices.

In the Visual Intelligence Team of SRUKR, we developed solution that allows using efficient uniform low-bit quantization of depth prediction DNNs and reconstruct higher bit-precision depth [16]. To overcome low-precision arithmetic limitation we use depth representation as points on a 2D Hilbert curve. This transform codes high dynamic range depth as two low dynamic range Hilbert curve components that can be represented in low-bit precision with minimum quality loss. Our method adds a computationally simple on-device post-processing step that converts low-bit precision Hilbert components to higher precision depth.

In this blog post, we explain our solution including idea of depth representation in the form of two components of Hilbert curve, full precision model modification, training process modification, on-device inference and obtained performance gain.

Proposed approach

Let us consider a DNN that predicts disparity (or depth) $d$ normalized to the range [0, 1]. A quantized model runs on a low-end device that has an efficient DNN inferencing module (e.g. NPU or DSP) with $b$-bit output data representation and a general-purpose CPU with full-precision arithmetic. We seek to increase the bit-width of the predicted value of $d$ with minimum computational overhead.

Due to DSP limitations, the bit-width of the model output cannot be increased, so we can operate only with the number and structure of the output channels. These channels should be transferred from DSP to CPU and used to reconstruct disparity in higher bit precision. To make this scheme effective, the amount of data transferred from DSP to CPU and complexity of post-processing run on CPU should be minimized.

Our idea is to represent disparity $d$ as a point on a 2D parametric curve $(x(d), y(d))$ with length $L>1$, where both $x$ and $y$ are bounded in range [0, 1] (Figure 2). The full-precision DNN is trained to directly predict $x(d)$ and $y(d)$. When running inference on DSP, the value of d is calculated from $x$ and $y$ values predicted in $b$-bit precision. The parametric curve of length $L$ will pass through approximately $L·2^b$ discreet points $(x(d), y(d))$ effectively increasing precision of reconstructed value of $d$ by $log_2L$ bits. In this implementation, the amount of data to transfer from DSP to CPU increases only twofold, post-processing on CPU is simple and implemented using LUT.

On-device deployment of the modified depth prediction DNN consists of the following steps: (a) training the full-precision model to directly predict the Hilbert curve components of depth representation, (b) applying standard quantization methods (either PTQ or QAT; QAT should be applied at the training stage), (c) running inference of the quantized model on-device and obtaining Hilbert curve components in low-bit precision, (d) applying post-processing to Hilbert curve components and reconstruct depth in higher-bit precision.

Selection of the parametric curve is crucial for successful implementation of the proposed idea. We selected low-order Hilbert curve [18, 19] which is continuous fractal space-filling curve constructed as a limit of piece-wise linear curves. It possesses the following properties: continuity, non-self-intersection, boundedness and self-avoidance enabling efficient training of the full precision model and following quantization process. Curve length is $Lp=(2^p+1)(1+2k)$ , where p is the curve order, $k=0.1$ is scaling factor to avoid boundary effects. Curves of the second order with $L_2=4$ and third order with $L_3=7.2$ allows increasing bit precision by 2-3 bits. In this case, predicting Hilbert curve components in eight bit precision will provide necessary 10-11 bits for the final disparity/depth map.

To implement the proposed approach, a DNN with one head predicting disparity should be modified to have two heads predicting Hilbert curve components $x$ and $y$. Training is performed with a modified loss function that includes original disparity/depth loss and additional terms applied directly to the Hilbert curve components.

Experimental results

We validated our method on the stereo matching task. Training dataset is based on ScanNet v2 [1]. The DispNet [17] and Dense Prediction Transformer (DPT) [8] models are selected for the experiments. Modified models with third order Hilbert curve ($L=7.3$) dubbed as h3DispNet and h3DPT. All models were quantized to INT8 precision using SNPE SDK v.2.24 [20] and tested on Samsung S24 device. We characterize the quality of predicted depth maps using mean absolute relative error (Abs Rel) [21] and cosine similarity [22] between discrete cosine transform (DCT) [23] coefficients of ground truth and predicted depth maps, $S_c$. The measure $S_c$ takes values less than 1, with 1 corresponding to the perfect match between GT and predicted depth maps. Experimental results show that $S_c$ is sensitive to bit-precision loss. For the full precision and quantized models we confirmed that training the modified model that predicts Hilbert components is possible without quality degradation (See Figure 7).

Quantitative results are presented in Table 1. For both DispNet and DPT models, quantization leads to quality degradation on DSP. The modified DispNet model in W8A8 format shows better quality than the original model in W8A16 format while simultaneously reducing energy consumption by 35% and latency by 30%. Compared to the original model in W8A16 format, the modified DPT model in W8A8 format shows significantly better Abs Rel metric and only slightly worse Sc. It also lowers energy demands by 34% and processing time by 54%. Examples for predicted depth maps and quantization error reduction are shown in Fig. 8.

We evaluate our approach on KITTI 2012 [24] dataset. KITTI 2012 is a real-world dataset for autonomous driving domain with sparse ground-truth disparities collected with a LiDAR system. Our evaluation is based on 194 images in the training part of KITTI 2012. Training dataset is composed of ScanNet and Virtual KITTI 2 [25] datasets with 25/75% balancing. The DispNet and h2DispNet models were trained in 256 by 1152 px input resolution and 128 by 576 px output disparity resolution. Training settings are the same as for the ScanNet experiment.

For the original DispNet model we reached EPE 1.38 px and D1 9.13%. Interestingly, h2DispNet shows better results with EPE 1.07 px and D1 5.22%. The DispNet model quantized to W8A8 format and run on DSP shows degradation of both EPE (to 1.63 px) and D1 (to 9.95%). At the same time the h2DispNet in W8A8 format on DSP shows very minor quality degradation (EPE 1.09 px and D1 5.36%). Measured as EPE between disparity predicted by FP32 and W8A8 models, quantization error is 1.01 px for the DispNet and 0.38 px for the h2DispNet. It corresponds to gain in disparity prediction quality on device of 2.6 times. Overall, on DSP the h2DispNet improves D1 by approximately 4.6% and EPE by about 33% compared to the original DispNet model. Examples of predicted disparity are given in Fig. 9. This experiment demonstrates that the proposed approach can be successfully applied to real datasets.

Conclusions

We proposed a novel method for high dynamic range depth prediction on devices with low-precision arithmetic that exploits depth representation as points on a 2D Hilbert curve. Experiments demonstrate that for the stereo matching task our method reconstructs depth in 10-11 bits for a model in W8A8 format and with quality similar to or even better than the original model in W8A16 format. In this manner, depth can be predicted on-device 1.4-2 times faster and with 65% of power consumption without sacrificing its quality. The proposed approach is beneficial for on-device application of different dense depth prediction methods including monocular and stereo depth prediction, Multi-View-Stereo, depth completion, depth quality enhancement, and depth inpainting. It can also be extended to other domains such as HDR images prediction on Human Pose Estimation with heatmaps.

Link to the paper

View More

Close