IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) is an annual flagship conference organized by IEEE Signal Processing Society. And ICASSP is the world’s largest and most comprehensive technical conference focused on signal processing and its applications. It offers a comprehensive technical program presenting all the latest development in research and technology in the industry that attracts thousands of professionals. In this blog series, we are introducing our research papers at the ICASSP 2025 and here is a list of them. #1. Evaluation of Wearable Head BCG for PTT Measurement in Blood Pressure Intervention (Samsung Reseach America) #2. Better Exploiting Spatial Separability in Multichannel Speech Enhancement with an Align-and-Filter Network (AI Center - Mountain View) #3. Vision-Language Model Guided Semi-supervised Learning for No-Reference Video Quality Assessment (Samsung R&D Institute India-Bangalore) #4. Text-aware adapter for few-shot keyword spotting (AI Center - Seoul) #5. Single-Channel Distance-Based Source Separation for Mobile GPU in Outdoor and Indoor Environments (AI Center - Seoul) #6. Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning (Samsung R&D Institute United Kingdom) #7. Find Details in Long Videos: Tower-of-Thoughts and Self-Retrieval Augmented Generation for Video Understanding (Samsung R&D Institute China-Beijing) #8. Globally Normalizing the Transducer for Streaming Speech Recognition (AI Center - Cambridge)

Introduction

The widespread adoption of mobile devices has led to a rapid growth of video content that is captured, transmitted and shared on various social media platforms. Various camera product manufactures and social media platforms use Video Quality Assessment (VQA) metrics to monitor and control the quality of user-generated videos for a better user experience. The lack of a reference/ distortion-free videos in such applications require the design of No-Reference (NR)-VQA metric.

Existing deep learning-based algorithms demand a large amount of human-annotated video for learning a robust NR-VQA metric. However, the user-generated videos suffer from a wide range of distortions such as spatial & motion blur, compression, flicker, stabilization, poor object detailing, semantic artefacts, etc. In addition, due to the ever-evolving camera-pipeline and novel generative artefacts created by new-age generative models, it is difficult to collect and manually annotate such a wide variety of distorted videos.

In this blog post, we bring to you a semi-supervised learning based NR-VQA metric that learns from a small number of human-annotated & unlabelled videos. This algorithm could be finetuned for any application-specific use-case using a handful of relevant annotated videos. Towards this, we propose a quality-aware semantic-aware VQA (QASA-VQA) that incorporates both low-level quality distortions e.g. blur, compression, noise and high-level semantic artefacts.

Overview of Quality-Aware Semantic-Aware VQA (QASA-VQA)

QASA-VQA consists of a dual-model framework where a quality-aware model captures low-level distortions and a vision-language (VL) model captures high-level semantics.

Quality Aware Model (QAM):

To effectively capture the low-level information present in video frames, we consider fragmented samples of the original video. Fragments are generated by stitching randomly cropped patches at the original resolution of video frames to learn video quality using end-to-end in deep networks. In order to preserve temporal distortions in videos, the fragments are selected from continuous video frames. We use a feature encoder viz. spatio-temporal video quality representation learning (ST-VQRL) [1], that is pre-trained in a self-supervised manner to capture low-level video distortions. Further, we attach a regressor-head on top of ST-VQRL to predict final quality of the input video to this framework. ST-VQRL has a Video Swin transformer [2] architecture as its backbone while the regressor-head is two 3D CNN layers of 1 × 1 ×1 convolution filter size. Let, g_ϕ(⋅) and F_θ(⋅) denote the 3D regressor-head and ST-VQRL backbone, respectively.

The predicted quality of any video x∈(V_v∪U), where V_v={v_i}_(i=1)^N_l is given as:

Semantic Aware Model (SAM):

To capture semantic information in videos, we use a popular vision-language model viz. CLIP [3]. Given any video x={I_i}^N_(i=1)where I_i is the i^th frame of the video, we apply CLIP to encode the video using its visual encoder E_v. While feeding frames to CLIP, we spatially downsample the video frame using S(⋅) and feed temporally sparse frames of the videos to the CLIP visual encoder. Thus, the CLIP visual encoder has lesser sensitivity to low-level spatial distortions such as blur, noise, and compression and temporal distortions such as motion blur, camera shake, and flicker.

We choose two pairs of quality aware antonym prompts (1) [````a good photo.’’, ````a bad photo.’’] (P_0,+,P_0,-) and (2) [````a high quality photo.’’, ````a low quality photo.’’] (P_1,+,P_1,-) . Let the frozen CLIP text encoder be E_t(⋅), and the text representation be f_t^P= E_t(P). The predicted quality for any video x with CLIP visual representation f_x=1/N ∑ ^N_i=1E_v(S(I_i)) is:

where D(S(x),P_i,+,P_i,-)=f_x.f_t^P_i,+-f_x.f_t^P_i,-. E_v is trainable while E_t is frozen. And the overall quality of the video is given as (Q_L (τ' (x))+Q_H (S(x)))/2.

Training Objective:

Let V = {(v₁,y₁),(v₂,y₂),⋯ ,(v_Nl,y_Nl)} be a set of labelled (human-annotated) videos and U = {u₁,u₂,⋯ ,u_Nu} be a set of unlabelled videos. As shown in Fig. 3, we design a dual-model learning setup comprising of a low-level Quality Aware Model (QAM) and a high-level Semantic Aware Model (SAM). On the labelled set, both the models learn video quality using human opinion supervision. We learn from the unlabelled data by transferring knowledge between QAM and SAM by estimating their reliability.

Supervised Loss:

We use the differentiable Pearson’s Linear Correlation Coefficient (PLCC) loss on a mini-batch of labelled videos v={vi}Bli=1 with y={yi}Bli=1 as human-opinion score, where {(vi,yi)}Bli=1∈V and Bl as the mini-batch size. Thus, the supervised loss is formulated as,

where S(v)={S(vi)}Bli=1 and τ'(v)={τ'(vi)}Bli=1.

Intra-Model Consistency (IMC) Loss:

We evaluate the model consistency through the intra-model prediction error between the augmented video clips for each model. Since video fragments are generated by stitching random crops of patches, we randomly crop multiple local quality-consistent patches to generate multiple video fragments. Let τ'(x), and τ''(x) be the two-quality consistent augmented fragments for a mini-batch of videos x∈(V_v∪U) fed to the QAM. We apply semantically invariant geometric flips to the downscaled video frames to generate augmented video pairs S(x) and S^F(x) for SAM. The intra-model consistency loss is given as

Knowledge Transfer (KT)-based loss:

The major contribution is leveraging quality information in unlabelled videos by transferring knowledge between high-level SAM and low-level QAM. Depending on the distortions present in a certain video (low-level or high-level or both), either the SAM or QAM prediction could be more accurate. Thus, if knowledge is transferred in both ways between the two models, an inaccurate prediction of one model may affect the learning of the other model. To overcome this limitation, we incorporate a teacher-student type learning framework, where the model whose prediction is more stable with respect to an augmented pair of a video is considered the teacher, while the other model is the student. Let ϵ_H and ϵ_L denote the consistency error of SAM and QAM where,

The prediction of the more stable (teacher) model (with lesser consistency error) is used as a pseudo-label for the other model i.e. the student model. The knowledge transfer loss function for a minibatch is given as:

where m= 1(ϵ_H>ϵ_L) is the stability condition and sg(⋅) denotes the stop gradient operation. We use a combination of the supervised loss, intra-model consistency loss, and knowledge transferable loss to train our model as

where λ_c and λ_U are the weights to balance the loss terms.

Experiments and Results:

Training Database: We train our model using the official trainset of LSVQ [4]. We randomly sample 2000 videos from the entire set for our work. Of these 2000 videos, 500 are randomly selected to be used as labelled data, whereas the remaining 1500 are used as unlabelled data. This selection is done 3 times such that the labelled videos are different in each set.

Evaluation Database: In intra-database setting, we evaluate our model using the official test split of LSVQ viz. LSVQ-test comprising of 7400 videos with resolution between 240p and 720p. We also evaluate on the 1080p resolution test split of LSVQ namely LSVQ-1080p encompassing 3600 videos. For inter-database setting, we evaluate on KoNViD-1k [5], and LIVE VQC [6] consisting of 1200 and 585 UGC videos, respectively. We analyse performance using Spearman’s rank order correlation coefficient (SRCC) and PLCC.

Implementation Details: We use the pre-trained ST-VQRL[1] encoder having Video Swin-transformer [2] architecture as the backbone of our QAM. τ' and τ'' sample a clip of 32 continuous frames of size 224 ×224 from the video. For SAM, we use a pre-trained CLIP [3] with Vit-B/32 backbone. S spatially resizes 32 frames randomly sampled from the video to a size 224 ×224, while S_F also randomly flips it.

Semi-supervised performance analysis: We compare our QASA-VQA with semi-supervised NR-VQA methods such as LPR [14] and SSL-VQA [1] in Tab. 1. We also benchmark popular semi-supervised learning methods such as Mean Teacher [7], Meta Pseudo-Label [8], and FixMatch [9]. We see QASA-VQA gives a substantial improvement over existing methods by leveraging quality and semantic information in videos. We also infer that the performance of QASA-VQA deteriorates if trained without IMC or KT losses. Our KT loss is crucial in achieving superior semi-supervised performance.

Evaluation on Limited Labelled Data: In this experiment, we evaluate QASA-VQA with popular NR-VQA methods trained on only limited labelled data and without any unlabelled data. In Tab. 2, we see that QASA-VQA outperforms recent state-of-the-art methods such as CSVT-BVQA [10], FAST-VQA [11], DOVER [12] and ModularVQA [13]. We also analysed the individual components of QASA-VQA viz. QAM and SAM. Though individually quality (QAM) and semantic (SAM) aware models give acceptable performance, their ensemble (QASA-VQA) gives a considerable performance boost.

Qualitative Analysis of QAM and SAM: To show the complementarity of QAM and SAM, we analyse the absolute error in prediction of QAM (E_QAM) and SAM (E_SAM) for two unlabelled videos. In Tab. 3, we show a frame from different videos. For Video 1, QAM’s prediction is poorer comparative to SAM’s as the background blur is measured without taking the context into account. For Video 2, QAM’s prediction is more accurate as the video suffers from compression artefacts while SAM’s prediction is highly erroneous as the semantics of the scene is good.

Conclusion:

We leverage the potential of vision-language (VL) models to effectively capture and utilize semantic information for semi-supervised no-reference video quality assessment (NR-VQA). Specifically, our approach integrates the strengths of a low-level quality-aware model and a high-level semantic-aware model, enabling knowledge transfer on unlabelled data. Our proposed QASA-VQA method significantly narrows the performance gap between semi-supervised and fully supervised NR-VQA models, achieving competitive results using minimal human-annotated labels.

Link to the paper

https://ieeexplore.ieee.org/abstract/document/10890466