Iterative transfer learning with large unlabeled datasets for no-reference image quality assessment

1.   Introduction

A growing demand for high-quality images and videos has prompted an increasing need for image quality assessment (IQA) tools ^[1]. The assessment of image quality is crucial for Internet-based platforms like YouTube and Facebook, as well as for handheld devices such as smartphones, as it significantly influences user perceptions ^[2]. Given that reference images may not be available in these applications, no-reference image quality assessment (NR-IQA) techniques are widely employed to estimate image quality. The basics of NR-IQA design is extracting and analyzing image features to detect distortions present in the images.

Images are often subjected to various types of artifacts. For instance, blocking artifacts resulting from JPEG compression and ringing artifacts caused by JPEG2K were explored in ^[3,4,5] and ^[6,7], respectively. Papers ^[8,9,10] investigated image blurriness, while ^[11,12] examined artifacts related to super-resolution techniques. Additionally, noise has been the subject of study for years, for instance in ^[13], a technique based on the discrete cosine transform (DCT) was proposed to reduce speckle noise, and similarly in ^[14], DCT is used for estimating non-stationary noise. Authors in ^[15] applied principal component analysis (PCA) to estimate noise levels from image patches. Lately, deep learning methods have been applied for estimating non-stationary noise ^[16,17].

However, in real-life applications, one usually encounters images that are corrupted by a combination of multiple distortions. Therefore, there is a need to develop a comprehensive NR-IQA metric capable of detecting a wide range of possible distortions. Distortions normally alter statistical properties of images both in spatial and frequency domains ^[18]. These statistical properties are known as natural scene statistics (NSS) and there are various methods that can extract such properties in the spatial domain ^[19,20,21], and transform-based domain, e.g., ^[22,23] which used a DCT transform or ^[24,25] which used a wavelet and curvelet. Additionally, techniques presented in ^{[26,27,28,29]} combine both spatial and frequency characteristics to provide more robust and reliable results. When an image is distorted, the distribution of its NSS values shifts, reflecting the degree of degradation. This shift allows us to measure the deviation of a distorted image from a normal pristine. For NR-IQA, this measurement is performed independently for each type of distortion (e.g., noise, blur, or compression artifacts) to capture the unique impact of each distortion on the image's quality. Finally, these individual measurements are combined in a pipeline using regression techniques, e.g., support vector regression (SVR) ^[30], and the final score value is obtained. NSS thus serves as a powerful tool to extract meaningful statistical differences between distorted and pristine images in the NR-IQA framework.

The emergence of deep learning and convolutional neural networks (CNN) has brought about a significant transformation in the field of image processing, including NR-IQA. The first CNN to design NR-IQA was a relatively shallow network utilized on synthetic datasets ^[31]. CNNs have been mostly used for end-to-end optimization ^[32,33] and training NR-IQA models requires extensive datasets labeled with mean opinion scores (MOS) as the target value. Some well-known datasets for this purpose are FLIVE ^[34], KonIQ-10k ^[35], NRTID ^[36], HTID ^[37], SPAQ ^[38], and LiveInWild ^[39]. However, many of these datasets are insufficient in size for training deep models, which can lead to overfitting as deep networks tend to memorize key features from tens of thousands of images. One of the approaches to address this issue is employing transfer learning ^[40,41].

The idea behind transfer learning is that knowledge acquired from solving one problem can be used for solving a different but related problem. Instead of training a model from scratch on a new task, transfer learning allows utilizing the learned features, representations, or parameters from a pre-existing base model. In the case of NR-IQA design, pre-trained weights can effectively avoid overfitting and compensate for the insufficiency of datasets. In most cases, the desired task is different from the initial task which the base model was trained for. Therefore, the last layers of the base model should be modified according to the new task. It means that the number of output neurons and loss function in the last layer should be changed. Authors in ^[32] used pre-trained models with 27 layers, added 5 extra layers to modify the architecture according to the desired task and fine-tuned the model on NR-IQA datasets.

Generally, there are popular architectures that are frequently used for transfer learning. VGG16 ^[42], Inception-v2 ^[43], and MobileNet ^[44] are examples of such architectures, and authors of ^[45] and ^[35] used them to develop NR-IQA models. In ^[45], a new loss function, Earth mover's distance (EMD), was used as the new regression loss. Likewise, in ^[40], authors used Xception ^[46] as the base model to predict the opinion score distribution instead of MOS. In ^[47], pre-trained CNNs were used as feature extractors where both the image and its sub-regions were fed to the pre-trained model to extract features. In ^[48], a Siamese network ^[49] was trained to rank images based on the image quality, and then, by fine-tuning, the knowledge gained by the Siamese network was transferred to a CNN that estimated a final image quality. However, transfer learning has this limitation that is only feasible between similar network architectures, because it is using the pre-trained weights of a strcuture so it is bound to use the same structure.

The primary motivation for employing CNN-based models is their efficient handling of local features and spatial dependencies. This is especially critical in the context of NR-IQA, where the extracted features should closely align with the human visual system (HVS), considering both local and global characteristics. Both of these characteristics which can be also be called low-level and high-level features, are important in estimating the image quality because the HVS analyzes the composition of the scene and colors, which are high-level features, and blur or noisy artifacts that are found in low-level features. One of the important parts of an image is the salient regions. The salient regions generally contain rich semantic information and thus have a great impact on the overall quality. HVS also focuses on salient regions, therefore, utilizing techniques to separate such regions for better analysis have been used extensively because they can model the focusing characteristic of HVS and assign larger weights to the focused parts. Authors of ^[50] and ^[51] used CNN architecture in combination with the visual attention modules or saliency maps, which helped them to focus on specific regions of the image. Another instance of employing attention mechanisms can be found in ^[52], where feature-product networks were crafted by considering principles inspired by biological vision. One of the state-of-the-art methods that utilized saliency for image quality, specifically, on surveillance face images, was presented in ^[53]. The model called Hyper-IQA proposed in ^[54] also fits in the category of semantic-based or content-aware techniques. It consists of three main steps: 1) content understanding, 2) perception rule learning, and 3) quality predicting.

One of the recent methods, proposed in ^[55], presented a hybrid method based on a CNN and transformer. An off-the-shelf pre-trained CNN was used as a feature extractor and its output was used as an input to the transformer. The authors combined the self-attention mechanism of transformers and a locality bias in the extracted features of the CNN, which led to a robust analysis of the image because the correlation between different local features was measured and considered during quality assessment.

In this paper, we addressed the limitation of transfer learning and incorporaing HVS in NR-IQA design by introducing a two-branch CNN-based network that accepts both high- and low-frequency image patches as inputs. In our study, we propose an improved version of the transfer learning proposed in ^[56] and created a dataset by collecting thousands of images from Google Open Images (GOI) ^[57]. This extensive dataset covers a broad spectrum of concepts and categories, including common objects, animals, people, and nature.

Many CNN-based NR-IQA approaches frequently face high computational costs and varying execution times due to variations in input image sizes and, at the same time, pre-trained models often require image resizing, which, when performed non-proportionally, has the potential to jeopardize image quality and the image aspect ratio. Moreover, many of the pre-trained network parameters are large, rendering them unsuitable for hardware implementation. Our models possess a relatively small number of parameters. Additionally, our approach to image patch selection, which involves the random sampling of local patches and global patches that incorporate low-level and high-level features, respectively, both TBCNet parameters and the image patch selection procedure aid in accelerating the model's processing speed, eliminating the need for image resizing.

Our contributions in this paper can be explained in two main parts: first proposing a two-branch convolutional neural network (TBCNet), that uses both low-level and high-level features of images for estimating image quality. The network has low memory requirements and a small constant computational time for images of different resolution. This feature makes our model particularly well-suited for real-time applications and resource-constrained environments.

Second, proposing iterative transfer learning (ITL), which is an iterative version of the new transfer learning method introduced in ^[56], we propose an iterative training scheme and ensure the robustness of the method by training on noisy MOS.

The rest of this paper is organized as follows: Section 2 details the proposed iterative transfer learning method. Section 3 describes the TBCNet model, used for verifying the ITL approach. Section 4 explains the application of transfer learning on the KonCept512 NR-IQA model and TBCNet. Section 5 presents numerical analysis, comparisons, efficiency and generalization of our method, and analysis of the denoising capability of our proposed training scheme. Conclusions and future work are explained in Section 6.

2.   Iterative transfer learning

In this research, we enhance and expand upon the transfer learning technique that was initially presented in our prior work ^[56]. The framework of the proposed method is depicted in Figure 1. Additionally, a comprehensive description of the (ITL) process is provided in Algorithm 1. The principle of this method which is proposed in ^[56] involves transferring knowledge from a well-developed, pre-trained NR-IQA model, referred to as metric $A$, to another NR-IQA model with a different structure, named metric $B$.

This approach utilizes two separate datasets: a smaller, MOS-annotated dataset ($MD$) for training metric $A$, and a larger, unlabeled dataset, referred to as the unlabeled large dataset ($ULD$), for training metric $B$. The $ULD$ is instrumental in preventing the model from overfitting during training. To train model $B$ using the $ULD$ dataset, MOS values (target labels) are necessary for each image.

As such, the transfer learning process presented in ^[56] begins by using the pre-trained model $A$ to estimate the quality values of images in the $ULD$. Therefore, metric $A$ evaluates the quality of images and produces a predicted assessment ($PA$) array for the $ULD$. Assuming model $A$ is sufficiently accurate, the $PA$ values can serve as substitute MOS, albeit with inherent noise:

Here, $MOS_{ULD}$ denotes the MOS of the images in the $ULD$, $PA$ denotes predicted pssessments of image quality by metric $A$, and $MOS_{true}$ denotes the true, but unknown, MOS of the $ULD$.

The noise variable $\epsilon$ represents the prediction errors of metric $A$. If metric $A$ performs satisfactorily, it is anticipated that the error will exhibit a distribution that closely approximates a Gaussian distribution. Notably, a noisy MOS situation is a common condition in NR-IQA metric training. MOS are prone to noise due to the sample size of observers' opinions that is used for averaging and obtaining MOS.

Above, we detailed the method described in ^[56]. In this paper, we have elevated this technique by introducing an iterative training procedure leveraging both the $ULD$ and $MD$ datasets. With this approach, we harness the combined power of $ULD$ and $MD$ for training purposes. Following this, we use metric $B$ to calculate the $PA$ of $ULD$ for another round (updated $PA$ values). These updated $PA$ values are then used to train metric $B$ again. This iterative process can be repeated multiple times, enabling the $PA$ values to evolve and eventually stabilize. Once stable, these values represent the MOS of $ULD$.

To gain a thorough understanding of the image selection process from both datasets and the updating of $PA$ values, please refer to Algorithm 1.

The role of $ULD$ is essential in the training process and it is created based on three characteristics:

1) Comprehensiveness: $ULD$ must encompass a diverse range of image quality factors typical of existing MOS-annotated image databases.

2) Consistency: $ULD$ images should span the same $PA$ value range as the MOS of $MD$ used in training model $A$.

3) Challenge: $ULD$ images should pose a sufficient challenge for $PA$ prediction.

For refining $PA$, the equation in line 10 of Algorithm 1 is used. The $K$ coefficient is introduced within this equation, and in this experiment, it is set to 1. Choosing a smaller value for $K$ provides smoother, and potentially more effective training and increasing the number of iterations that can lead to the better performance.

3.   TBCNet architecture for NR-IQA

This section introduces TBCNet, a CNN-based architecture for NR-IQA, which we train using ITL. Our model, illustrated in Figure 2, incorporates two branches for image analysis, as our goal is to encompass both high-level and low-level image features. High-level image characteristics contain spatial scene configurations, object placements, color composition, and image clarity. The majority of NR-IQA models exclusively depend on the examination of patches with dimensions of $224\times224$ or $512\times512$. This approach tends to neglect the elements of high-level image characteristics and misinterpret the global attributes of the image. Additionally, our model effectively handles low-level image characteristics, including factors like noise type and intensity, blur type and intensity, sharpness, and the presence of compression artifacts.

This network has a minimal number of parameters, making it lightweight and efficient for implementation. Furthermore, its processing time remains consistent across varying image sizes due to the method of input patch selection. In contrast, many neural networks that follow the full-image analysis method such as KonCept512, do not adhere to this criterion, resulting in prolonged processing times.

3.1.   Structural scheme of the proposed metric

The structural scheme of TBCNet is presented in Figure 2. TBCNet accepts two inputs, each receiving RGB images standardized to a size of $720\times128$, regardless of the original image size. This design guarantees consistent analysis time for all images. The first branch is designed to input the low-level image features and the second branch is set up to analyze high-level image features. Each input comprises five segments of the input image, separated by black stripes, which simplifies the network's edge analysis.

Each branch of TBCNet consists of 11 blocks that perform analysis and downscaling. The analysis and feature extraction are performed by eight blocks. Regularly, after each two/three blocks of analysis, one block of downscaling is applied to reduce the size of the feature maps. In total, it has nine blocks of analysis and three blocks of downscaling. The architecture of the blocks are similar and consist of three parallel connections. An Example of one analysis block is illustrated in Figure 3.

In the first connection, there are three convolution kernels followed by ReLU activations. The second and the third connections have two and one kernel, respectively. For downscaling, instead of pure pooling layers, the last convolution kernel of each connection uses strides and no padding.

Three parallel connections are inspired by skip connections in ResNet ^[58]. Skip connections link features before and after processing ^[49,58]. In our model, as a skip connection, we incorporated information from three distinct parallel processes. By concatenating features from different depths, the network can capture a more comprehensive representation of the data. Each depth level, in the parallel connections within each block, has the ability to capture different features of the image patch. Concatenating them enables us to integrate the maximum amount of image feature information.

Previously, it was mentioned that a black stripe is used to separate image patches. This separation of patches allows the network to distinguish between different scenes and their compositions. Since each input contains five image patches, at the end of the process for each branch, five tensors with the size of 1 $\times$ 128 are created and result in a 5 $\times$ 1 $\times$ 128 feature space, which is delivered to fully connected layers. Finally, two branches will unite into one fully connected layer by concatenating the outputs. The final unified layer is responsible for integrating the feature maps and the final score. As the loss function, L2 is adapted for this model.

3.2.   Formation of network inputs

The proposed TBCNet model is a deep network with two branches and two specially prepared inputs (see Figure 2). The input for analyzing low-level features consists of five image fragments selected randomly with a size of 256 $\times$ 256 pixels, which are downsampled to a size of 128 $\times$ 128. An example of the patch selection process is shown in Figure 4 (left). These selected patches are desirable for analyzing noise level, blur level, and other low-level features and were selected randomly, which guaranteed a wide variety of patches, from the simplest to those with high-frequency features. This element of randomness ensures the diversity of the patches chosen.

For high-level features, however, a larger square window is used to crop parts of the image. To select the square size, we used the concept of the golden ratio (demonstrated in Figure 4 (right)), the ratio of the desired square edge to the rest of the image. This proportion has been appreciated for its aesthetic qualities and is believed to be visually pleasing to the human eye. It has been used in various fields, such as art and architecture. The window is moved to all directions (center, top, bottom, left, and right), five image fragments containing high-level features are cropped, and afterward, they are downscaled to 128 $\times$ 128 size. Upon extracting and preparing high-level and low-level patches, they undergo separation by inserting dark pixels between them.

4.   Transfer learning from KonCept512 to TBCNet

In this paper, we select KonCept512 as metric $A$ due to its good performance. Koncept512 is a conventional method of transfer learning that employs the pre-trained Inception-v2 model ^[43] as its base model. To train this metric, we need to construct our dataset $MD$. For this purpose, we selected six datasets: FLIVE ^[34], Konik-10k ^[35], NRTID^[36], HTID ^[37], SPAQ ^[38], and WILD ^[39]. These datasets are among the most commonly used no-reference (NR) datasets for training and validating the performance of NR-IQA methods. They encompass a vast number of images, collectively around 60,000 images, and contain authentic distortions (except for HTID, which has synthesized distortion), ensuring the robustness and universality of the metric $A$. FLIVE is one of the largest datasets, consisting of real-world images from user-generated content (images captured by digital cameras and smartphones). The distortions it covers stem from poor lighting, camera quality, and compression artifacts. Konik-10k, with nearly 10,000 images, is also considered one of the large datasets. Its content is gathered from online photo and video hosting platforms, making it diverse in content. On the other hand, SPAQ is a dataset with images captured by smartphones that reflect the limitations of smartphone cameras in its distortions. Finally, the WILD and NRTID datasets are smaller but worth including among the datasets because they contain images from different sources with naturally occurring distortions, including noise, blur, and artifacts from hardware limitations. To create a large and diverse dataset with MOS values, we followed the method outlined in ^[59] to construct the $MD$ and combine the MOS values.

As previously stated, our approach also requires the $ULD$ dataset. The $ULD$ dataset was formed by randomly choosing 360,000 images from the Google Open Images (GOI) datasets ^[57], which contain a vast range of images with varied content. The reason behind selecting images from GOI is that it has a reach image selection that helps the model to adapt to different real-world scenarios, and it includes images from 60,000 different categories. Subsequently, the KonCept512 metric trained on $MD$ was utilized to compute quality scores for $ULD$. In our setup, the suggested TBCNet functions as the metric $B$, which means that knowledge is to be transferred from Koncept512 to TBCNet. As we are using ITL, TBCNet is trained on both the $ULD$ and $MD$ datasets for two iterations.

TBCNet is trained in three states. The first state is the transfer learning proposed in ^[56], and then we use the proposed iterative training for one and two iterations. TBCNet is designed and trained in MATLAB 2022 with the custom training loop technique. The model was trained on 100,000 epochs. The learning rate starts from 0.0001 and is decreased by half at every 20,000 epochs.

According to the curves of training and validation in Figure 5 (left), which shows the direct training on the $MD$ dataset, the decrease of loss stops near epoch 50,000 (evidence of overfitting) though in Figure 5 (right), the reduction continues near to epoch 100,000 (no visible overfitting). Hence, we can assert that our approach also addresses the issue of overfitting.

5.   Experimental results

In this section, we present the results of our proposed NR-IQA method and compare it to state-of-the-art metrics. spearman's rank order correlation coefficient (SROCC) is adopted as the primary quality metric for evaluating the performance. SROCC is a non-parametric measure of statistical dependence between two ranked variables, and it is commonly used to evaluate the correlation between predicted and ground truth quality scores in IQA tasks.

For testing purposes, a selection of images was gathered from each of the six NR datasets: FLIVE, Konik-10k, NRTID, HTID, SPAQ, and WILD. These images were not part of the training set $MD$. The number of images selected from each dataset is as follows: FLIVE = 250 images, HTID = 200 images, KonIQ-10k = 200 images, NRTID = 50 images, SPAQ = 200 images, and WildLive = 100 images. Table 1 illustrates the training of TBCNet in three modes: 1) trained with the $ULD$ dataset (based on transfer learning in ^[56]), 2) trained with the $ULD$ and $MD$ datasets (ITL), and 3) trained with $ULD$ and $MD$ in two iterations (ITL 2 iterations).

We included several well-known metrics in the analysis and the results of the comparison are demonstrated in Table 1. It can be seen that using both $ULD$ and $MD$ together in training TBCNet results in a better SROCC compared to using $ULD$ alone. Furthermore, employing two iterations marginally enhances this SROCC value. It is important to note that the KonCept512 model, trained exclusively on the KonIQ-10k dataset, delivers the highest SROCC value of 0.925 for the same test set.

This can be explained by the model's specific training on the KonIQ-10k dataset, leading to a bias and optimal performance on this particular dataset. Given these considerations, allow us to discuss the results for the three modes of TBCNet and compare them with KonCept512. TBCNet, trained on a vast dataset with artificial MOS, demonstrates increased generality as the dataset size expands.

Our TBCNet, trained on $ULD$ and $MD$, benefits from extensive data and does not display bias toward specific test sets. The rationale behind employing ITL is to show that even without MOS data, our model can approximate the performance of traditionally trained CNNs. This shows the robustness of our model and its ability to achieve high accuracy without being influenced by the dataset on which it was trained.

5.1.   Generality of ITL

In the next experiment, we will perform a cross-dataset evaluation of the efficiency of ITL in comparison with KonCept512 and the methods that use a single-stage transfer learning. We made a slight change in data selection so the training data and the test data are taken from different databases. In our experimental setup, FLIVE and SPAQ datasets serve as the $ULD_{new}$, and Wild and Koniq as the dataset $MD_{new}$, and NRTID as the dataset for testing.

In Table 2, we have presented the results of SROCC for the different versions of TBCNet and the KonCept512 method trained on $MD_{new}$.

It should be noted that in this experimental setup, we made modifications to the patch selection process. Specifically, we did not downsample the low-level patch dataset. For the high-level image patch, we selected a single patch of size $348\times512$, extracted from the center of the image.

5.2.   Efficiency of TBCNet implementation

Another critical characteristic of TBCNet is its light structure and suitability for implementation, which is analyzed and compared with other state-of-the-art metrics in Table 3. This table compares model sizes (number of trained parameters) and the time (seconds) it takes to predict the quality of one image in different resolutions on a computer with CPU i5-9300H 2.4 GH and GPU NVIDIA GeForce GTX 1660 Ti 6 GB. The results displayed in Table 3 illustrate TBCNet's remarkably low computation time, indicating the efficiency of our method in terms of parameter count. This provides evidence for its suitability in practical implementation. Also, the execution time for TBCNet for images with varying resolution is constant due to the efficient strategy of selecting input patches.

5.3.   Analysis of the denoising ability of iterative transfer learning

The phenomenon of noisy MOS frequently occurs due to the large number of images and the limited number of evaluators available to rate them. For example, FLIVE is among the largest datasets available. However, the ratings for each image are only averaged across 20 subjects, leading to a noisy MOS.

In response to this challenge, ITL can be an effective solution. We demonstrate that TBCNet, when trained via ITL for two iterations, can effectively reduce noise in $PA$ values, leading to more consistent image scores. This results in a denoising effect, suggesting the potential application of our model in enhancing the quality of MOS in recently created image databases.

To validate the effectiveness of ITL in this scenario, we generated noisy $PA$ values by introducing additive white Gaussian noise with a variance of 1 to the ground truth $PA$ values. The visual demonstration of noisy values can be seen in Figure 6 (left). The results show that the noisy $PA$s are converging toward the ground truth $PA$ after being updated by ITL in two iterations (Figure 6 (right)), and the noisy values of $PA$ are scattered with more deviation from the ground truth $PA$ compared to the reconstructed $PA$s (Figure 6 (left)). It should be mentioned that the model used for this experiment is TBCNet, trained by ITL in two iterations. $PA$ values predicted by TBCNet after two iterations of the ITL are visualized in Figure 6 (right).

Figure 7 shows the curve of the quality of TBCNet in terms of SROCC versus the level of noise presented in the $PA$. It is clearly seen that as the noise level in the $PA$ increases, the efficiency of the proposed transfer learning decreases. However, it still remains operational over a vast range of noise levels.

6.   Conclusions and future work

This paper introduces a robust and efficient solution to address key challenges in no-reference image quality assessment. We proposed an iterative transfer learning methodology that overcomes the limitations of traditional transfer learning and addresses the issue of over-learning. By utilizing large and diverse datasets, our model ensures a better generalization compared to the conventional transfer learning for a wide range of image quality conditions. Additionally, we introduced TBCNet, a two-branch CNN architecture designed to achieve fast and efficient implementation while maintaining high accuracy. TBCNet's multi-scale and multi-level feature extraction enables it to capture both local details and global semantics, making it a powerful tool for image quality assessment. The experimental results showcase the notable performance of TBCNet, as evidenced by the short execution time and the good SROCC values simultaneously. Moreover, the TBCNet model, along with iterative transfer learning, demonstrates exceptional resilience to noisy MOS data, a common occurrence in real-world scenarios. This robustness allows the model to be used effectively for denoising MOS, enhancing the accuracy and credibility of NR-IQA predictions. In our current work, we exclusively choose random segments for low-level features and utilize large patches of the image for high-level features. However, for future studies, there is potential to incorporate attention modules into the TBCNet architecture for improved segmentation of the image's semantic regions. Additionally, the use of various types of loss functions could yield enhanced results.

Conflict of interest

The authors declare no conflict of interest.