A review of different deep learning techniques for sperm fertility prediction

1.   Introduction

Human semen is a very complex biological fluid, a mixture of seminal plasma and sperm ^[1]. It is retrieved from multiple organs of a reproductive system. This fluid is different from other body fluids and contains peptides and proteins. These proteins play a vital role in fertilization, hence showing the overall quality of sperm.

Researchers have observed that human semen quality has declined gradually ^[2]. To seek the reason behind this, scientists perform semen analysis according to the rules given by World Health Organization (WHO), like assessing sperm concentration, semen volume, total sperm count, sperm morphology, sperm vitality and sperm motility ^[3]. Research shows that semen quality in men is decreasing gradually with time, the authors in ^[2,4] reviewed more than 60 papers and found out seminal fluid quantity and number of sperms in a given sample are declining in the past 50 years ^[2]. Many researchers compare a sample of different geographical areas, and a few studies different characteristics of sperm motility in their geographical regions. Swan et al studied 101 papers and figured out that reduced sperm count was due to different reasons in past years ^[4]. Huang et al examined more than 115 papers and tested about 23000 young Chinese men, concluding that there is a remarkable decrease in seminal plasma (SP) and concentration of complex biological fluid composed of sperms ^[5]. Another reason for human semen decline observed is environmental contaminants, also known as persistent organic pollutants (POPs) and polychlorinated biphenyls (PCBs). These are highly toxic industrial compounds and shows high health risk to humanity. PCBs are found in almost all parts of the world in fishery and dairy products, even in women's breast milk. Recent studies show that PCBs also affect the reproductive system in men and women, like miscarriage and infertility in men ^[6] One more factor of men's infertility or loss in sperm quality is the sexual abstinence period. The ejaculatory abstinence period may increase the motility and viability of sperms but damaged DNA due to increased RSO levels ^[7].

There are several machine learning techniques that can be used for sperm fertility prediction are Convolutional Neural Networks (CNNs): Due to their capacity to automatically extract characteristics from images, CNNs are frequently employed in image processing and recognition tasks ^[8].

Long Short-Term Memory Networks: In order to analyze time series data, recurrent neural networks of the LSTM variety have been employed. To simulate the evolution of sperm motility over time and forecast future sperm motility values, LSTMs can be employed in the context of sperm fertility prediction ^[9].

Generative Adversarial Networks: GANs are a kind of deep learning model that can create fresh samples by studying the distribution of the input data. GANs can be utilized to create artificial samples of sperm pictures for deep learning model training in the area of sperm fertility prediction ^[10].

Autoencoders: A particular kind of unsupervised deep learning model called an autoencoder is capable of learning a compressed representation of the input data. Autoencoders can be utilized in the context of sperm fertility prediction to reduce the dimensionality of the sperm image data and extract practical characteristics for classification ^[11].

Semen analysis is vital in predicting male fertility ^[12]. WHO sets some standards for semen analysis which include sperm motility, morphology and vitality etc. However, with the help of these guidelines, we cannot differentiate between fertile samples with unfertile samples. On the other hand, manual semen analysis is challenging, clinical results for male fertility prediction are uncertain, and a complex procedure requires trained medical staff and special equipment ^[13].

Therefore, researchers have developed automatic systems for semen analysis for several decades. After the digitization of images, Computer Aided Sperm Analysis (CASA) was introduced in the 1980s and made it possible to analyze the image using computer systems ^[14]. CASA provide a fast and objective assessment of sperm motility and concentration, but with low accuracy due to the presence of other particles in sperm samples, CASA is not recommended for clinical use. So, to improve methodology, Steven et al apply classical and modern machine learning techniques to the dataset of 85 videos and try to predict sperm motility automatically. Figure 1 shows extracted frame from a video dataset.

Urbano et al propose an automatic sperm tracking technique called a fully automated multi-sperm Tracking Algorithm. It can track hundreds of spermatozoa simultaneously ^[16]. Dewan et al. also proposes a technique that uses Convolutional Neural Network (CNN) to classify into sperm or non-sperm. Their results were evaluated on a close dataset, so they compared it with other approaches ^[17].

Graph Neural Networks (GNNs) ^[18] is an important tool for investigating graphs, which have become global in various applications, including recommendation systems, social networks, and computational biology ^[19]. Due to their effectiveness in handling a variety of graph-structured datasets, such as social networks, molecular structures, and 3D models, Graph Neural Networks (GNNs), a class of deep learning models, have gained increasing amounts of popularity in recent years. B. Jiang et al ^[20] introduce a new learning framework, termed Multiple Graph Learning Neural Networks (MGLNN), for performing multi-graph learning and semi-supervised classification across multiple views. The primary aim of MGLNN is to acquire the most optimum graph structure that can facilitate both the learning of GNNs and the integration of multiple graph structures in representation.

In previous research methodologies, CASA tool was used with classic image processing and machine learning. But in recent years, deep learning has played a vital role in many computer vision applications. For example, deep learning techniques became more capable of handling human behavior. Just like Robots, because they are now more capable of handling human behaviour with the help of deep learning techniques. Singh et al introduce Tinku, a socially intelligent robot with low operating costs that was created to teach special need children. Tinku is inexpensive, has a ton of features, and can make expressions akin to those of a human. The well-known animated character "WALL-E" served as inspiration for its design. Its features include offline speech processing and computer vision—for obstacle avoidance, non-verbal communication, anthropomorphic emotion expression, etc., author also employed light object detection models like Yolo v3-tiny and single shot detector (SSD). It localizes the items in the scene using an inbuilt deep learning approach and uses the data for semantic perception.

In our research, we evaluate the efficiency of modern machine learning and deep learning techniques over sperm microscopic videos of human sperm microscopic datasets/videos and related parameters to automate the prediction of human sperm fertility. The current study looks on the possibilities of deep learning approaches for predicting sperm fertility. The goal of this research is to create a model that is more accurate and trustworthy than current state-of-the-art models. The current work provides an in-depth examination of many deep learning approaches utilized in sperm fertility prediction. The review focuses on each technique's strengths and weaknesses, as well as potential for improvement.

2.   Literature review/background

Infertility is public health issue around the world that affect 15% of all men at the age of reproduction ^[21]. In male infertility, semen quality is a critical parameter that reduces daily. When couples are diagnosed with infertility, women undergo ovulation and tubal patency tests, and men need semen analysis. Semen analysis includes many factors that need to be examined, like sperm morphology, density and motility.

Semen quality is an essential parameter in determining male fertility and infertility. A fertile sample could provide us data from which researcher can classify a given sample as Normozoospermic (A normal sperm that can be fertile). Semen sample was taken from 4 continents, 14 countries and 4500 men and selected men as normozoospermic. Those men have selected whose wives have time to pregnancy (TTP) is 12 months.

According to WHO a fertile sample volume must be greater than 1.5 ml, total sperm count in one ejaculation near 39 million, sperm concentration 15 million per ml, energy 58%, progressive motility 32%, and total progressive and non-progressive motility must be 40% and morphology 4%. Research shows that human semen quality varies from time to time. According to WHO, three reasons are essential in sample variation ^[3].

a) Delay in sexual activity

b) Human error during testing

c) Inheritance

To make this process up to standard, researchers have to follow the guidelines by WHO. Global demographic population studies give us a better understanding of semen quality. Before experimenting in the laboratory, research needs to preserve semen sample; this process is called Cryopreservation ^[22]. But this process damage spermatozoa which cause reduces in semen quality. So, the researcher introduces an additional method called (sil-select density gradient centrifugation) before cryopreservation. The new method gives us more sperm count and motile count and less DNA fragmentation than conventional methods.

Minh Tam Le compares vitrification and conventional freezing procedure for preserving semen samples in terms of vitality, motility and morphology of a semen sample. After evaluating the results, they conclude that the conventional freezing preservation method is more effective than the vitrification method ^[23].

Another reason that affects semen quality is the sexual abstinence period. Delay in sexual activity increases sperm motility and viability but damages DNA due to an increase in reactive oxygen species (RSO) level ^[7].

Huang et al examined more than 115 papers and tested about 23000 young Chinese men, concluding that there is a remarkable decrease in the concentration of semen samples ^[5]. G Anifrandis et al also divide 13892 infertile men of different nationalities into two regions. One is called mena and the other is non-mena regions. This experiment is carried out in health care provider in Qatar. Results shows persons belonging to Mena region are more infertile than non-mena region ^[24].

Infertility also causes due to different diseases. Luca Boeri finds out Papillomavirus is one of them. In this research work, he tested 729 samples of infertile European men. He found that 15.5% of men are effect by Papillomavirus, and their sperm motility is lower than the of an average person that is not have Papillomavirus ^[25]. Avi Harlev uses Ritalinic Acid (RA), a drug to treat attention-deficit hyperactivity disorder. The author concludes that ritalinic acid increases sample concentration and sperm motility in the infertile semen sample. He also considers the effect of the drug on other parameters of a sample ^[26].

Fabiana Faja discusses the consequence of mitochondrial DNA on sperm motility. If they increase the quantity of mitochondrial genome, it helps to restore damages in the semen sample. The quality of the semen sample is based on the sperm count, which is an essential factor for predicting a person's fertility and infertility ^[27].

Based on these infertility factors, motion analysis of sperm is very important for determining male fertility. As the detection and tracking of sperms be performed more accurately, it results in a more accurate diagnosis of infertility problems. Usually, sperm analysis is a complex procedure done by experts by spotting numbers on sperm through a microscope, detecting their motion quality and morphology etc. ^[28].

Besides the manual way, Computer-aided sperm analysis (CASA) was introduced in 1980s for sperm analysis. It was a very successful software for measuring sperm characteristics like sperm concentration, and progressive and non-progressive motility in many animal species. But in the case of human semen analysis CASA tool does not show good results due to complications present in human seminal fluid ^[29].

So, researchers need more advanced methods for semen analysis, Hamza et al propose smart phone base hybrid system for automatic analysis of sperm morphology, as sperm morphology is an essential factor of male fertility. The author divided this procedure into two steps, ⅰ) classification of normal and abnormal sperm and ⅱ) automatic segmentation of sperm shape. Machine learning models have been used for the sake of classification. Moreover, deep neural network architectures were used for high-level feature extraction from raw images. Similarly, Mobile-Net, a very efficient network for smartphones that gave 87% accuracy. On the other hand, wave-let and descriptor-base features gave 80.5 % and 83.8% accuracy with the support vector machine, respectively ^[30].

Soroush et al also worked on sperm morphology analysis by malformation detection of sperm morphology in sperm image dataset and applied the deep learning algorithms. They performed their analysis on the images from MHSMA dataset were out of 1540 sperm images, and 235 infertile patients. Deformities were detected in different parts of human sperm using trained deep neural network architecture. Results show F0.5 scores of 83.86%, 84.74%, and 94.65% in head, acrosome and vacuole abnormality detection, respectively ^[31]. Furthermore, they used deep learning for the classification of sperms. They apply VGG16, a deep CNN architecture, to train on ImageNet and apply to freely available sperm image datasets. This technique shows efficient results against previously proposed techniques in head-to-head comparison and classify sperms with high accuracy ^[32].

Steven takes a dataset of 85 videos and applies modern and classical machine learning techniques to automate the prediction of sperm motility. He adopts convolutional neural networks with simple linear regression to do so. Results show that deep learning gives very quick and consistent output, and this analysis may become a valuable tool in male fertility prediction ^[33].

In this latest research work, the author uses RetinaNet, a deep, fully convolutional neural network, as an object detector. Sperms are microscopic objects with few characteristics, and it is challenging to identify millions of sperm with other particles in semen samples. One main characteristic of sperms is their movement. Still, only one image is not enough to detect the movement of sperm and consecutive images were needed to identify the movement and motility of sperm, as sperm motility is an essential factor for predicting male fertility. For sperm tracking, a modified CSR-DCF algorithm was used. This technique works very well even in high density of sperms, occlusions, sperm colliding, and when sperms exit from a frame and re-enter in the subsequent frames. Results show F1 score of the tracking method is 96.61% and the detection phase is 99.1% ^[34].

3.   Organization of paper

In the next section, we will discuss different techniques used by the researchers regarding dataset collection. These datasets and techniques are selected from different scientific databases published from 2012 to 2022, including ACM, Google Scholar, IEEE, Xplore, Springer etc. The selection procedure for datasets is explained in Figure 2.

Moreover, in Figure 3, we also discuss pre-processing techniques, feature extraction techniques, segmentation techniques, classification techniques and their evaluation matrices from the above-selected articles. For better understanding, we create different tables respectively. We further divided feature extraction and segmentation techniques into two tables, one for machine learning techniques and the second for deep learning techniques. But pre-processing, classification and evaluation metrics are discussed in one table.

4.   Dataset and techniques

In this section we discuss shorty about dataset used in above selected articles. Table 1 shows dataset name, its source, type either it is in image form or in video form, their number of instances, resolution and target class.

In the first dataset, 100 slides were used, containing 1 to 5 sperms. Each slide contains 10 to 25 images. Image resolution was 576 × 764 pixels in RGB color space ^[36]. The article by Tseng et al. took datasets from the hospital and named them original images. The dataset is based upon 160 sperms, 80 normal and 80 abnormal sperms. The images are provided on https://code.google.com/p/support-vector-machine-for-sperm-morphology diagnosis/ as an Additional material ^[12]. Wang et al used a public dataset in their research work. This dataset contains 100 university students in good health, from 18 to 36 years old. Every case of the dataset consists of 9 different attributes like health, life habit, semen quality and concentration etc. ^[37]. Another famous dataset is the human sperm morphology analysis dataset (HSMA-DS). This dataset contains 1457 sperms cells with normal and abnormal sperms with different deformities in the head, midpiece, and tail. Images were captured at ×400 and ×600 magnification using IX70 microscope and DP71 CCD camera with chromatic infinity objective lenses and a resolution of 576 × 764 pixels in RGB color space ^[35]. Some researchers also consider video sample for sperm fertility prediction, in the same year Tan et al recorded video samples for semen analysis in Hospital University Sains Malaysia (HUSM). 80 samples were recorded using x40 magnification ^[38]. In the same year, another researcher selected 100 images with 7 attributes including age, alcohol consumption, smoking habits, accident/trauma, surgical intervention, number of hours spent sitting per day, childish disease and high fever with two target classes ^[39]. Another research took a video dataset for sperm analysis. The Videos were recorded in the andrology lab of the Academic Unit of Reproductive and Developmental Medicine (University of Sheffield). A healthy volunteer was ready to give his sperm sample for research purposes. Image resolution was set to 2040 × 1086 pixels. University Research Ethics Committee approved all procedures for experiment ^[40]. In 2016 tan et al performed another experiment on a video dataset recorded in the same university i.e (Hospital University Sains Malaysia (HUSM)). This time age of donors was at least 15 years and 50 microliters of fresh semen samples were collected but the equipment and parameters for recording the video were the same as used by the author in 2015. The dataset consists of 20 original images; each frame shows 60 sperm cells. So total 1200 (60 × 20) sperm cells were used for segmentation ^[41]. In same year Shaker et al use the Gold-standard dataset for semen analysis. This dataset consists of 19 stained sperm samples, total of 210 consider valid out of 264 sperm cells. Images in this dataset are in RGB format with 780 × 580 pixels ^[42]. In 2017 Mirsky et al took semen sample from 8 healthy volunteers, after one week of the sexual absence period. Total 886 images were selected for this experiment ^[43]. Garcia et al used video dataset for experiment. Images were divided into different classes based on orientation and morphology of sperm cell ^[44]. Similarly, Chang et al introduced human sperm head classification gold-standard SCIAN-MorphoSpermGS, introduced in ^[45]. Sample were collected in the Laboratory of Spermiogram, Program of Anatomy and Developmental Biology (ICBM) from young volunteers ages from 28 to 35-year-old ^[45]. The Human Sperm Head Morphology dataset (HuSHeM) is very famous, freely available online dataset, mostly researchers used this dataset for sperm analysis. It is consisting of 216 sperm heads (57 pyriform, 53 tapered, and 52 amorphous and 54 normal). Images are in RGD format with 131 × 131 pixels size ^[46]. Shaker et al also work on another dataset in the same year 2017. Healthy donors collected samples in Isfahan Fertility and Infertility center (IFIC). Slides were dried then fixed and stained using Diff-Quik method as described in WHO 2010 manual ^[3]. Image resolution was 720×576 pixels and stored in bitmap format. Then sperm head was classified by expert staff available at IFIC. Final dataset consists of 170 sperm heads (39 pyriform, 23 tapered, and 42 amorphous and 66 normal) ^[47]. Nissen et al constructed a dataset of 35 samples and 765 grayscale images. These 35 samples were diluted using Bicarbonate-Formalin solution as recommended by WHO ^[3], to get specific number of cells in each image i.e. 2 to 290 cells per image. The image resolution was 1920 × 1440 pixels ^[48]. Engy et al perform Experiments on the fertility-dataset, which is available at UCI repository. This dataset is based on 100 instances with 9 attributes like age, the rate of alcohol, diseases in childhood and surgeries etc. ^[49]. Author use Ocular images of the microscope through a smartphone-based data acquisition technique. In this dataset total 13 stained images of 13 persons were collected and analyse in this study(Ilhan et al., 2018). In next paper author captured images using inverted microscope (Olympus IX71) with camera (Optronis 5000) with speed of 3000 images per second. Image resolution was 512 × 512 pixels ^[51]. Movahed et al introduce Gold-standard Dataset. This dataset consists of 20 stained normal and abnormal sperm images with resolution of 780 × 580 pixels. The author divides this dataset into 14 training and 6 testing sets for the sake of the experiment ^[52].

In last paper of 2018 author use video dataset for semen analysis. In this digital experiment microscope was used at 400× magnification with 3200 × 2400 digital resolution. The slides of semen sample consist of 10µL volume and videos captured by automated microscope at the rate or 90 frames with 30 frames per second. These images were categorized by expert pathologist into sperm and non-sperm ^[17].

Hicks et al used online available video dataset (VISEM) from 85 persons for experiment. These videos were recorded at 400× magnification and saved as AVI file format. The videos were between 2 to 7 minutes with 50 frames per second. Expert manually assessed this sample for sperm motility and concentration. Author also stores donor's data like BMI, age, sexual absence period etc. (Hicks et al., 2019). In the next paper, the author employed an in-house dataset of fluorescence images and bright-field with the confocal microscope at ×100 objective magnification, 488 nm wavelength and 598–660 nm emission filter for red and 500–550 nm emission filter for green ^[53]. Riordon et al used 2 datasets to evaluate VGG16-based approach; first one is (HuSHeM) and second (SCIAN) dataset. These two datasets are freely available online for research purposes. HuSHeM dataset contains 216 RGB images with 131 pixels × 131 pixels at ×100 magnification. All images were labelled and classified by three experts as 57 Pyriform, 52 Amorphous, 53 Tapered and 54 Normal. SCIAN dataset contains 1132 greyscale images with 35 pixels × 35 pixels at × 63 magnification. Similarly, all images were labelled and classified by expert as 7 Pyriform, 262 Amorphous, 69 Tapered and 35 Normal ^[32]. In 2018 Movahed et al again work on public Gold-standard dataset in 2019. This dataset consists of 20 images with a resolution of 780×580 pixels and is divided into normal and abnormal sperm cells. However, handmade ground truths were publicly available, but some errors should be corrected ^[54].

In next study, the dataset contains a semen sample of 18 persons from Shenzhen Sixth People's Hospital. The videos were recorded by BA210 contrast microscope handed with 20× phase contrast objective and a Monochrome CCD Camera. The videos were captured at 25 frames per seconds and duration was 3 seconds with resolution of 720 × 540 pixels. Author recorded 3 videos per sample, total dataset contains 54 videos ^[55]. Javedi et al use new dataset named as Modified Human Sperm Morphology Analysis dataset (MHSMA); this dataset is an extended form of the Human Sperm Morphology Analysis dataset (HSMA-DS) introduced in ^[35]. This dataset consists of 1,540 RGB images with a pixel size of 1280 × 1024 ^[31]. In 2020 Mohammadi et al introduce dataset that contains 36 different videos that were recorded in the Royan institute Research Lab. at Tehran. The recorded videos are 8bit grayscale with 50 frames per second frame rate and 768 × 576 pixels resolution. Each video consists of 25 frames. The number of sperms in the videos is in the range of 4 to 95 in each video, and in total, all the videos contain 1628 sperms ^[56].

In next study 119 non-obstructive azoospermia (NOA) patients were referred to the IVF unit at SUMC for conventional testicular sperm extraction (cTESE) between 1995 and 2017. Azoospermia means absence of sperm after ×400 magnification and 3000g centrifugation. According to WHO criteria, all patients submit their semen sample on 4 different occasions ^[3]. None of these patients underwent TESE before ^[57]. Lesani et al collected semen samples from 41 patient's ages between 30 and 45 years old. These patients were referred to Avicenna Infertility Clinic for male infertility treatment ^[58].

SCIAN is a gold-standard dataset for the morphological classification of human sperm heads with five categories: Normal, Tapered, Pyriform, Amorphous, and Small. The images in this dataset are of greyscale with stained sperm heads, taken at 63 × magnification and their height and width are both 35 pixels and 7 µm. Domain experts have three separate agreement settings: no agreement, partial agreement, and total agreement. The first set consists of 1854 sperm head images (175 Normal, 420 Tapered, 188 Pyriform, 919 Amorphous, and 152 Small). Still, an image in this set can be labelled manually into three different classes by three domain experts. The second set comprises 1132 images (100 Normal, 228 Tapered, 76 Pyriform, 656 Amorphous, and 72 Small), but an image can be labelled into two different sperm head classes. The third set includes 384 images (35 Normal, 69 Tapered, 7 Pyriform, 262 Amorphous, and 11 Small). All three experts assigned the same class label to a sperm head image. From the number of images in these three sets, we can appreciate the difficulty of the morphological classification of human sperm heads even by human experts. For illustration (Figure 4), we show typical samples of human sperm heads of microscopic images of the five classes in the partial agreement setting of the SCIAN dataset and the four classes of the HuSHeM dataset ^[59].

Ilhan et al used Sperm Morphology Image Data Set (SMIDS) for sperm detection and classification. Sample were collected from 17 different persons from ages 19 to 39 years. They had no sexual activity (no ejaculation) for at least 2 days but not more than 7 days. Software-based image stabilizer was used for better image acquisition. Figure 5 shows sample capturing procedure ^[30]In next study author collect semen sample and make sure that human must not ejaculate from 4 to 7 days. For recording videos of semen. Videos were record at 30 frames per second with 288 × 352 resolution. Figure 6 shows procedure of microscopic semen analysis ^[60].

Marin et al used the human sperm segmentation gold-standard SCIAN-SpermSegGS dataset. Semen samples were collected at the Laboratory of Spermiogram, Program of Anatomy and Developmental Biology (ICBM), Faculty of Medicine, University of Chile, ages 28 to 35. This dataset contains total of 19 images with 264 sperm cells. The image resolution was 780 × 580 pixels. Field experts designed hand-made segmentation masks for every image in the dataset. Figure 7 shows a detailed image from SCIAN-SpermSegGS dataset. (a) Number of sperm cells in original RGB colour space (780 × 580 pixels) (b)–(d) Nucleus, acrosome and head of valid sperm in (a), Handmade segmentation mask respectively ^[61].

Ottl et al also used a visem dataset from the Simula Research Laboratory. This dataset consists of 85 videos of live spermatozoa from men aged 18. Figure 8 shows still image from the sperm sample videos ^[62]. In next paper, according to the best of the author's knowledge, there are no existing large-scale image datasets of testicular biopsies. After institutional review board (IRB) approval at Stanford University, Stanford, CA, USA (Approval No. 41652), they collected a novel dataset of 702 de-identified images from testicular biopsy samples of 30 patients ^[63].

Next dataset is based on 20 images with 15 to 20 sperms cells in each image. The author uses morphological analysis for edge detection and filtering. Figure 9 shows sperm input images, after applying the morphological process, image representation and segmentation will be performed ^[64].

In the year 2022 SVIA dataset was introduced by ^[65]. This dataset was obtained from JingHua Hospital of Shenyang, and the preparation of the SVIA dataset production began in 2017. After about four years of work, more than 278000 objects were annotated, as shown in Figure 10. Table 2 shows three subsets of given dataset.

Subset-A is mainly used for object detection tasks, providing more than 125000 objects' location and category information in 101 videos ^[66], their formats were.mp4, and the frame rates were 30 frames per second. Moreover, as object detection datasets, 3590 images and annotation files are obtained from 19 microscopic videos. The formats of the images are.png, and the sizes were 698 × 528 × 3. The annotation files contain the object categories and locations.

Subset-B was prepared for image segmentation and object tracking tasks, providing more than 26,000 sperms in 10 videos as ground truth (GT) for tiny object tracking and 451 images as GT for segmentation. Each GT contains the location of each sperm in different videos and frames, their formats are.png, and the sizes are 698 × 528 × 3.

Subset-C was developed for image denoising and classification tasks, which provides more than 125; 000 independent images of sperms and impurities, including 121; 401 sperm images and 4; 479 impurity images. Meanwhile, this large-scale dataset can also test the robustness of various deep learning models to noises. To highlight the advantages of this dataset, author compare the SVIA dataset with three publicly available datasets. HuSHeM ^[46], Visem ^[8] and MHSMA ^[31] (as shown in Table 3). However, for all the above datasets, the dataset size is relatively small and quickly bottlenecked for cases where large datasets need to be used. This dataset contains 101 videos, 130042 images, and 127600 object information, which perfectly solves the problem of too little information and quantity in the above three datasets.

5.   Pre-processing techniques

In this section we discuss pre-processing techniques that were applied on each dataset one by one. Table 4 demonstrate pre-processing techniques with their respective outcomes. In first artical Bijar et al scaled the image as a pre-processing technique using Red component of RGB. M was the range between zero to maximum intensity, and it was 12-byte data I initially, then scaled to a new intensity between 0 to 255 (8-bit). It was formulated as I s = I/M × 255 ^[36]. Wang el al. Introduced Joint Rank Difference and Gray Level Method as a pre-processing step to identify between normal and abnormal sperms. They also pre-process the dataset by only removing the majority Tomek Links and balance data with the help of SMOTE technique ^[37]. Ghasemian et al apply Gaussian filter for noise reduction on very famous dataset (HSMA-DS); this algorithm converts RGB color values to YcbCr Y-channel sharpening. By applying Gaussian Filter on Y channel Noise Reduction by wavelet transform, the noise percentage decreased. However, this percentage was not acceptable for further processing ^[35]. Duggal et al has proposed an optional attribute selection process technique that helps to filter attributes in dataset. The cumulative data was pre-processed using an optional attribute selection process. This intermediate step was used to filter only the relevant and important attributes from the bulk data. Attribute selection can affect the overall results derived from the data ^[39].

In 2016 researcher used video dataset for sperm analysis. First, they convert images into gray scale from RGB. Because colour image required more processing time as compare to gray scale images. In next step Laplacian of Gaussian (LoG) filter was deployed in kernel as a pre-processing step so that it automatically detects and segment human spermatozoa ^[41].

In same year Shaker et al used Ostu method for sperm head detection. Moreover, they apply a histogram of the Hue channel of HSV color space along with size criterion for successful sperm head detection in the microscopic image ^[42]. Chang et al introduced Anisotropic diffusion method for noise reduction. Moreover, this method also preserves the border of images ^[45].

Ilhan et al use Modified Overlapping Groups Shrinkage (MOGS) as a pre-processing step to remove the noising data from the sperm images that is not required for image analysis. Generally, these signal (y) of length N is modelled as

Where x was the noise free regions and w is the additive white Gaussian noise. In this technique, large magnitude pixels for an image tend not to be isolated. If noise found by using prior knowledge, then the standard deviation of the noise is applied and the noise free signal (the area of sperm and non-sperm particles) was known, the noise free groups can be separated from the noisy signal without knowing the group.

Where R(x) was the penalty function, λ was the regularization parameter and satisfies λ > 0. In MOGS, convex function was used to detect the correct region of sperm, therefore denoising problem was accepted in which a non-convex regularization term was employed and this verify the desired group sparsity property. Figure 11 shows effect of MOGS on given image as a pre-processing technique ^[50].

Movahed et al has presented logarithm transform for pre-processing. To shrink the range of intensity values, log transformation (V log (~ x) = log (V (~ x) + 1)) was applied on image stack V. So, it increases the contrast of low intensity values ^[51]. Homomorphic filtering was another pre-processing method in the frequency domain. This method helps to highlight each sperm in the background by correcting uneven illumination. Moreover, some post-processing steps were also involved in improving segmentation procedures like removing unwanted objects with respect to size, filling holes, closing/opening morphology operations etc. ^[52]. Similarly, in 2019 author used serialized method of pre-processing on Gold-standard dataset. This method helps to highlight sperm cells in the image rather than other particles. It also restrains unnecessary image characteristics like low contrast between the sperm's tails and its neighbouring region and non-uniform distribution of light ^[54]. Iqbal et al adopt three common techniques as previously used in the training set of the SCIAN dataset. For rotation, they rotate the training image by 25 to 25 degrees. For translation, they shift ~6% of the original image to the left, the right, up, and down. For flipping, they vertically flip the image. Due to the same distribution of classes within this dataset, they apply equal augmentation options to each class ^[59]. Ilhan et al. again used the same pre-processing technique he used in 2018 called MOGS. This technique helps to remove random noise without changing sperm shapes ^[77].

Median filter was also used for pre-processing; this technique was applied to each image. The edges of images were preserved when they pass through the median filter ^[60]. Some researcher also employs CNN for pre-processing. CNN is a specialized type of network used that has a grid-like topology for processing data. They are most commonly used on image and video datasets ^[8]Ottl el al; remove Geometric distortion, non-uniform contrast, and noise by spatial warping, background removal and filtering, respectively ^[62]. Wu et al use glare filters and diffraction correction for the pre-processing dataset. A single embryologist annotated each image with bounding boxes around each identified spermatozoon. At the time of collection, each image was normalized, passed through glare filters and diffraction correction (Hamilton Thorne, HT video and image capture software version 3, Hamilton Thorne, Inc., Beverly, MA, USA), and had microscopy artefacts removed ^[63]. In last article, Prabaharan et al used Median filtering as a pre-processing step. It helps to reduce noise in the image, perform edge detection, and improve image resolution. It also sorts pixel value by comparing the mean filter. It does not create any unreal surrounding pixels, so it gives efficient result after pre-processing ^[64]

6.   Features extraction through machine learning

In this section, we discuss methods used by different researchers for feature extraction for their respective datasets. Table 5 represent state-of-the-art Systematic comparison of Feature Extraction approaches through Machine Learning. ^[12] emian et al apply Sobel algorithm as a feature extraction technique for edge detection ^[35]. (Van Raemdonck et al presented Otsu's threshold as a feature extraction technique. It converts the image to grayscale and then eliminates the background from the foreground ^[40]. Shaker et al use HSV color space for sperm head detection in a pre-processing step. But HSV was a typical perceptual color space, instead, they use GVF model and achieve 92% success rate which outperformed state-of-the-art techniques ^[42]. In 2017 Mirsky et al used Phase Map Acquisition (PMA) technique to extract 89 custom-design features from the dataset. These features included the acrosome area, the ratio of head length to width, mean OPD and the correlation of the OPD map of the head to a model of the ideal head etc ^[43]. In the same year Garcia et al used SURF (Speeded-Up Robust Features) method for feature extraction, it also detects key point from a given dataset. Hessian matrix was used to select the most demonstrative pixels found in low-level features ^[44]. Chang et al introduce a new method for feature extraction called, five shape-based descriptors. It easily recognizes human sperm head and also helps in sperm morphology analysis ^[67].

Ilhan et al apply Pixel-Based Spatial analyses, Blob analyses for feature extraction (spatial-based features), over segmented parts. Normally a sperm is 2.5–3.5 micrometers wide and 5–6 micrometers long and contain an oval head with a single long tail. But abnormal have different shapes and sizes depending on their abnormality type. Moreover, images also contain different clusters than normal/abnormal sperm due to the staining process, which is indicated as non-perm ^[50]. Hicks et al used Lucene Image Retrieval (LIRE) for feature extraction. LIRE is a java library used to extract features from images and photos with the help of texture and color characteristics. The author test more than 30 features with machine learning algorithms, but Tamura image features work best with the proposed algorithm in this study ^[33]. The gaussian mixture model is another feature extraction technique. This technique used statistical information of pixels for background representation, such that the target pixel was figured out using statistical difference and judged probability density in a long period with a large sample value. As a result, a complex dynamic background could be modelled ^[55].

7.   Segmentation

In this section we discuss different techniques for segmentation using Machine Learning. These Traditional Segmentation approaches with Machine learning can be seen in Table 6. In 2012 Bijar et al used Markov random field (MRF) Model and Entropy based (EM) algorithm for segmentation. This segmentation includes sperms acrosome, mid piece, nucleus and sperms tail. They also place some estimated points on sperms tail for identification of morphological characteristics like length and shape. In first section Bayesian classifier was used to segment sperm's Nucleus, Mid-piece and Acrosome. In second section EM and MRF model was used to upgrade and obtain class conditional probability density function (CCPDF) and apriori probability for each class ^[36].

In 2013 Tseng et al used Average Rank Difference method for segmentation between normal and abnormal sperms. By using Average Rank Difference between abnormal and normal sperms the segmentation accuracy of 55% was achieve ^[12]. Wang et al propose Clustering-Based Decision Forests (CBDF) Algorithm to deal with unbalanced class learning problem for sperm analysis. Result shows that CBDF Algorithm outclass state of the art techniques like logistic regression, multilayer perceptron neural networks, random forests, Support Vector Machines and decision tree etc. with great difference ^[37]. Van Raemdonck et al proposed 3 different techniques for segmentation i.e. the structural similarity measure (SSIM), correlation (CORR) coefficient and the Bhattacharyya distance (BHAT). The original image was labelled with 1 or 0 and count of sperm, non-sperm, immotile and motile cells. Only motile cells were kept for experiment and all other non-sperm and immotile cells were removed ^[40].

For more refine segmentation, Shaker et al used edge-based active contour technique. Moreover, they also used a novel tail point detection technique that trace and eliminate midpiece from segmented head, for more efficient segmentation. Results shows 92% achievement against hand segmented ground truth, that outperform famous techniques ^[42].

In previous section shaker et al used different pre-processing and feature extraction techniques for efficient segmentation process. In 2017 Mirsky et al use Support Vector Machines (SVM) classifier for segmentation. The SVM classifier achieves receiver operating characteristic curve of 88.59%, precisions of 90% and precision-recall curve of 88.67%. Researcher believed that this automatic process of sperm analysis will become base of automatic and objective sperm cell collection in IVF ^[43]. Similarly, Garcia et al also used Support Vector Machines (SVMs) for segmentation. Simplified feature vectors that extracted from dataset were used to classify sperm cells into normal, abnormal and non-cell categories ^[44]. Again, Chang et al also used SVM base classifier called SVM-based cascade classification. This two-stage classification scheme shows better results than monolithic classifiers and it also work like a human expert ^[67] Ilhan et al apply Fuzzy C- Means (FCM) and K-Mean for segmentation. FCM segmented the gray scaled imaged more successfully than K-Means because of fuzzy technique. But there is room of improvement because only 50% sperms were correctly classified ^[50]. In last paper of Machine Learning techniques, Qi et al used the commonly used threshold segmentation. This technique was applied on background image for the sake of counting immotile sperm ^[55].

8.   Segmentation through deep learning

In this section we discuss different techniques for segmentation using deep learning. Table 7 elaborate State of the art Segmentation approaches through deep learning with their respective results. Tan et al used Pulse Coupled Neural Network (PCNN) technique for segmentation. PCNN is very famous method for sperm image segmentation. It helps to solve inaccurate sperm head shape problems as well. It also uses to differentiate agglutinated sperms with high accuracy and precision. Moreover, PCNN model include eight parameters so, optimized PSO algorithm was used improve PCNN parameters ^[41]. In 2016 they also proposed ICM mode for segmentation of same video type dataset. For efficient sperm head segmentation ICM is derived from several visual cortex models. There is some parameter selection problem, so researcher optimize ICM network with PSO. New fitness function know as feature mutual information was introduced here. This model was more efficient and precise than other famous segmentation techniques ^[41]. In 2017 Shaker et al introduced Active contours method for segmentation. This method automatically segmented cropped sperm heads. Colour images were transformed to gray scale images and apply threshold using Otsu method for fast locating of sperm heads. Moreover, edges of the images were extracted by Canny edge detector method and Gradient Vector Flow (GVF) active contour method was employed to extract outer contour of sperm heads ^[47]. Nissen et al apply CNN for segmentation. CNN outperform classical image analysis approach by achieving 91.89 % recall and 93.87 % precision on a given dataset ^[48]. In 2018 Movahed et al also apply CNN for segmentation on Gold standard dataset. For segmentation of sperm head a deep convolutional neural network was used. For improve head segmentation, geometric constraints and filling holes operation were also employed. Then SVM helps to classify all pixel of segmented heads to acrosome and nucleus regions ^[52]. Similarly, in 2019 Movahed et al introduced two CNN networks with different technique for segmentation of head and axial filament of the sperms. Overlapping patches were extracted from RGB images at the end of background using CNN model 1. Moreover, at Histogram Stretching stage, patches were extracted from RGD images using CNN model 2 ^[54]. Javedi et al used CNN network to deal with morphological deformities present in human sperm like head, vacuole, and acrosome. It was first technique that focus on acrosome. It also works very efficient in low resolution and non-stained images ^[31].

As previously discussed, that MOGS technique was used as a pre-processing step, and it helps to reduce noise present in sample images and improve the overall segmentation process. Similarly, Fuzzy C-Means clustering shows impressive results when MOGS was applied on images as a pre-processing method. Moreover, Fuzzy C-Means clustering apply fuzziness for the belongingness of each image pixels, which increase efficiency of segmentation process ^[68].

In past many algorithms were unsuccessful to detect tail region from video dataset because of minor gradient in the tail region. Some researchers able to do so while denoising images. Somasundaram et al successfully separate tail region with binary images with the help of local region contour segmentation ^[60]. Two well-known deep learning techniques U-Net and Mask-RCNN were proposed for sperm cell segmentation, using cross validation, hyper parameter tuning, transfer learning and data augmentation technique. Figure 12 shows that U-net with transfer learning, outclass state of the art sperm segmentation method with 95% overlapping efficiency against hand-segmented masks for sperm head 0.96, acrosome 0.94 and nucleus 0.95 ^[61].

In 2021 Valiuškaitė et al develop the sperm counting algorithm using the Faster R-CNN-Inception_V2-COCO model. Its advantage is that it is quite accurate to learn to distinguish objects even from a small database. The model takes about 50 ms to analyze the image. Because the model uses regions, it is easier to detect a larger number of objects in a single frame. Additionally, the R-CNN model can read information from frames of any size, so there is no need to crop frames before submitting them to the neural network. This approach achieved 91.77% accuracy of sperm head detection on the VISEM sperm sample video dataset, which is largest available video dataset (32 GB) that is available online ^[69]. Ottl et al introduced new segmentation process for each individual particle (mean squared displacement of each particle (imsd)) or again as an average for all of the particles (ensemble mean squared displacement of all particles (emsd)). These results show a relative improvement of 4.4% against state-of-the-art results ^[62]. Abbasi et al used Deep Transfer Learning (DTL) technique and Deep Multi-task Transfer Learning (DMTL) for segmentation. DTL technique accurately label head, vacuole, and acrosome with accuracy of 84%, 94%, and 79% respectively. Similarly, DMTL technique accurately label head, vacuole, and acrosome with 82%, 92.33%, and 80.66% respectively. Figure 13 shows visual representation of both models ^[70].

A new CASA system was developed with the help of deep learning that gave proximate human level performance on testicular sperm extraction (TESE). VGG16 convolution network employed or segmentation and trained on custom dataset. It classifies each proposal as either a spermatozoon or background and regresses the bounding box shape to match identified component objects ^[63]. Enhanced Otsu's threshold method reduces the computation complexity for segmentation. Sobel operator was introduced by Prabaharan et al for-edge reduction, which improve overall performance of segmentation. The abnormal regions were detected by convolution layer ^[64]. Chen at al used subset – B of SVIA dataset for segmentation. To evaluate the effectiveness of SVIA dataset author employ four traditional image segmentation techniques and four deep leering techniques, among the classical image segmentation methods, OT has the best effect. But it is lower than the four image segmentation methods based on deep learning. In general, the effects of U-net and Deeplab_v3 were the best, and other segmentation methods have a certain gap compared with these two segmentation methods ^[65]. In next study Ottl et al shows best results by unsupervised tracking of sperm cells with the Crocker-Grier algorithm, extracting imsd features for each detected track and aggregating those features into a histogram representation using BoW. With this feature representation, a linear SVR improved the mean (3-fold) MAE from 8.83 to 7.31, a decrease of over 17%. The results further show that the unsupervised feature quantization helps to achieve more consistent and robust results, regardless of which feature representation is chosen as input ^[71]. Aristoteles et al use YOLOv4 architecture for sperm detection from famous online available video dataset called VISEM. In this experiment dataset was available in the form of video and then converted into 1330 images. The training process was carried out with 3 different learning rate experiments, namely 0.002, 0.0002, 0.00002. In each of these experiments, 3 data divisions were made for each of the reading rates being tested. The best accuracy results are found in experiments with a learning rate of 0.0002 which has an accuracy value of 79.58% mAP on 70% train data distribution, 25% validation and 5% test. Each trial process for training uses 6000 iterations to create the training data. The test in this study uses video, the results of which are that all objects can be detected properly and have been labelled with a bounding box. In this study there were cases where the model was not able to detect optimally because the video data used contained blurred objects and sperm objects that were cut off by the frame ^[72] Thambawita et al present YOLOv5 deep learning model that is trained on the VISEM-Tracking dataset present baseline sperm detection performances. YOLOv5l performed best among all other versions of yolov5. This network achieves a precision of 43 per cent, recall of 25, and mAP is 22.3. Other networks are achieving lower results. The second-best network is medium, with a mAP of 22.1 and precision of 57.1 per cent. This dataset can be used to train complex deep learning models to analyze spermatozoa ^[75] Dobrovolny et al also use yolov5 for sperm cell detection and achieved very efficient results. The best-performing model is yolov5l. This network achieves a precision of 88.6 per cent, recall of 52.6, and mAP is 72.1. Other networks are achieving lower results. The second-best network is nano, with a mAP of 69.6 and precision of 64.7 percent ^[74].

Future research directions and challenges

In this section we discussed Systematic comparison of state-of-the-art Segmentation approaches through deep learning as well as we also suggest future research directions and challenges regarding these segmentation approaches ^[48]. Moreover, this process needs clinical testing for verification if we put this technique in lab practice for fast and quick semen analysis. Table 8 summarized Systematic comparison of state-of-the-art Segmentation approaches through deep learning. In first paper MOGS technique was used as a pre-processing step and it helps to reduce noise present in sample images and improve the overall segmentation process. Similarly, Fuzzy C-Means clustering shows impressive results when MOGS was applied on images as a pre-processing method. Moreover, Fuzzy C-Means clustering apply fuzziness for the belongingness of each image pixels, which increase efficiency of segmentation process. Author used K-Fold Cross Validation, with K = 5, Result shows Precision 83%, recall 82%, f-measure 83% and accuracy 88% respectively ^[68]. If Android-based embedded system will be deployed in clinical environment with real time working version of this proposed technique, we may get fast and robust diagnosis easily. In past many algorithms were unsuccessful to detect tail region from video dataset because of minor gradient in the tail region. Some researchers able to do so while denoising images. Somasundaram et al successfully separate tail region with binary images with the help of local region contour segmentation. Other methods that were used in literature measured with different densities of fluid. But in this study precision will not drop below 91%, even density was increased. So overall sperm detection process will be improved. Overall accuracy of this system is 97.37%. ^[60]. Minimum execution time was 1.12s and will be improved in future. Two well-known deep learning techniques U-Net and Mask-RCNN were proposed for sperm cell segmentation, using cross validation, hyper parameter tuning, transfer learning and data augmentation technique. Results shows that U-net with transfer learning, outclass state of the art sperm segmentation method with 95% overlapping efficiency against hand-segmented masks for sperm head 0.96, acrosome 0.94 and nucleus 0.95 using Dice coefficient as the evaluation metric ^[61]. These techniques show several opportunities for future research. First one, only focus on sperm head segmentation using U-net architecture rather than whole architecture. Second, computer-based sperm morphological analysis is hot research area and we plan to continue researching this area, and finally try to improve segmentation and classification result for fast and accurate analysis.

The sperm counting algorithm using the Faster R-CNN-Inception_V2-COCO model, also gave us impressive result on sperm head segmentation. Its advantage is that it is quite accurate to learn to distinguish objects even from a small database. The model takes about 50 ms to analyse the image. Because the model uses regions, it is easier to detect a larger number of objects in a single frame. Additionally, the R-CNN model can read information from frames of any size, so there is no need to crop frames before submitting them to the neural network. This approach achieved 91.77% accuracy of sperm head detection on the VISEM sperm sample video dataset, which is largest available video dataset (32 GB) that is available online ^[69]. As a future work there is great option available to apply this modern technique on another online available standard dataset like MHSMA, SMIDS, HUSHEM AND SCIAN etc. Ottle et al predicted motility of a given sperm sample by using machine learning framework, embedded with unsupervised technique of feature extraction using downstream regression models. The results show a relative improvement of 4.4% against state-of-the-art techniques ^[62]. Author shows interest to provide this solution to everyone at low cost by integrating this method into digital microscope in future. Abbasi et al used Deep Transfer Learning (DTL) technique and Deep Multi-task Transfer Learning (DMTL) for segmentation. DTL technique accurately label head, vacuole, and acrosome with accuracy of 84%, 94%, and 79% respectively. Similarly, DMTL technique accurately label head, vacuole, and acrosome with 82%, 92.33%, and 80.66% respectively. Figure 10 shows visual representation of both models. We suggest this deep transfer learning approach should be applied on other online available datasets ^[70]. Regardless of outperforming all state-of-the-art techniques, this proposed method cannot give perfect results and f 0.5 score on all labels of the MHSMA dataset. So, there is room to improve results and accuracy. A new CASA system was developed with the help of deep learning that gave proximate human level performance on testicular sperm extraction (TESE). VGG16 convolution network employed or segmentation and trained on custom dataset. It classifies each proposal as either a spermatozoon or background and regresses the bounding box shape to match identified component objects. This deep CASA system gave mean average precision (mAP) of 0.741, with an average recall (AR) of 0.376 on custom dataset ^[63]. Researchers were focused on creating hand on tool for real-time sperm identification. Yet there is no accurate enough tool is available in market to embryologist on the number of spermatozoa found in each microscopy image but must be fast and accurate enough to outperform an embryologist in the amount of time it takes to find spermatozoa in a sample. We expect that future work will focus on locating and identifying sperm in TESE samples in tandem with embryologists. Enhanced Otsu's threshold method reduces the computation complexity for segmentation. Sobel operator was used for edge reduction, which improve overall performance of segmentation. The abnormal regions were detected by convolution layer. After comparing with other techniques author concluded that this method gave 98.99% accuracy ^[64]. Results may be improved if in future with the help of embedded genetic algorithms using deep CNN and multiple datasets will be chosen for comparative analysis.

In year 2022 Chen et al used subset–B of SVIA dataset for segmentation. To evaluate the effectiveness of SVIA dataset author employ four traditional image segmentation techniques and four deep leering techniques, among the classical image segmentation methods, OT has the best effect. But it is lower than the four image segmentation methods based on deep learning. In general, the effects of U-net and Deeplab_v3 were the best, and other segmentation methods have a certain gap compared with these two segmentation methods ^[65]. In the future, Author plans to increase the number of sperm microscopy videos and images, expand the number of GTs, add more sperm and impurity images information, and improve the quality of the images. Ottle et al continue their research work in 2022 by reducing the mean absolute error (MAE) from 8.83 to 7.31. Researcher focused on unsupervised tracking of sperm cells with the Crocker-Grier algorithm, extracting imsd features for each detected track and aggregating those features into a histogram representation by using BoW. With this feature representation, a linear SVR improved the mean (3-fold) MAE from 8.83 to 7.31, a decrease of over 17%. The results further show that the unsupervised feature quantization helps to achieve more consistent and robust results, regardless of which feature representation is chosen as input ^[71]. For future work, the presented framework can be extended and improved upon by pursuing a number of additional research directions. In last article of Systematic comparison. Aristoteles et al also use VISEM dataset for experiments. This dataset was available in the form of video and then converted into 1330 images. The training process was carried out with 3 different learning rate experiments, namely 0.002, 0.0002, 0.00002. In each of these experiments, 3 data divisions were made for each of the reading rates being tested. The best accuracy results are found in experiments with a learning rate of 0.0002 which has an accuracy value of 79.58% mAP on 70% train data distribution, 25% validation and 5% test. Each trial process for training uses 6000 iterations to create the training data. The test in this study uses video, the results of which are that all objects can be detected properly and have been labelled with a bounding box ^[72]. In this study there were cases where the model was not able to detect optimally because the video data used contained blurred objects and sperm objects that were cut off by the frame. In future author will try to reduce this problem and achieve better results by using new model of yolo architecture like yolov7 and yolov8.

9.   Classification

In this section we discuss different classification method used in different researches and their outcomes. Ghasemain et al used SMA algorithm for classification and results shows above 90% of accuracy for sperm abnormality detection. It also shows high rates of True positive and True negative ^[35]. Duggal et al has presented Particle Swarm Optimization PSO algorithm for classification. PSO gives the highest accuracy then both MLP and SVM methods, PSO obtains superior specificity values at 88%. Therefore, PSO seems to be the preferred method for predicting the quality of fertility data with respect to the environmental factors and lifestyle, this method seems to be useful with the new data also as it can handle generalization potential ^[39]. In 2017 Shaker et al introduced Linear Discriminant Analysis (LDA) method for classification. To do so LDA minimize variance within class and maximize between classes, as a result maximum separation between classes was achieved ^[47].

Engy et al introduced Sperm Whale Optimization algorithm (SWA) with Artificial Neural Networks (ANN-SWA) on fertility dataset for classification. ANN-SWA was based on four phases. First one was trained by fertility diseases. In second multilayer perceptron (MLP) was employed to get maximum fitness function. In third phase parameter were optimized like biases and weights to get rapid conjunction with high accuracy. In final step ANN-SWA was employed to get sperm fertility prediction ^[49]. In next article researcher used 3D+t Human Sperm Flagellum Tracing in low SNR Fluorescence Images. Sperms were assigned by an ID for the sake of tracking. Proposed algorithm extract similar traces as compared with ground truth. Results shows that this algorithm was more accurate and robust than multi-branch algorithms. This enhanced 3D image permits guiding an iterative center line algorithm towards the flagellum's centerline. Each sperm is assigned an ID to keep track of it in 3D+t, this algorithm was extracting similar traces as compared with ground truth and it was more robust and accurate to trace the flagellum's centerline than multi-branch algorithms ^[51]. Dewan et al also employ CNN for classification. This CNN contain 5-layer network with 3 convolution layers with 16, 32 and 64 3 × 3 filters respectively. These were followed by 2 fully connected layers with 256 neurons in each layer and finally two node soft max layer that classify input into sperm and non-sperm ^[17]. Hicks et al also use CNN for classification. CNN was trained that learned temporal and spatial combined features of sperm motility. That was based on image preparation and representation of single or stacking frame. Then participant data and output of CNN model was combined. Before performing multivariate prediction on the three motility variables, this combined vector was passed through two fully-connected layers ^[33].

In 2019 McCallum et al presented CNN base VGG16 (CNN) technique for classification. Results shows that moderate correlation (bivariate correlation ~0.43) between DNA quality and sperm cell image, and ability to identify higher DNA integrity cells was related to median. This deep learning model will help clinicians for the microscopy-based sperm selection, it predicts fast DNA quality with in 10ms per cell and up to 86 % successful sperm selection for fertility analysis ^[53]. Movahed el al also introduce SVM classifier for classification. Multi-channel image generation and SFFS algorithm was optimized with SVM classifier to compete with other classifiers. As a result, axial filament pixels show improved performance and accurate classification ^[54]. Javadi et al employed CNN for classification of sperm image dataset into normal and abnormal sperms with the help of multiple morphological characteristics. CNN achieved F 0.5 score of 94.65%, 83.86% and 84.74% in vacuole, head and acrosome abnormality revealing, respectively. Figure 14 shows the proposed architecture ^[31].

Riordon et al used deep learning technique for sperm head classification. This Deep Neural Network architecture gave high accuracy in head-to-head comparison against other famous methods with identical dataset. Result shows better true positive rate 94.1% vs 92.3% on HuSHeM dataset, over cascade ensemble of SVM (CE-SVM) ^[32]. In 2020 Mohammadi et al used CSR-DCF tracker algorithm for classification. CSR-DCF algorithm was a robust multi-sperm tracker that works accurately, even when sperms collide or cross each other pass. This technique Obtained 96.61% F1 score from evaluation (Mohammadi et al., 2020). Zeadna et al proposed GBT and random forest method for classification and compare it with multivariate logistic regression model (MvLRM). GBT technique produced improved prediction results of successful and unsuccessful TESE against MvLRM, with little bit lower sensitivity. This model efficiently deals with missing values of several variables by automatically self-learning based on training loss reduction ^[57]. Lesani et al introduced two models, Full Spectrum and Selected Peak Neural Network for classification. To achieve low root, mean square error these models were boosted for transfer function, structure of hidden layer and different combination of training techniques. FSNN shows 93% prediction accuracy for sperm concentration and gave 100% assessments for differentiating the samples of patients and healthy donors ^[58]. Ilhan et al shows that SVM model with RBF kernel achieved highest accuracy that is 83.82% and tested by KAZE descriptor-based features. On the other hand, k-NN over DWT-based features shows worst classifier result with 47.23% efficiency. In terms of sperm morphology representation problem, KAZE features were the most informative features for RusBoost, Polynomial, Bagging and RBF kernel SVM classifiers (Ilhan et al., 2020). Iqbal et al used Morphological Classification of Human Sperm Heads MC-HSH for classification. In this deep learning architecture, there are a total of 53 convolutional layers. Before each convolutional layer, the batch normalization and LeakyReLU were used. They also apply the channel-wise concatenation and element-wise addition to make this model more efficient classification of human sperm heads in terms of morphology ^[59].

For classification of human sperm author introduced a novel Faster Region Convolutional Neural Network (FRCNN) with Elliptic Scanning Algorithm (ESA). More over for tracking and motility analysis Novel Tail to Head movement algorithm (THMA) was used. These proposed techniques improve the efficiency of computer assisted semen analysis (CASA) ^[60]. Priyansi et al use ResNet18 model for classification. The first ResNet18 model reached 100% accuracy on both validation and test sets. The second ResNet18 + Tabular Data model correctly predicted with an accuracy of 100% on the validation set and an average accuracy 88.89% on the test set. The third ResNet34 + Tabular Data model correctly predicted with an accuracy 87.5% on the validation set and 77.78% on the test set ^[8]. Figure 13 shows proposed framework by Ottl et al for-motility predictions. This framework shows best results for motility prediction. Figure 15 shows the proposed framework for motility prediction. The lower the Mean Absolute Error (MAE) the better, showing that SVR models achieve superior results than the other models for every type of features. BoWs outperform models trained on emsd features. Considering the best validation result, a minimum MAE of 8.60 was obtained on evaluation with the SVR trained on emsd features. Measured against the state-of-the-art, this result shows a relative improvement of 2.6% ^[62].

Abbasi et al used Deep Transfer Learning (DTL) technique and Deep Multi-task Transfer Learning (DMTL) for classification. The proposed algorithm achieved accuracy of vacuole, acrosome and head by 94%, 80.66% and 84% respectively by gaining improvement of 1.33%, 3% and 6.66% on these labels respectively. Therefore, this algorithm can be used in fertility institutes and make work easier for hospital staff ^[70]. Again, convolutional layers of VGG16 and GoogleNet used for classification. Deep learning CASA system achieved a mean average precision (mAP) of 0.741, with an average recall (AR) of 0.376 on given dataset ^[63].

Similarly, Prabaharan et al also use CNN for classification. MATLAB 2018a was employed to perform overall procedure. Classification process was improved with Deep CNN statistical training and morphology of sperm images. After comparison results shows efficient performance of proposed technique by providing neck and tail accuracy 97.99%, head utilize 98.22% and vacuole achieve 99.66% accuracy ^[64]. Chen et al applied 18 deep learning techniques on SVIA dataset. The precision, recall, F1-score and accuracy of Inception-v3 and DenseNet-121 were among the top. However, the precision, recall F1-score and accuracy of ViT were the lowest. So, for the image classification task of subset-C, Inception-v3 and DenseNet have the best classification effect, and ViT has the worst classification effect ^[65]. In next experiment author used all famous pre-trained deep learning models for classification like VGG16, VGG19, ResNet50, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, DenseNet, NASNetMobile, NASNetLarge and Xception. VGG19 and ResNet50 shows best accuracy of 87.33%, 71% and 73% for vacuole, acrosome, and head label, respectively ^[76]. In last article of classification Miahi et al also use MHSMA dataset for Sperm morphology analysis. Author introduce a novel automatic SMA technique that is based on the neural architecture search algorithm, named Genetic Neural Architecture Search (GeNAS). This Neural Architecture method shows accuracy of 91.66%, 77.66%, and 77.33% in the vacuole, acrosome, and head detection, respectively ^[77] Mashaal et al work on a dataset consists of 1200 images of human sperm heads divided into healthy and unhealthy. Here, the VGG16 model is fine-tuned and achieved an accuracy of 97.92% and a sensitivity of 98.82%. Moreover, it achieved an F1 score of 98.53%. this is effective and real time system model for detecting healthy sperms that can be injected into eggs for achieving successful fertilization (Mashaal et al., 2022). In year 2023 shahzad et al introduced a new Sequential Deep Neural Network architecture (SDNN) that detected abnormalities in the acrosome, head, and vacuole with an accuracy of 89%, 90%, and 92%, respectively.

We evaluate the efficiency of SDNN technique, against 11 pre-trained state-of-the-art deep learning models mentioned in ^[78] tested with the same dataset. We found that SDNN model outperforms all modern deep-learning models by a significant margin. Table 9 presents the big picture of these models, besides our model with respect to Accuracy, F1-score, Recall and Precision in different parts of sperm cells.

Neural Networks also used to solve multiple graph learning problem. In many machine learning applications, data are coming with multiple graphs, which is known as the multiple graph learning problem. Jiang et al proposed a novel learning framework, called Multiple Graph Learning Neural Networks (MGLNN), for multiple graph learning and multi-view semi-supervised classification. The objective of MGLNN is to simultaneously integrate multiple graph learning and GNN representation to learn an ideal graph structure from a variety of graph topologies that best supports GNN learning. The suggested MGLNN is an all-encompassing framework that can include any particular GNN model to deal with multiple graphs. To enhance/train the suggested MGLNN model, a general algorithm has also been created. MGLNN outperforms various other related approaches on semi-supervised classification tasks, according to experimental results on a number of datasets ^[20].

Similarly, after comparing with pre-trained modern deep learning models, author also compare this result with the previously proposed technique by different researchers with the same dataset. Eventually, as shown in Table 10, the proposed model shows higher accuracy, precision, recall and 𝑓0.5 on the test set for all three labels. Moreover, they demonstrated that, when applied to all three head, vacuole and acrosome labels of the MHSMA dataset, SDNN model can identify better than modern architectures than previously proposed model by different research, for instance manually designed CNN architectures, random search and image processing approaches, in terms of accuracy, precision, and f0.5 ^[77]. In Table 11 we elaborate all different classification Techniques and their respective outcomes.

Future research directions and challenges

In this section we discussed Systematic comparison of state-of-the-art classification approaches through deep learning as well as we also suggest future research directions and challenges regarding these segmentation approaches. Table 12 represent method of classification with proposed neural network and their evaluation metrics. In first paper author use CSR-DCF algorithm with a robust multi-sperm tracker that works accurately, even when sperms collide or cross each other pass. Result shows precision 99.1% and achieve 96.61% F1 score from evaluation of their proposed tracker method(Mohammadi et al., 2020). This experiment was performed on small dataset of 25 frames per video, so there is a space of research available and this technique will be applied on other larger dataset available online and compare results with previous ones. Zeadna et al proposed GBT and random forest method for classification and compare it with multivariate logistic regression model (MvLRM). GBT technique produced improved prediction results of successful and unsuccessful TESE against MvLRM, with little bit lower sensitivity. This model efficiently deals with missing values of several variables by automatically self-learning based on training loss reduction. GBT technique shows sensitivity of 91% and MvLRM shows sensitivity of 97% ^[57]. Machine-learning models may lay the foundation for a decision support system for clinicians together with their NOA patients concerning TESE. The findings of this study should be confirmed with further larger and prospective studies. Well trained Full Spectrum and Selected Peak Neural Network was used for sperm classification. To achieve low root, mean square error these models were boosted for transfer function, structure of hidden layer and different combination of training techniques. FSNN shows 93% prediction accuracy for sperm concentration and gave 100% assessments for differentiating the samples of patients and healthy donors ^[58]. However, by increasing sperm concentration by increasing sample size, must improve prediction accuracy of this method and increase chance of clinical adoption. Ilhan et al shows that SVM model with RBF kernel achieved highest accuracy that is 83.82% and tested by KAZE descriptor-based features. On the other hand, k-NN over DWT-based features shows worst classifier result with 47.23% efficiency. In terms of sperm morphology representation problem, KAZE features were the most informative features for RusBoost, Polynomial, Bagging and RBF kernel SVM classifiers (Ilhan et al., 2020). If Android-based embedded system will be deployed in clinical environment with real time working version of this proposed technique, we may get fast and robust diagnosis easily. Similarly, a novel Faster Region Convolutional Neural Network (FRCNN) was used for efficient sperm classification with Elliptic Scanning Algorithm (ESA). These proposed techniques improve the efficiency of computer assisted semen analysis (CASA). Results shows accuracy of 97.37%. Moreover, for tracking and motility analysis a Tail head movement-based (THMA) algorithm was used ^[60]. Researcher were satisfied with results of proposed architecture and plan to implement this technique in infertility centres in near future for healthy sperm detection in semen sample. Priyansi et al use ResNet18 model for classification. The first ResNet18 model reached 100% accuracy on both validation and test sets. The second ResNet18 + Tabular Data model correctly predicted with an accuracy of 100% on the validation set and an average accuracy 88.89% on the test set. The third ResNet34 + Tabular Data model correctly predicted with an accuracy 87.5% on the validation set and 77.78% on the test set ^[8]. This experiment concluded that the addition of tabular data decreased the accuracy of the model, which isn't necessarily an architecture limitation as the accuracy decreased moreover when a heavier 3D ResNet34 was applied, they found that these features were redundant for the training and classification of spermatozoa motility as they cause data leakage. The proposed models can be applied to automating the semen analysis process in the future which currently uses images instead of video and can even extend to predicting other quality parameters. Figure 12 shows proposed framework by Ottl et al for-motility predictions. This framework shows best results for motility prediction. The lower the Mean Absolute Error (MAE) the better, showing that SVR models achieve superior results than the other models for every type of features. BoWs outperform models trained on emsd features. Considering the best validation result, a minimum MAE of 8.60 was obtained on evaluation with the SVR trained on emsd features. Measured against the state-of-the-art, this result shows a relative improvement of 2.6% ^[62]. Author shows interest to provide this solution to everyone at low cost by integrating this method into digital microscope. Abbasi et al used Deep Transfer Learning (DTL) technique and Deep Multi-task Transfer Learning (DMTL) for classification. The proposed algorithm achieved accuracy of vacuole, acrosome and head by 94%, 80.66% and 84% respectively by gaining improvement of 1.33%, 3% and 6.66% on these labels respectively. Therefore, this algorithm can be used in fertility institutes and make work easier for hospital staff ^[70]. Regardless of outperforming all state-of-the-art techniques, this proposed method cannot give perfect results and f 0.5 score on all labels of the MHSMA dataset. So, there is room to improve results and accuracy. A new CASA system was developed with the help of deep learning that gave proximate human level performance on testicular sperm extraction (TESE). VGG16 convolution network employed or segmentation and trained on custom dataset. It classifies each proposal as either a spermatozoon or background and regresses the bounding box shape to match identified component objects. This deep CASA system gave mean average precision (mAP) of 0.741, with an average recall (AR) of 0.376 on custom dataset. Again, convolutional layers of VGG16 and GoogleNet used for classification. Deep learning CASA system achieved a mean average precision (mAP) of 0.741, with an average recall (AR) of 0.376 on given dataset ^[63]. After initial testing of the model, it will be deployed in an academic in vitro fertilization centre as a research tool for further testing. they intend to deploy the model as a real-time video classification pipeline, which automatically identifies sperm in testicular tissue. This novel tool will improve the efficiency of searching for sperm in testicular biopsy samples, which is currently a labour-intensive process dependent on the skill of the embryologist. Similarly, Prabaharan et al also use CNN for classification. MATLAB 2018a was employed to perform overall procedure. Classification process was improved with Deep CNN statistical training and morphology of sperm images. After comparison results shows efficient performance of proposed technique by providing neck and tail accuracy 97.99%, head utilize 98.22% and vacuole achieve 99.66% accuracy ^[64]. Results may be improved if in future through embedded genetic algorithms with deep CNN and multiple datasets will be chosen for comparative analysis. In year 2022 Chen et al applied 18 deep learning techniques on SVIA dataset. The precision, recall, F1-score and accuracy of Inception-v3 and DenseNet-121 were among the top. However, the precision, recall F1-score and accuracy of ViT were the lowest. So, for the image classification task of subset-C, Inception-v3 and DenseNet have the best classification effect, and ViT has the worst classification effect ^[65]. In the future, Author plans to increase the number of sperm microscopy videos and images, expand the number of GTs, add more sperm and impurity images information, and improve the quality of the images. Similarly in same year Chandra et al used pre-trained deep learning models as a feature extraction like VGG16, VGG19, ResNet50, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, DenseNet, NASNetMobile, NASNetLarge and Xception. VGG19 and ResNet50 shows best accuracy of 87.33%, 71% and 73% for vacuole, acrosome, and head label, respectively ^[76]. Moreover, the size of the current dataset is one of the enormous existing available datasets, but it needs to be enlarged for further experimental assessment of the algorithms in future. In last article of classification Miahi et al also use MHSMA dataset for Sperm morphology analysis. Author introduce a novel automatic SMA technique that is based on the neural architecture search algorithm, named Genetic Neural Architecture Search (GeNAS). This Neural Architecture method shows accuracy of 91.66%, 77.66%, and 77.33% in the vacuole, acrosome, and head detection, respectively ^[77]. Author proved that GeNAS can ascertain better state-of-the-art architectures – in terms of accuracy, precision, and 𝑓0.5 measure–compared to previously proposed methods, such as hand designed CNN architectures, image processing approaches, and random search, with less amount of computational power and human effort on all three acrosome, head, and vacuole labels of MHSMA dataset. Moreover, the architectures discovered by GeNAS have exceptionally fewer parameters on the head and vacuole labels. Finally, concerning the lack of NAS research to address the challenges of real-world datasets, author recommend that further research should be done in this area of research. In last paper shahzad et al evaluate the efficiency of SDNN technique, against 11 pre-trained state-of-the-art deep learning models mentioned in ^[78] tested with the same dataset. We found that SDNN model outperforms all modern deep-learning models by a significant margin. Table 9 presents the big picture of these models, besides our model with respect to Accuracy, F1-score, Recall and Precision in different parts of sperm cells. In future we suggest to compare this technique with other dataset as well as provide a mobile application to test sperm samples in real time environment.

10.   Evaluation metric

In this section we discuss Evaluation metrics techniques used by different authors. Table 13 explain different Evaluation metrics used by different researcher and their respective results. In first paper Bijar et al used expectation–maximization (EM) algorithm and Markov random field (MRF) model for segmentation. Results shows 97.6% overall success rate of this model. Moreover, Accuracy of sperm's head, Accuracy (A c) 96.82%, Sensitivity (S c) 97.52% and Specificity (S p) 95.27% respectively ^[36]. Ghasemain et al uses confusion metrics for evaluation. SMA algorithm shows its performance in less than 9 seconds. True negative rate was 100% for tail malformation detection. However, there was small number of false negative due to high rate of noise in the image ^[35]. Tan et al used Confusion metrics for evaluation in term of Accuracy, sensitivity, specificity and precision. Results show Accuracy 0.97, sensitivity 0.97, specificity 0.71 and precision 0.99 respectively ^[41]. In next paper the presented methods and their improvements can at least replace the initial tests for checking the fertility of a population or an individual as well as in selecting the donors for data collection. Stating and testing environmental and lifestyle factors trough the mentioned techniques were able to give an accuracy of up-to 90%, these results support the argument ^[39]. Similarly, Tan et al apply Intersecting cortical model (ICM) technique on video dataset in 2016. Results shows that this model was more robust and accurate than four state of the art segmentation models, in terms of accuracy, sensitivity, specificity and precision with success rate of 98.14%, 98.82%, 86.46% and 99.81% respectively ^[41]. In same year shaker et al apply different techniques for segmentation and achieve remarkable results. They use Dice coefficient to evaluate segmentation performance. Dice coefficient was 3% more than Chang method. Total performance was 88% ^[42]. The proposed method of Chang at al shows 73% efficiency as well as human experts were fully satisfied with the classification of sperm heads ^[67]. Shaker et al used machine learning based approach for classification of sperm head into 4 different classes. This method achieved average F-score of 92.9%, average recall of 92.3%, average precision of 93.5%, and average accuracy of 92.2%. This method shows very impressive result in abnormalities recognizing for all above mentioned four classes ^[81]. In next paper results shows the average accuracy of overall classes reached the 78% by using confusion metrics measurements ^[44]. Engy et al used 10-fold cross validation for evaluation metric and show that artificial neural network based on SWA achieved more than 99.96% of the accuracy ^[49]. Movahed et al apply 3 techniques for evaluation metric, namely Dice, Jaccard, and F1-score metrics respectively. The CNN was validated with 20 images of sperm sample. Results shows 0.94, 0.87, and 0.88 of Dice Similarity Coefficient for the head, the acrosome, and the nucleus segments, respectively ^[52]. In last paper of 2018 dewan et al use SQA-V Gold for evaluation. They conclude that SQA-V Gold obtain a B0 value of 1.91 and confidence interval of (−12.6 to −0.57) and B1 value of 1.665 with 95% confidence interval of (1.5 to 1.8) ^[17]. Javadi et al used CCN for segmentation. The proposed technique attains F0.5 Score of 83.86%, 84.74% and 94.65% in head, acrosome and vacuole abnormality detection, respectively ^[31].

In 2020 Mohammadi et al used Average precision (AP) metric for evaluation. Result shows precision 99.1% and achieve 96.61% F1 score from evaluation of their proposed tracker method (Mohammadi et al., 2020). Zeadna et al propose leave-one-out cross-validation for evaluation. The leave-one-out cross-validation procedure allows us to use the largest available training set and achieve an unbiased estimate for the expected accuracy ^[57]. To validate the performance of two model, 13-fold cross validation was employed by Lesani et al. Results shows average classification accuracy of 92% was achieved by the system ^[58]. Ilhan et al used K-Fold Cross Validation, with K = 5, Result shows Precision 83%, recall 82%, f-measure 83% and accuracy 88% respectively ^[80].

The Faster Region Convolutional Neural Network (FRCNN) was used for classification and improve the efficiency of computer assisted semen analysis (CASA). Other methods that were used in literature measured with different densities of fluid. But in this study precision will not drop below 91%, even density was increased. So overall sperm detection process will be improved(Somasundaram et al., 2021). Author use Region Based Convolutional Neural Networks (R-CNN) for segmentation. The motility of the semen sample calculated by the algorithm differs from the experimental result by only 2.92% on average, while the accuracy of sperm head detection was 91.77% ^[69]. In this article author use dice confident as an evaluation metrics. Results show that U-net with transfer learning, outclass state of the art sperm segmentation method with 95% overlapping efficiency against hand-segmented masks for sperm head 0.96, acrosome 0.94 and nucleus 0.95 ^[61]. Abbasi et al used Confusion metrics and Gradient-weighted Class Activation Mapping (Grad-CAM) for evaluation. To classify the vacuole part of a sperm image, our model should pay more attention to the vacuole of the sperm during the classification process. Chen et al applied 18 deep learning techniques on SVIA dataset. The precision, recall, F1-score and accuracy of Inception-v3 and DenseNet-121 were among the top. However, the precision, recall F1-score and accuracy of ViT were the lowest. So for the image classification task of subset-C, Inception-v3 and DenseNet have the best classification effect, and ViT has the worst classification effect ^[65]. In next study Ottl et al shows best results by unsupervised tracking of sperm cells with the Crocker-Grier algorithm, extracting imsd features for each detected track and aggregating those features into a histogram representation using BoW. With this feature representation, a linear SVR improved the mean (3-fold) MAE from 8.83 to 7.31, a decrease of over 17%. The results further show that the unsupervised feature quantization helps to achieve more consistent and robust results, regardless of which feature representation is chosen as input ^[71]. In this experiment dataset was available in the form of video and then converted into 1330 images. The training process was carried out with 3 different learning rate experiments, namely 0.002, 0.0002, 0.00002. In each of these experiments, 3 data divisions were made for each of the reading rates being tested. The best accuracy results are found in experiments with a learning rate of 0.0002 which has an accuracy value of 79.58% mAP on 70% train data distribution, 25% validation and 5% test. Each trial process for training uses 6000 iterations to create the training data. The test in this study uses video, the results of which are that all objects can be detected properly and have been labelled with a bounding box. In this study there were cases where the model was not able to detect optimally because the video data used contained blurred objects and sperm objects that were cut off by the frame ^[72]. Miahi et al also use MHSMA dataset for Sperm morphology analysis. Author introduce a novel automatic SMA technique that is based on the neural architecture search algorithm, named Genetic Neural Architecture Search (GeNAS). This Neural Architecture method shows accuracy of 91.66%, 77.66%, and 77.33% in the vacuole, acrosome, and head detection, respectively ^[77]. In last article author used pre-trained deep learning models as a feature extraction like VGG16, VGG19, ResNet50, InceptionV3, InceptionResNetV2, MobileNet, MobileNetV2, DenseNet, NASNetMobile, NASNetLarge and Xception. VGG19 and ResNet50 shows best accuracy of 87.33%, 71% and 73% for vacuole, acrosome, and head label, respectively ^[76]. Thambawita et al present YOLOv5 deep learning model that is trained on the VISEM-Tracking dataset present baseline sperm detection performances. YOLOv5l performed best among all other versions of yolov5. This network achieves a precision of 43 per cent, recall of 25, and mAP is 22.3. Other networks are achieving lower results. The second-best network is medium, with a mAP of 22.1 and precision of 57.1 per cent. This dataset can be used to train complex deep learning models to analyze spermatozoa ^[75]. Similarly, Dobrovolny et al aslo use yolov5 for sperm cell detection and achieved very efficient results. The best-performing model is yolov5l. This network achieves a precision of 88.6 per cent, recall of 52.6, and mAP is 72.1. Other networks are achieving lower results. The second-best network is nano, with a mAP of 69.6 and precision of 64.7 per cent ^[74]. Mashaal et al work on a dataset consists of 1200 images of human sperm heads divided into healthy and unhealthy. Here, the VGG16 model is fine-tuned and achieved an accuracy of 97.92% and a sensitivity of 98.82%. Moreover, it achieved an F1 score of 98.53%. this is effective and real time system model for detecting healthy sperms that can be injected into eggs for achieving successful fertilization ^[79]. In year 2023 shahzad et al introduced a new Sequential Deep Neural Network architecture (SDNN) that detected abnormalities in the acrosome, head, and vacuole with an accuracy of 89%, 90%, and 92%, respectively.

11.   Conclusions

In this study we consider 29 image dataset and 15 video datasets from latest published articles. We also mention source of dataset, no of images or videos included, method of recording dataset, their color space, resolution and target class. In next section we discussed pre-processing technique applied by different researchers. Total 17 techniques were applied on their respective datasets. Famous pre-processing technique was MOGS. Ilhan et al used these techniques in 2018 and 2020, In MOGS, convex function was used to detect the correct region of sperm, therefore denoising problem was accepted in which a non-convex regularization term was employed and this verify the desired group sparsity property. This technique helps to remove random noise without changing sperm shapes. Other techniques included CNN, Median filter, MobileNetV2, Rotation Translation flipping and elliptic curve were also used in literature for pre-processing. In next segment feature extraction techniques was discussed after pre-processing, that were based on machine Learning. Ten techniques were mention in this study included Phase Map Acquisition, five shape-based descriptors, Pixel-Based Spatial analyses, Blob analyses, Lucene Image Retrieval (LIRE) Tamura image features and Gaussian mixture model etc. similarly we also discussed feature extraction techniques based on deep learning. Thirteen deep learning technique were considered like DNA Fragmentation Index (DFI), CNN, VGG16 CNN, Feature Pyramid Network (FPN), 3D ResNet 18, RPN network, Inception, Resnet, MobileNet, MobileNetV2, Single-Shot Detector (SSD), ImageNet etc. Similarly, we also discuss 10 segmentation techniques with machine learning. Most researcher use Support Vector Machines for segmentation but some researcher also used Fuzzy C-Means (FCM) and K-Mean, Threshold segmentation, Clustering-Based Decision Forests (CBDF Algorithm) etc. on the other hand we also discuss eighteen deep learning techniques for segmentation, because capable of automatically extracting high-dimensional features from the input raw data. Marin et al used U-Net and Mask-RCNN for segmentation. Results show that U-net with transfer learning, outclass state of the art sperm segmentation method with 95% overlapping efficiency against hand-segmented masks for sperm head 0.96, acrosome 0.94 and nucleus 0.95. Other methods also show remarkable results like CNN, VGG16, DTL and DMTL etc. In next section we combine discuss machine learning and deep learning methods for classification. Most researcher used CNN, VGG19 and SVM classifier for classification process. Another network included Full Spectrum and Selected Peak Neural Network. To achieve low root, mean square error these models were boosted for transfer function, structure of hidden layer and different combination of training techniques. FSNN shows 93% prediction accuracy for sperm concentration and gave 100% assessments for differentiating the samples of patients and healthy donors. Similarly, the first ResNet18 model reached 100% accuracy on both validation and test sets. The second ResNet18 + Tabular Data model correctly predicted with an accuracy of 100% on the validation set and an average accuracy 88.89 on the test set. The third ResNet34 + Tabular Data model correctly predicted with an accuracy 87.5% on the validation set and 77.78% on the test set. Moreover, different version of YOLO algorithm also plays an important role in sperm cell detection with great efficient and accuracy. Like Dobrovolny et al also use yolov5 for sperm cell detection and achieved very efficient results. The best-performing model was yolov5l. This network achieves a precision of 88.6 per cent, recall of 52.6, and mAP is 72.1. Other networks are achieving lower results. The second-best network is nano, with a mAP of 69.6 and precision of 64.7 per cent. Similarly Sequential deep learning approach significantly outperformed existing Sperm Morphology Analysis (SMA) methods regarding the accuracy, precision, recall, f0.5, and f1 score. Notably, the accuracy of the head, acrosome and vacuole labels using SDNN technique was 90%, 92%, and 89%, respectively. Significantly, Sequential deep learning has never been used to examine sperm morphology, mainly when combined stack network-based deep learning is used. Furthermore, we hypothesize that the ground-breaking SDNN technique can also be applied to address issues related to the SMA in the real world, like fertility clinics, etc.

Acknowledgment

This research received no external funding.

Conflicts of interest

The authors declare no conflict of interest.