Uniform in number of neighbors consistency and weak convergence of $k$NN empirical conditional processes and $k$NN conditional $U$-processes involving functional mixing data

1.   Introduction and motivations

$U$-statistics were first introduced by ^[115] in connection with unbiased estimators, following initial work by ^[106]. In brief, $U$-statistics of order $m$ and kernel $h$ based on a sequence $\left\{X_{i}\right\}_{i = 1}^{\infty}$ of random variables with values in a measurable space $(S, \mathfrak{S})$ and a measurable function $f: S^{m} \rightarrow \mathbb{R}$ are given by

where

$U_n(h)$ is the nonparametric uniformly minimum variance estimator of $\theta = \mathbb E\left(h\left(X_1, \ldots, X_m\right)\right)$. It is the minimizer with respect to $\alpha$ of

Empirical variance, Gini's mean difference, and Kendall's rank correlation coefficient are common examples of estimators based on $U$-statistics. The Wilcoxon signed rank test for the hypothesis of the location at zero is a classical test based on $U$-statistics, as discussed by ^[190] in Example 12.4. Asymptotic results for the case of independent and identically distributed underlying random variables were first provided by ^[115], who also referred to related work by ^{[57,83,172,176]}. Similar results were obtained for $V$-statistics by ^[94,198]. Extensive literature on the theory of $U$-statistics has been developed, as reviewed by ^[10,138,176], and others. A detailed review and major historical developments in this field can be found in the book by ^[20]. $U$-processes are sets of $U$-statistics indexed by a family of kernels, which can be viewed as infinite-dimensional variants of $U$-statistics with a single kernel or as stochastic processes that are nonlinear extensions of empirical processes. $U$-processes have been applied to solve complex statistical problems such as density estimation, nonparametric regression tests, and goodness-of-fit tests. Considering a large group of statistics instead of a single statistic is more statistically interesting, and ideas from the theory of empirical processes can be used to construct limit or approximation theorems for $U$-processes. However, obtaining results for $U$-processes is not easy and requires significant effort and distinct methodologies. Generalizing from empirical processes to $U$-processes is particularly difficult, especially in the stationary setting. $U$-processes appear in statistics in many instances, such as the components of higher-order terms in von Mises expansions, and play a role in analyzing estimators (including function estimators) with varying degrees of smoothness. For instance, the product limit estimator for truncated data is analyzed in ^[183] using a.s. uniform bounds for $\mathbb{P}$-canonical $U$-processes. In addition, ^[11] introduce two new tests for normality based on $U$-processes, while ^[173] use weighted $L_{1}$-distances between the standard normal density and local $U$-statistics based on standardized observations to propose new tests for normality, utilizing the results of ^[101]. Moreover, in ^[122], the median-of-means approach, which is based on $U$-statistics, is introduced to estimate the mean of multivariate functions in case of possibly heavy-tailed distributions. $U$-processes play a significant role in various statistical applications, including testing for qualitative features of functions in nonparametric statistics ^{[1, 100]}, cross-validation for density estimation ^[157], and establishing limiting distributions of $M$-estimators ^{[10, 68, 10]}. ^[177] provide necessary and sufficient conditions for the law of large numbers and sufficient conditions for the central limit theorem for $U$-processes. For further references on $U$-statistics and $U$-processes, interested readers may refer to ^{[28,40,46,47,48,54,138,179]}, while a comprehensive insight into the $U$-processes theory is provided by ^[68]. $U$-statistics are also naturally found in other contexts, such as the theory of random graphs, where they count occurrences of specific subgraphs like triangles, as presented in ^[120]. In machine learning, $U$-statistics arise naturally in various problems such as clustering, image recognition, ranking, and learning on graphs, where natural risk estimates take the form of $U$-statistics, as discussed in ^[63]. For instance, the empirical ranking error of any given prediction rule is a $U$-statistic of order $2$, as stated in ^[62]. For $U$-statistics with random kernels of diverging orders, readers may refer to ^{[97,112,178,180]}. Infinite-order $U$-statistics are also useful for constructing simultaneous prediction intervals that quantify the uncertainty of ensemble methods like subbagging and random forests, as presented in ^[161]. The MeanNN approach estimation for differential entropy, introduced by ^[90], is a particular application of the $U$-statistic. Additionally, ^[143] proposed a new test statistic for goodness-of-fit tests using $U$-statistics. Moreover, ^[65] have explored a model-free approach for clustering and classifying genetic data based on $U$-statistics, leading to alternative ways of addressing genetic problems. Their motivation was based on the versatility and adaptability of $U$-statistics to various genetic problems and different data types. ^[140] proposed using the $U$-statistics, in a natural way, for analyzing random compressed sensing matrices in the non-asymptotic regime. Extending the above exploration to conditional $U$-processes is practically useful and technically more challenging.

We first introduce Stute's estimators. Let us consider regular sequence of random elements $\{(\mathbf{ X}_i, Y_i), i\in \mathbb{N}^*\}$ with $\mathbf{ X}_i \in \mathbb{R}^d$ and $Y_i \in \mathcal{Y}$ some polish space and $\mathbb{N}^* = \mathbb{N} \backslash\{0\}$. Let $\varphi: \mathcal{Y}^m\rightarrow \mathbb{R}$ be a measurable function. In this paper, we are primarily concerned with the estimation of the conditional expectation, or regression function, for $\mathbf{ t} \in \mathbb{R}^{dm}$,

whenever it exists, i.e.,

We now introduce a kernel function $K:\mathbb{R}^d\rightarrow \mathbb{R}$ with support contained in $[-B, B]^d$, $B > 0,$ satisfying:

^[182] introduced a class of estimators for $r^{(m)}(\varphi, \mathbf{ t})$, called conditional $U$-statistics, which is defined for each $\mathbf{ t}\in\mathbb{R}^{dm}$ to be :

where

is the set of all $m$-tuples of different integers between $1$ and $n$ and $\{h_K: = h_{n}\}_{n\geq1}$ is a sequence of positive constants converging to zero at the rate $nh_{K}^{m} \rightarrow \infty$. In the particular case $m = 1$, the $r^{(m)}(\varphi, t)$ is reduced to

and Stute's estimator becomes the Nadaraya-Watson ^{[153, 200]} estimator of $r^{(1)}(\varphi, t)$, refer to ^[50] for details. The work of ^[175] was devoted to estimating the rate of the uniform convergence in $\mathbf{ t}$ of $\widehat{r}_{n}^{(m)}(\varphi, \mathbf{t}; h_{K})$ to $r^{(m)}(\varphi, \mathbf{ t})$. In ^[165], the limit distributions of $\widehat{r}_{n}^{(m)}(\varphi, \mathbf{t}; h_{K})$ are analyzed and compared to those derived by Stute. Under proper mixing settings, ^[111] extended the results of ^[182] to weakly dependent data and applied their findings to validate the Bayes risk consistency of the corresponding discrimination rules. As alternatives to the conventional kernel-type estimators, ^[186] offered symmetrized closest neighbor conditional $U$-statistics. ^[98] evaluated the functional conditional $U$-statistic and determined its finite-dimensional asymptotic normality. Nonparametric estimate of the conditional $U$-statistics in a functional data context has gotten comparatively little attention despite the subject's significance. Recent developments are described in ^{[42,44,45,52,56]}, in which the authors examine challenges associated with the uniform in bandwidth consistency in general settings. ^[119] evaluated the test of independence in the functional framework using the Kendall statistics, which may be viewed as special cases of the $U$-statistics. ^[14] introduced a comprehensive framework for clustering within multiple groups, employing a U-statistics-based approach specifically designed for high-dimensional datasets. This method classifies data into three groups while evaluating the significance of these partitions. In a related context, ^[128] focused on dimension-agnostic inference, devising methods whose validity remains independent of assumptions regarding dimension versus sample size. Their approach utilized variational representations of existing test statistics, incorporating sample splitting and self-normalization to yield a refined test statistic with a Gaussian limiting distribution. This modification of degenerate U-statistics involved dropping diagonal blocks and retaining off-diagonal blocks. Exploring further, ^[59] delved into U-statistics-based empirical risk minimization, while ^[121] examined asymmetric U-statistics based on a stationary sequence of $m$-dependent variables, with applications motivated by pattern matching in random strings and permutations. Additionally, ^[188] developed innovative $U$-statistics considering left truncation and right censoring. As an application, they proposed a straightforward non-parametric test for assessing the independence between time to failure and the cause of failure in competing risks, particularly when observations are subject to left truncation and right censoring. In a different context, ^[136] investigated the quadruplet U-statistic, offering applications in statistical inference for network analysis. It will be interesting to find connection of the U-statistics with the problems investigated in ^{[187,201,204,205]}. The extension of the preceding investigation to conditional empirical $U$-processes is both practically beneficial and technically difficult.

Recently, there has been a growing interest in regression models in which the response variable is real-valued and the explanatory variable is represented by smooth functions that vary arbitrarily between repeated observations or measurements. This form of data, known as functional data, appears in numerous disciplines, such as climatology (hourly concentration of pollutants), medicine (the knee angle of children as functions of time), economics, linguistics, etc. Functional time series are commonly encountered in practice, for example, when a long continuous time process is divided into smaller natural units, such as days. In this instance, every intraday curve is a functional random variable. This paper focuses mostly on the instance of functional data and the theory of the $U$-processes. We give an excerpt from ^[6]: Functional data analysis (FDA) is a branch of statistics that focuses on the study of variables having an unlimited number of dimensions, such as curves, sets, and images. It has had spectacular growth over the past two decades, fueled partly by technological advances that have led to the "Big Data" revolution. FDA is today one of the most active and significant disciplines of research in data science, despite its reputation at the turn of the century as a fairly obscure area of study. The reader is recommended to the works of ^[167,168], ^[91] for an introduction to this field. These sources offer a variety of case studies in numerous fields, including criminology, economics, archaeology, and neurophysiology, as well as basic analysis techniques. It should be noted that the extension of probability theory to random variables with values in normed vector spaces (such as Banach and Hilbert spaces), as well as extensions of certain classical asymptotic limit theorems, predates the recent literature on functional data; the reader is referred to ^[7]. ^[99] examined density and mode estimates for data occupying a normed vector space. This study examines the topic of the curse of dimensionality for functional data and proposes solutions to the problem. In the context of regression estimation, ^[91] examined nonparametric models. We may also refer to ^[21,116]. ^[130] provided a nice mix of foundational material, accessible theory, and practical examples. Recently, the contemporary theory was used in the analysis of functional data. ^[93], who has provided the consistency rates of several functionals of the conditional distribution, such as the regression function, the conditional cumulative distribution, the conditional density, and others, uniformly over a subset of the explanatory variable. ^[125] established the consistency rates for some functionals nonparametric models, such as the regression function, the conditional distribution, the conditional density, and the conditional hazard function, uniformly in bandwidth (UIB consistency). ^[35] extended these results to the ergodic setting. ^[12] examined the issue of local linear estimation of the regression function when the regressor is functional and demonstrated strong convergence (with rates) uniformly in bandwidth parameters. ^[141] examined the $k$-nearest neighbours (kNN) estimate of the nonparametric regression model for strong mixing functional time series data and demonstrated the uniform nearly complete convergence rate of the $k$NN estimator under several moderate conditions. ^[30] provided several limiting law results for the conditional mode in the functional setting for ergodic data; for more recent references, see ^{[3,4,5,53,55,151]}.

The primary objective of this study is to examine a generic framework and characterize the weak convergence and the uniform convergence of $k$NN conditional $U$-processes based on a regular sequence of random functions. This is motivated by the fact that the $k$-NN method is a fundamental statistical method presenting several advantages. Recall that the $k$-NN method considers the $k$ neighbors of $X_{i}$ nearest to $x$ with respect to some distance $d(\cdot, \cdot)$. Although the local bandwidth of the $k$-NN is random and depends on the data $X_{i}$ respecting the local structure of the data, which is essential in the infinite dimension. Historically, the $k$-NN was first introduced by ^[95]-see also ^[96]-in the context of nonparametric discrimination, and further investigated by ^[144], for more details, we refer to ^[17]. It is commonly used in practice (see ^[91]) and is simple to handle because the user has only one parameter to control the number $k$ of nearest neighbors, valued in a finite set. In addition, it allows us to build a neighbor adapted to the data at any point. The $k$-NN method is widely studied if the explanatory variable is an element of a finite-dimensional space, for instance, see ^{[16,64,77,104,145]}. In an infinite dimensional space, i.e., a functional framework, there are three different approaches for the $k$-NN regression estimation. The first, published by ^[135], examines a $k$-NN kernel estimate when the functional variable is an element of a separable Hilbert space $H$. In this approach, ^[135] established a weak consistency result. The strategy of ^[135] is to reduce the infinite dimension of $H$ by using a projection on a finite dimension subspace by considering only the first $m$ coefficients of an expansion of $X$ in an orthonormal system of $H$ and then applying the multivariate techniques on the projected data to perform the $k$-NN regression. The second approach is based on the $k$-NN procedure and the functional local linear estimation, the consistency with the convergence rate is obtained in ^[60] and ^[142].

More precisely, in this paper, we are interested in establishing the a.co uniform consistency and the a.co uniform in the number of neighbors (UINN) consistency of the nonparametric functional regression estimator and also the functional conditional $U$-processes (statistics). ^[157] were the first to introduce the notion of uniform in bandwidth consistency for kernel density estimators and they applied empirical process methods in their study. This is motivated by a series of papers, among many others, ^{[15,25,33,36,37,38,42,44,51,52,54,71,78,87,88]} the authors established uniform in bandwidth (UIB) consistency results for such estimators in the i.i.d. finite-dimensional setting, where $h_n$ varies within suitably chosen intervals indexed by $n$. In the FDA, several authors have been interested in studying non-parametric functional estimators. For example, ^[93] provided the consistency rates of some functionals of the conditional distribution, including the regression function, the conditional cumulative distribution, the conditional density, and some others, uniformly over a certain subset of the explicative variable. ^[125] established the uniform consistency rate for some conditional models, including the regression function, the conditional distribution, the conditional density, and the conditional hazard function. The last mentioned paper is extended by ^[42]. ^[124] and ^[3] established the almost complete convergence of the $k$-nearest neighbors ($k$-NN) estimators, which are uniform in the number of neighbors, under some classical assumptions on the kernel and on the small ball probabilities of the functional variable in connection with the entropy condition controlling the space complexity. ^[12] considered the problem of local linear estimation of the regression function when the covariate is functional and proved the strong uniform-in-bandwidth (UIB) convergence. ^[141] investigated the $k$-NN estimation of the nonparametric regression model for strong mixing functional time series data and established the uniform a.co convergence rate of the $k$-NN estimator under some mild conditions. ^[158] stated some new uniform asymptotic results for kernel estimates in the functional single-index model. Most of this literature focuses on UIB or UINN consistency or on uniform consistency on some functional subset but never the both together, which was investigated in ^[44] in the independent framework. We aim to fill this gap in the literature by combining results from the FDA and the empirical processes theory in the dependent setting. The second problem for the weak convergence that we investigate is not simple, and the main merits of our contribution are the control of the asymptotic equi-continuity under minimal conditions in this general setting, which constitutes a fundamentally unresolved open problem in the literature. We intend to fill this gap in the literature by integrating the results of ^[8] and ^[27] with the strategies described in ^[147] and ^[43] for handling functional data. But, as will be demonstrated in the following section, the challenge requires much more than "just" merging concepts from the current outcomes. In reality, intricate mathematical derivations will be necessary to deal with the typical functional data in our framework. This necessitates the effective application of large sample theory tools, which were established for dependent empirical processes and for which we have used results from the work of ^[8,27,43]. Even with i.i.d. functional data, no weak convergence for the $k$NN conditional $U$-processes has been proven up to the present.

1.1.   Paper contribution

The current paper looks into the challenges high-dimensional functional data presents and presents important findings that are applicable to high-dimensional data models. This research expands the classical kernel estimator beyond samples that are independent and identically distributed to include stationary random processes. In addition, the main focus is on the $k$NN kernel estimator. It specifically examines four fundamental aspects that pertain to the $k$NN kernel estimator for regression and its uniform consistency. The study investigates the UINN and UIB scenarios to guarantee consistent results within the domain of functional regression (the UIB in Theorem 3.3 and the UINN in Theorem 3.1). The outcomes include the prediction of relative error, which enhances the overall comprehension of the $k$NN kernel estimator's performance (the UIB in Corollary 3.6 and and the UINN in Corollary 3.5). The paper includes the $k$NN method's for functional conditional $U$-statistics with consistent results, thereby extending the application area of these estimators. The study additionally investigates the uniform consistency and UINN consistency of functional conditional $U$-statistics, offering a comprehensive assessment of their limiting behaviour (the UIB in Theorems 3.8–3.10 and Corollary 3.12 and the UINN in Theorems 3.14, 3.16 and Corollary 3.18). In addition, the study makes an advanced contribution to the field by establishing a uniform central limit theorem for classes of functions that, subject to specific moment conditions, are either bounded or unbounded (for the conditional process the Normality is provided in Theorem 4.1 and the equicontinuity in Theorem 4.2; for the conditional $U$-process the Normality is provided in Theorems 4.5, 4.6 and the equicontinuity in Theorem 4.7). The process of establishing our main results uses advanced methodologies, including the k-nearest neighbours ($k$NN) method, covering number, small-ball probability, the Hoeffding decomposition, the decoupling methods, and the modern theory of the empirical process indexed by functions. At this stage, we mention that the Hoeffding decomposition cannot be used directly in our setting and needs some intricate preparation. All these results are established under fairly general conditions on function classes and underlying distributions, the majority of which are derived from prior works, thus guaranteeing their feasibility. The obtained results have the potential to be utilised in numerous statistical domains, such as time series prediction, set-indexed conditional U-statistics, and the Kendall rank correlation coefficient. Key technical tools in the proofs are the maximal moment inequalities for $U$-processes, and ^[84]'s results on $\beta$-mixing.

1.2.   Paper Organization

The layout of the present article is as follows. Section 2 is devoted to introducing the functional framework and the definitions that we need in our work, we give the assumptions used in our asymptotic analysis with a short discussion. Section 3 is devoted to the strong uniform convergence with rate. Section 4.1 provides the weak convergence of empirical processes in the functional framework. Section 4.2 gives the main results of the paper concerning the uniform TCL for the conditional $U$-processes. In Section 5, we collect some potential applications, including the set indexed conditional U-statistics in Section 5.1, Kendall rank correlation coefficient in Section 5.2, the discrimination problems in Section 5.3 and the time series prediction from a continuous set of past values in Section 5.4. We discuss a bandwidth choice for practical use in Section 6. Some concluding remarks and possible future developments are relegated to Section 7. To prevent interrupting the flow of the presentation, all proofs, based upon modern empirical process theory, are gathered in Section 8. Due to the lengthiness of the proofs, we limit ourselves to the most important arguments. A few relevant technical results are given in the Appendix.

2.   The functional framework

2.1.   Generality on the model

Let $\{\left(X_i, Y_i\right): i\geq 1\}$ be a sequence of stationary^† random copies of the random vector [rv] $(X, Y)$, where $X$ takes its values in some abstract space $\mathcal{X}$ and $Y$ in the abstract space $\mathcal{Y}.$ Suppose that $\mathcal{X}$ is endowed with a semi-metric $d(\cdot, \cdot)$^‡ defining a topology to measure the proximity between two elements of $\mathcal{X}$ and which is disconnected from the definition of $X$ to avoid measurability problems. We are mainly interested in establishing the weak convergence of the conditional $U$-process based on the following $U$-statistic in the $k$-NN setting introduced in ^[44] by

^†In the case of the Hilbert space valued elements not necessarily strictly stationary is needed, a second order stationarity suffices. An Hilbert space valued sequence $\left\{X_{t}\right\}_{t \in \mathbb{Z}}$ is second-order (or weakly) stationary if $\mathbb{E}\left\|X_{t}\right\|^{2} < \infty, \mathbb{E} X_{t}=\mu$, and

for all $s, t \in \mathbb{Z}$. We say that $\left\{X_{t}\right\}_{t \in \mathbb{Z}}$ is strictly stationary if the joint distribution of $\left\{X_{t_{1}}, \ldots, X_{t_{n}}\right\}$ and the joint distribution of $\left\{X_{t_{1}+h}, \ldots, X_{t_{n}+h}\right\}$ coincide, for all $t_{1}, \ldots, t_{n} \in \mathbb{Z}, n \geq 1$, and $h \geq 1$.

^‡A semi-metric (sometimes called pseudo-metric) $d(\cdot, \cdot)$ is a metric which allows $d(x_{1}, x_{2}) = 0$ for some $x_{1}\neq x_{2}$.

as an estimator for the multivariate regression function

where

and $\varphi: \mathcal{Y}^m = \mathcal{Y} \times \ldots \times \mathcal{Y} \longrightarrow \mathbb{R}$ is a symmetric measurable function belonging to some class of functions $\mathscr{F}_m,$ and $\mathbf{h}_{n, k}(\mathbf{t}) = \left(H_{n, k}(t_{1}), \ldots, H_{n, k}(t_{m})\right)$ is a vector of positive random variables that depend on $\left(X_{1}, \ldots, X_{n}\right)$ such that, for all $\mathbf{x} = (x_1, \ldots, x_m) \in \mathcal{X}^m$ and $j = 1, \ldots, m:$

where $B(t, r) = \left\{z \in \mathcal{X}: d(z, t) \leqslant r\right\}$ is a ball in $\mathcal{X}$ with the center $t \in \mathcal{X}$ and radius $r$, and $\mathbb{1}_{A}$ is the indicator function of the set $A.$ In fact, this $k$-NN estimate can be considered as an extension to the random and locally adaptive neighbor of the functional conditional $U$-statistics estimate of $r^{(m)} (\varphi, \mathbf{t})$ defined for all $t \in \mathcal{X}^m$ as :

where $h_{K}(\mathbf{t}) = (h_{K, n}(t_1), \ldots, h_{K, n}(t_m)) = :(h_{K}(t_1), \ldots, h_{K}(t_m))$ are positive real numbers decreasing to zero as $n$ goes to infinity. At this stage, we highlight that kernel estimation is popular since the classical Akaike-Parzen-Rosenblatt kernel density estimation [refer to ^[2,160,171]]. However, the first appearance of kernel estimators is likely to be ^[95]: as the original technical report is difficult to find, it has been re-published as ^[96]. ^[160] has shown, under some assumptions on $K(\cdot)$, that $f_n(\cdot)$ is an asymptotically unbiased and consistent estimator for $f(\cdot)$ whenever $h_n \rightarrow 0, n h_n^d \rightarrow \infty$ and $\tilde{\mathbf{x}}$ is a continuity point of $f(\cdot)$. Under some additional assumptions on $f(\cdot)$ and $h_n$, he obtained an asymptotic normality result, too. The kernel estimators have been extensively studied in the literature, see, e.g., ^{[28,31,34,49,73,75,76,85,86,108,154,174,199]} and the references therein. The $k$-NN method is a fundamental statistical tool with various advantages. Generally, the procedure is computationally efficient and requires minimal parameter adjustment. In addition, the $k$-NN approaches are nonparametric, which allows them to automatically adapt to any continuous underlying distributions without relying on any particular models. The $k$-NN techniques were proven consistent for various significant statistical problems, including density estimation, classification, and regression, provided that a suitable $k$ is chosen. It should be noted that since our objective is to generalize the results obtained for the estimator defined in (2.5), and given the fact that one of the main differences is that the smoothing parameter, $\mathbf{h}_{n, k}(\mathbf{t})$ is a vector of random variables instead of a univariate parameter $h_K$ our first course of action would be to extend the results of ^[42,43] to the multivariate setting. First, we need to introduce some notation. Let $\mathscr{F}_{m} = \{\varphi: \mathcal{Y}^{m} \longrightarrow \mathbb{R}\}$ denote a pointwise measurable class of real-valued symmetric functions on $\mathcal{Y}^m$ with a measurable envelope function

For a kernel function $K(\cdot)$ and a subset $S_{\mathcal{X}}\subset \mathcal{X}$, we define the pointwise measurable class of functions, for $1\leq m \leq n$ :

Statistical observations are not always independent but are often close to being so. Dependence may lead to severe repercussions on statistical inference if it is not taken into consideration. The notion of mixing quantifies how close to independence a sequence of random variables is, allowing us to extend standard results for independent sequences to weakly dependent or mixing sequences. Let us specify the dependence that will be the focus of this study. Let $\mathbf{Z}_1, \mathbf{Z}_2, \ldots$ be a stationary sequence of random variables on some probability space $\left(\Omega, \mathcal{D}, \mathbb P\right)$ and let $\sigma_i^j$ be the $\sigma$-field generated by $\mathbf{Z}_i, \ldots, \mathbf{Z}_j,$ for $i, j \geqslant 1.$ The sequence $\mathbf{Z}_1, \mathbf{Z}_2, \ldots$ is said $\beta$-mixing or absolute regular, refer to ^[170,197], if :

It should be noted that ^[118] obtained a complete description of stationary Gaussian processes satisfying the last property. Throughout the sequel, we assume tacitly that the sequence of random elements $\{(X_i, Y_i), i\in \mathbb{N}^*\}$ is absolutely regular. The Markov chains, for instance, are $\beta$-mixing under the milder Harris recurrence condition if the underlying space is finite ^{[19,46,66,180]}. We also need to introduce some concepts that are related to the topological structure of functional spaces. First, we define the small-ball probability for a fixed $t \in \mathcal{X}$ and for all $r > 0$ by

this notion is widely used in nonparametric functional data analysis to avoid introducing density assumptions on the functional variable ${X}$ and address the issues associated with the infinite-dimensional nature of the functional spaces. At this point, we can refer to ^[91,99,147]. We also need to deal with the VC-subgraph classes ("VC" for Vapnik and Chervonenkis, for instance, see ^{[132,193,194]}).

Definition 2.1. A class of subsets $\mathcal{C}$ on a set $C$ is called a VC-class if there exists a polynomial $\mathfrak P(\cdot)$ such that, for every set of $N$ points in $C$, the class $\mathcal{C}$ picks out at most $\mathfrak P(N)$ distinct subsets.

Definition 2.2. A class of functions $\mathscr{F}$ is called a VC-subgraph class if the graphs of the functions in $\mathscr{F}$ form a V-C class of sets, that is, if we define the subgraph of a real-valued function $f$ on $S$ as the following subset $\mathcal{G}_f$ on $S \times \mathbb{R}$

the class $\{\mathcal{G}_f : f \in \mathscr{F}\}$ is a VC-class of sets on $S \times \mathbb{R}$. Informally, a VC-class of functions is characterized by having a polynomial covering number (the minimal number of required functions to make a covering on the entire class of functions).

Definition 2.3. Let $\mathcal{S}_{\mathcal{E}}$ be a subset of a semi-metric space $\mathcal{E}$ and $N_{\varepsilon}$ a positive integer, a finite set of points $\{e_1, \ldots, e_{N_{\varepsilon}}\}\subset \mathcal{E}$ is called, for a given $\varepsilon > 0$, a $\varepsilon$-net of $\mathcal{S}_{\mathcal{E}}$ if:

If ${N_{\varepsilon}}(\mathcal{S}_{\mathcal{E}})$ is the cardinality of the smallest $\varepsilon$-net (the minimal number of open balls of radius $\varepsilon$) in $\mathcal{E},$ needed to cover $\mathcal{S}_{\mathcal{X}}$, then we call Kolmogorov's entropy (metric entropy) of the set $\mathcal{S}_{\mathcal{E}},$ the quantity

From its name, one can figure that this concept of metric entropy was introduced by Kolmogorov ^[131] and was studied subsequently for numerous metric spaces. This concept was used by ^[80] to give sufficient conditions for continuity of Gaussian processes, and was the basis for striking generalizations of Donsker's theorem on the weak convergence of the empirical process. Suppose that $\mathcal{B}_\mathcal{X}$ and $\mathcal{S}_{\mathcal{X}}$ are two subsets of the semi-metric space $\mathcal{X}$ with Kolmogorov's entropy (for the radius $\varepsilon$) $\psi_{\mathcal{B}_{\mathcal{X}}}(\varepsilon)$ and $\psi_{\mathcal{S}_{\mathcal{X}}}(\varepsilon)$ respectively, then the Kolomogorov entropy for the subset $\mathcal{B}_\mathcal{X}\times \mathcal{S}_\mathcal{X}$ of the semi-metric space $\mathcal{X}^2$ by :

Hence, $m\psi_{\mathcal{S}_{\mathcal{X}}}(\varepsilon)$ is the Kolmogorov entropy of the subset $\mathcal{S}_{\mathcal{X}}^m$ of the semi-metric space $\mathcal{X}^m$. Noting that if we designate by $d$ the semi-metric on $\mathcal{X}$, then, we can define the semi metric on $\mathcal{X}^m$ by :

for

Notice that the semi-metric plays an important role in this kind of study. The reader will find useful discussions about how to choose the semi-metric in ^[91] (see Chapters 3 and 13).

2.2.   Conditions and comments

Let us present the conditions that we need in our analysis.

(C.1.) On the distributions/small-ball probabilities

(C.1.1) For $\mathbf{t} = (t_1, \ldots, t_m) \in {\mathcal{X}}^m$ and $\mathbf{h(t)} = (h_1(t_1), \ldots, h_m(t_m)) \in \mathbb{R}_{+}^m \backslash\left\{0\right\}$, we have

where $f_1(\mathbf{t})$ is a non-negative functional in $\mathbf{t} = (t_1, \ldots, t_m) \in {\mathcal{X}}^m$, $\tilde{\phi}(\mathbf{h}): = \prod\limits_{j = 1}^m \phi(h_j(t_j))$ and $\phi(0) = 0$ and $\phi(u)$ is an invertible function absolutely continuous in a neighbor of the origin.

(C.1.2) For $i\neq j,$ let $\mathbf{X_i} = \left(X_{i_1}, \ldots, X_{i_m}\right), \, \mathbf{X_j} = \left(X_{j_1}, \ldots, X_{j_m}\right) \; \mbox{and}\; \mathbf{t} = \left(t_1, \ldots, t_m\right) \in \mathcal{X}^m$, we have

where $f_2\left(\mathbf{t}\right)$ is a non-negative function, $\tilde{\Psi}(\mathbf{h(t)}): = \prod\limits_{i = 1}^m \psi(h_i(t_i))$ and $\tilde{\Psi}(\mathbf{h(t)})\to 0$ as $n\to \infty,$ satisfying $\tilde{\Psi}(\mathbf{h(t)})\left/\tilde{\phi}^2(\mathbf{h(t)})\right.$ is bounded.

(C.2) On the smoothness of the model

(C.2.1) The regression satisfies for $\varphi(\cdot) \in \mathscr{F}_m,$ and for $1\leq m \leq n:$

where for $1\leq i\leq 2 : \mathbf{t}_i: = (t_{i_1}, \ldots, t_{i_m})$ and the semi metric $d_{\mathcal{X}^m}\left(\cdot, \cdot\right)$ on $\mathcal{X}^m$:

for $\mathbf{x} = (x_1, \ldots, x_m), \mathbf{y} = (y_1, \ldots, y_m) \in\mathcal{X}^m.$

(C.2.2) The conditional variance, defined for $u \in \mathcal{X}^m, {\rm Var}\left[\varphi\left(\mathbf{Y}\right)|\mathbf{X} = \mathbf{u}\right] = :g_2\left(\mathbf{u}, \varphi\right)$ is continuous in some neighborhood of $\mathbf{t}$

Further, assume that for some $p > 2, \; \mathbb{E}\left|F\left(\mathbf{Y}\right)\right|^p < \infty,$ and $\varphi(\cdot) \in \mathscr{F}_m,$

is continuous in some neighborhood of $\mathbf{t}.$

(C.2.3) For $\mathbf{u}, \mathbf{v} \in \mathcal{X}^m,$ the function $g_\varphi\left(\mathbf{t}, \mathbf{u}, \mathbf{v}\right)$ does not depend on $\mathbf i, \mathbf {j}$ and is continuous in some neighborhood of $\left(\mathbf{t}, \mathbf{t}\right)$

(C.3) On the kernel function

(C.3.1) The kernel functions $K(\cdot)$ is supported within $[0, 1]$ and there exists some constants $0 < \kappa_1\leq \kappa_2 < \infty,$ such that

and

(C.3.2) The kernel $K(\cdot)$ is a positive function and differentiable function on $[0, 1\left]\right.$ with derivative $K^{\prime}(\cdot)$ such that

(C.4) On the classes of functions

(C.4.1) The class of functions $\mathscr{F}_m$ is bounded and its envelope function satisfies for some $0 < M < \infty:$

(C.4.2) The class of functions $\mathscr{F}_m$ is unbounded and its envelope function satisfies for some $p > 2:$

(C.4.3) The metric entropy of the class $\mathscr{F}_m\mathscr{K}^m$ satisfies, for some $\; 1\leq p < \infty$ :

where

(C.4.4) The class of functions $\mathscr{F}_m\mathscr{K}^m$ is supposed to be of VC-type with envelope function previously defined. Hence, there are two finite constants $b$ and $\nu$ such that:

for any $\epsilon > 0$ and each probability measure such that $Q(F)^2 < \infty$.

(C.5) On the dependence of the random variables

(C.5.1) Absolute regularity

for some $p > 2$ and $\delta > 1-2/p.$

(C.5.2) There is a sequence of positive integers $\left\{s_n\right\}_{n \in \mathbb{N}^*}$ such that, as $n \to \infty$,

(C.6.) On the entropy

For $n$ large enough and for some $\omega > 1$, the Kolomogorov's entropy satisfies :

where $\psi_{S_{\mathcal{X}}}(\varepsilon): = \log{N_{\varepsilon}}(S_{\mathcal{X}}),$ and ${N_{\varepsilon}}(S_{\mathcal{X}})$ is the minimal number of open balls of radius $\varepsilon$ in $\mathcal{X}$, needed to cover $S_\mathcal{X}.$

(C.7.) The sequences $\{\widetilde h_{n}\}$ and $\{\widetilde {h'}_{n}\}$ (resp. $\{ h_{n, 1}\}$ and $\{ h_{n, 2}\}$) verify

(C.8.) There exist sequences $\left\{\rho_{n, 1}, \ldots, \rho_{n, m}\right\} \subset (0, 1)^m$, $\left\{k_{1, n}\right\} \subset \mathbb{Z}^+$ and $\left\{k_{2, n}\right\} \subset \mathbb{Z}^+$ ($k_{1, n} \leq k \leq k_{2, n}$) and constants $\boldsymbol{\mu} = (\mu_1, \ldots, \mu_m)\; \; \mbox{and}\; \; \boldsymbol{\nu} = (\nu_1, \ldots, \nu_m)$, such that

and

Additional/alternative conditions

(C.1'.) For $\mathbf{X_i} = \left(X_{i_1}, \ldots, X_{i_m}\right), \, \mathbf{X_j} = \left(X_{j_1}, \ldots, X_{j_m}\right) \; \mbox{and}\; \mathbf{t} = (t_1, \ldots, t_m) \in {\mathcal{X}}^m:$

(C.1'.1) We have:

with $\mathbf{x} = (x_1, \ldots, x_m) \in \mathbb{R}_{+}^{m}$, $\phi(0) = 0,$ and $\phi(u)$ is absolutely continuous in a neighborhood of the origin.

(C.1'.2) we have:

where $f_2\left(\mathbf{t}\right)$ is a non-negative function, $\tilde{\Psi}(\mathbf{h(t)}): = \prod\limits_{i = 1}^m \psi(h_i(t_i))$, and $\tilde{\Psi}(\mathbf{h(t)})\to 0$ as $n\to \infty,$ satisfying $\tilde{\Psi}(\mathbf{h(t)})\left/\tilde{\phi}^2(\mathbf{h(t)})\right.$ is bounded.

(C.3'.) The kernel functions $K(\cdot)$ is supported within $[0, 1]$ and there exists some constants $0 < \kappa_2,$ $0 < \kappa'_1\leq \kappa'_2 < \infty,$ such that for $j = 1, 2$:

Comments

In our nonparametric functional regression model, we deal with a complex theoretical challenge. Mainly establishing functional central limit theorems for the conditional empirical processes and the conditional U-process under functional absolute regular data. We also use random (or data-dependent) bandwidths based on the $k$ nearest neighbors (kNN) approaches. Although standard statistical methods cannot be utilized in the functional setting, most imposed conditions overlap with some characteristics of the infinite-dimensional spaces, such as the topological structure of $\mathcal{X}^m$, the probability distribution of ${X}$, and the measurability concept for the classes $\mathcal{F}_m$ and $\mathcal{K}^m$. It is worth mentioning that most of the conditions that we will be using throughout this paper are inspired by ^{[43,91,99,141,147]}. Let us start with assumption (C.1.1), which was adapted from ^[147], who in turn was inspired by ^[99]. As explained by ^[147], if $\mathcal{X}^m = \mathbb{R}^m$, then condition (C.1.1) coincides with the fundamental axioms of probability calculus. Furthermore, if $\mathcal{X}^m$ is an infinite-dimensional Hilbert space, then $\phi(h_K)$ can converge to $0$ exponentially as $n\rightarrow \infty$. Condition (C.1.1) can be considered a standard condition on the small ball probability, which is used to control the behavior of $\phi_{\mathbf{t}}(\cdot)$ around zero. It shows that we can approximately write the small ball probability as a product of two independent functions $\tilde{\phi}(\cdot)$ and $f_1(\cdot)$; see, for instance, ^[148] for the diffusion process, ^[18] for a Gaussian measure, and ^[139] for a general Gaussian process. The most frequent result available in the literature is of the form $\varphi_{t}(\varepsilon)\sim g(t) \phi(\varepsilon)$ where $\phi(\varepsilon) = \varepsilon^{\gamma} \exp \left(-C / \varepsilon^{p}\right)$ with $\gamma \geq 0$ and $p \geq 0$. It corresponds to the Ornstein-Uhlenbeck and general diffusion processes (for such processes, $p = 2$ and $\gamma = 0$) and the fractal processes (for such processes, $\gamma > 0$ and $p = 0$). For more examples, refer to ^[92]. It is worth noting that, in general, when we deal with functional data, we need some information about the variability of the small-ball probability to adapt to the bias of nonparametric estimators; this information is usually obtained by supposing that:

(C.1.1'')

Condition (C.2) concerns the regularity of the model; it consists of mild conditions on the continuity of certain conditional moments, in addition to the standard Lipschitz assumption on the regression ((C.2.1). Assumption (C.3) is another classical condition in nonparametric estimation models which concerns the kernel function $K(\cdot)$. It should be noted that condition (C.3.1) can be replaced with condition (C.3') to find an expression for the asymptotic variance. We use condition (C.4.1) when dealing with bounded functions. However, our interest also extends to conditional $U$-processes indexed by an unbounded class of functions. In this case, we replace (C.4.1) by (C.4.2). Keep in mind that there is a trade-off between the moment order $p$ in (C.4.2) and the decay rate of the mixing coefficient $\beta(s)$ imposed in (C.5): the larger $p$ is, the weaker the decay of $\beta(s)$. Also note that if $\beta(s) = \mathrm{e}^{-a s}$, i.e, $\beta(s)$ decays exponentially fast, then (C.5.) is automatically satisfied. Furthermore, this condition is indispensable in our work since studying the weak convergence of the empirical processes entails establishing asymptotic equi-continuity. For Assumption (C.4.4), see [163, Examples 26 and 38], [157, Lemma 22], [82,§ 4.7.], [193, Theorem 2.6.7], [132,§ 9.1] provide a number of sufficient conditions under which (C.4.4) holds, we may also refer to [70,§ 3.2] for further discussions. For instance, it is satisfied, for general $d\geq 1$, whenever $g(\mathbf{x}) = \phi(p(\mathbf{x}))$, with $p(\mathbf{x})$ being a polynomial in $d$ variables and $\phi(\cdot)$ being a real-valued function of bounded variation, we refer the reader to [88, p. 1381]. We also mention that the class of function is assumed to be in general pointwise measurable, that is satisfied whenever $K(\cdot)$ is of bounded variation on $\mathbb{R}^d$ (in the sense of Hardy and Kauser ^{[110,133,195]}, see, e.g., ^{[61,114,156,196]}). The condition $\textbf{(C.6.)}$ takes into account the topological considerations by controlling the Kolmogorov entropy^§ of the set ${S}_\mathcal{X}^m$, which is standard in nonparametric models when we study the uniform consistency and the uniform in bandwidth consistency, we refer to ^[93] and ^[134] for discussions. As mentioned in ^[134], there are special cases of functional spaces $\mathcal{X}$ and subsets $S_{\mathcal{X}}$ where $\psi_{S_{\mathcal{X}}}(\log (n) / n) \gg \log (n)$. Some examples are the closed ball in a Sobolev space, the unit ball of the Cameron-Martin space, and a compact subset in a Hilbert space with a projection semi-metric (see ^[93,131,192], respectively, for further details). In all these cases it is easy to see that $\textbf{(C.6.)}$ is verified as soon as $\beta > 2$. Assumption (C.7.) is essential to establish the rates of convergence (consistency) of the estimator defined in (2.5), while assumption (C.8.) adapts condition (C.7.) to the case of the functional conditional $U$-statistics in the $k$-NN setting.

^§The concept of metric entropy was introduced by Kolmogorov ($cf.$ ^[131]) and studied subsequently for numerous metric spaces. This concept was used by ^[80] to give sufficient conditions for the continuity of Gaussian processes, and was the basis for striking generalizations of Donsker's theorem on the weak convergence of the empirical process, refer to ^[191] for the connection of this notion with Le Cam's work.

Remark 2.4. Note that the condition (C.4.2) may be replaced by more general hypotheses upon moments of $\mathbf{ Y}$ as in ^[70]. That is

(M.1)$^{\prime\prime}$ We denote by $\{\mathcal{M}(x) : x \geq 0\}$ a nonnegative continuous function, increasing on $[0, \infty)$, and such that, for some $s > 2$, ultimately as $x \uparrow \infty$,

For each $t \geq \mathcal{M}(0)$, we define $\mathcal{M}^{inv}(t) \geq 0$ by $\mathcal{M}(\mathcal{M}^{inv}(t)) = t$. We assume further that:

The following choices of $\mathcal{M}(\cdot)$ are of particular interest:

(ⅰ) $\mathcal{M}(x) = x^{p}$ for some $p > 2$;

(ⅱ) $\mathcal{M}(x) = \exp (s x)$ for some $s > 0$.

3.   Uniform consistency

For simplicity reasons, the condition (C.1.1) on the small ball probability will be replaced by:

(H.1) For $\mathbf{h}\in \mathbb{R}^m_+\backslash\{0\}$ and $\mathbf{t}\in \mathcal{X}^m$

this standard condition can be considered an extension of the multivariate case where we assume that the density function of the variable $\mathbf{X}$ is strictly positive. Also, it is worth mentioning that, if in particular, we denote $\mathbf{h}_K: = (h_K, \ldots, h_K) \in (\widetilde h_{n, 1}, \widetilde h_{n, 2})^m$, we can find two positive constants $C_1', C_2'$, such that

which is similar to condition (C.1) used in ^[42,181], so we will deal with $\phi(h_K)$ instead of $\phi^m(h_K),$ (whenever we encounter a similar situation in the proofs). This approach is not only for notational purposes but also to make it easier to bridge the UIB and the UINN results.

3.1.   Uniform consistency of the kNN kernel estimator for regression

In this section, we consider the uniform consistency of the functional regression operator in its general form, which is given for all $t \in \mathcal{X}$, by

where $k = k_{n}$ depending on $n$ and

In fact, the $k$-NN operator presented in (3.3) can be considered as a generalization of the usual kernel regression

where the bandwidth $h_K \in \mathbb{R}_{+}^*$ depends on $n$ (but does not depend on $t$).

3.1.1.   UIB consistency for functional regression

Recall the bandwidths $h_{n, 1}$ and $h_{n, 2}$ given in the condition (C.7.). The following theorem will play an instrumental role in the sequel.

Theorem 3.1. Under the assumptions (H.1.), (C.2.1), (C.3.1), (C.4.1), (C.4.3), (C.6.) and (C.7.) (for $m = 1$), we have, as $n\rightarrow \infty$,

^¶ Let $\left(z_{n}\right)$ for $n \in \mathbb{N}$, be a sequence of real r.v.'s. We say that $\left(z_{n}\right)$ converges almost-completely (a.co.) toward zero if, and only if, for all

Moreover, we say that the rate of the almost-complete convergence of $\left(z_{n}\right)$ toward zero is of order $u_{n}$ (with $u_{n} \rightarrow 0$) and we write $z_{n} = O_{a. c o.}\left(u_{n}\right)$ if, and only if, there exists $\epsilon > 0$ such that

This kind of convergence implies both the almost-sure convergence and the convergence in probability.

The following result gives uniform consistency when the class of functions is unbounded.

Corollary 3.2. Under the assumptions (H.1.), (C.2.1), (C.3.1), (C.4.2), (C.4.3), (C.6.) and (C.7.) (for $m = 1$), we have

3.1.2.   UINN consistency for functional regression

Now, we can state the main results of this section concerning the $k$-NN functional regression. Recall the bandwidths $k_{1, n}$ and $k_{2, n}$ given in the condition (C.8.).

Theorem 3.3. Under the assumptions (H.1.), (C.2.1), (C.3.1), (C.4.1), (C.4.3), (C.6.) and (C.7.) (for $m = 1$), if, in addition, condition (C.8.) is satisfied, then, we have

The following result gives uniform consistency when the class of functions is unbounded.

Corollary 3.4. Under the assumptions (H.1.), (C.2.1), (C.3.1), (C.4.2), (C.4.3), (C.6.) and (C.7.) (for $m = 1$), and if condition (C.8.) is satisfied, then we have

3.2.   Relative-error prediction

Recall that the operator $m$ is usually estimated by minimizing the expected squared loss function $\mathbb E\left[(Y-m(X))^2 \mid X\right]$. Nonetheless, this loss function, which is regarded as a measure of prediction performance, may be inappropriate in certain circumstances. In fact, the application of least-squares regression translates to giving all variables in the study equal weight. Consequently, the prevalence of outliers can render results irrelevant. In this paper, we, therefore, circumvent the limitations of classical regression by estimating the operator $m$ with regard to the minimization of the mean squared relative error (MSRE):

This criterion is clearly a more meaningful measure of the prediction performance than the least square error, in particular, when the range of predicted values is large. Moreover, the solution of (3.8) can be explicitly expressed by the ratio of the first two conditional inverse moments of $Y$ given $X$. In fact, in order to construct the regression estimator allowing the best MSRE prediction, we assume that the first two conditional inverse moments of $Y$ given $X$, that is $g_\gamma(x): = \mathbb E\left(Y^{-\gamma} \mid X = \right.$ $x$) for $\gamma = 1, 2$, exist and are finite almost-surely (a.s.). Then, one can show easily, cf. ^[74,123,159], that the best mean squared relative error predictor of $Y$ given $X$ is:

Thus, we estimate the regression operator $\breve{r}(\cdot)$, which minimizes the MSRE by:

and

By considering the special cases $\varphi(y) = y^{-1}$ and $\varphi(y) = y^{-2}$ in Corollaries 3.2 and 3.4, we obtain the following results complementing the work of ^{[22,23,24,74]}.

Corollary 3.5. Under the assumptions (H.1.), (C.2.1), (C.3.1), (C.4.2), (C.4.3), (C.6.) and (C.7.) (for $m = 1$), we have

The following result is not considered in the literature.

Corollary 3.6. Under the assumptions (H.1.), (C.2.1), (C.3.1), (C.4.2), (C.4.3), (C.6.) and (C.7.) (for $m = 1$), and if condition (C.8.) is satisfied, then we have

3.3.   Uniform consistency of the kNN functional conditional U-statistics

In addition to the conditions imposed before, the following assumptions are essential to obtain exponential inequalities for dependent data later in the proofs:

(A1) Assume $\left\{X_i\right\}_{i \in \mathbb{N}^*}$ is strictly stationary, and there exists an absolute constant $\delta \geq 1$ such that for any $n \geq 1$, we have the $\beta$-mixing coefficient, corresponding to $\left\{X_i\right\}_{i \in \mathbb{N}^*}$, satisfies $\beta(n) \lesssim n^{-\delta}$.

(A2) Assume, uniformly, for any integer $J$ such that $1 \leq J \leq m-1$ and arbitrary $1 \leq i_1 < \cdots < i_J \leq n$, conditional on $X_{i_1}, \ldots, X_{i_J}$, the sequence $\left\{X_i\right\}_{i = i_J+1}^{\infty}$ satisfies, for the $\alpha$-mixing coefficient corresponding to it,

where $\mathbb{P}\left(\cdot \mid X_{i_1}, \ldots, X_{i_J}\right)$ stands for the conditional probability. In particular, we have, for the $\alpha$-mixing coefficient corresponding to $\left\{X_i\right\}_{i \in \mathbb{N}^*}$ itself

The $\beta$-mixing condition (A1) is typically necessary to obtain asymptotic normality for $U$-statistics in the absence of a strict Lipchitz-continuity assumption for the kernel functions, for instance, see ^[72,203] and Remarks 2.2 and 2.3 of ^[107]. Assumption (A2) is often less restrictive than the $\phi$-mixing condition. As will be shown in ^[107], finite-state and vector-valued absolutely continuous data sequences of exponentially $\phi$-mixing decaying rate satisfy (A2). We note that this is more restrictive than the polynomially mixing decaying rate. ^[203]. This is because we need to calculate higher moments of the $U$-statistics to obtain sharp concentration inequality. The "exponentially decaying rate" condition is routine in the literature of deriving concentration inequalities for weakly dependent data, see ^[149] and Remarks 2.4 and 2.5 in ^[107].

This section considers the uniform consistency, the UIB, and the UINN consistency of the functional conditional $U$-statistic given by (2.1). First, let's introduce some notation. For some interval $\mathcal{H}_{n}^{(m)} \subset \mathbb{R}_{+}^m \backslash\left\{0\right\}$, we denote

where

In the sequel, we denote (unless stated otherwise)

For all $\mathbf{b} = (b_1, \ldots, b_m) \in (0, 1)^m$, let us denote

and

For notational convenience, in the case of $m = 1$, we denote $\mathcal{H}_n^{(1)}$ and $\mathcal{H}_0^{(1)}$ simply $\mathcal{H}_n$ and $\mathcal{H}_0.$ The same goes for other similar notation unless stated otherwise. Set

and for some symmetric measurable function $f(\cdot)$ define the $\mathbb{P}$-canonical function $\pi_{p, m},$ $p = 1, \ldots, m,$ see ^[10] and ^[68] (we replace the index $k$ with $p$ to avoid confusing it with the smoothing parameter $k$), by

where for measures $Q_{i}$ on $S$ we let

and $\delta_{ x}$ denote Dirac measure at point $x \in \mathcal{X}$. This decomposition follows easily by expanding

into terms of the form

It is very simple to check that $f(\cdot)$ symmetric is $\mathbb P$-degenerate of order $r-1$ ^‖ if $r = \min \left\{\ell > 0: \pi_{\ell} f \not \equiv 0\right\}$. For example,

^‖

Definition 3.7 A $\mathbb P^m$-integrable symmetric function of $m$ variables, $f$ : $S^m \mapsto \mathbb{R}$, is $\mathbb P$-degenerate of order $r-1, 1 < r \leq m$, if $\int f\left(\mathbf x_1, \ldots, \mathbf x_m\right) d\mathbb P^{m-r+1}\left(\mathbf x_r, \ldots, \mathbf x_m\right) = \int f d\mathbb P^m$ for all $\mathbf x_1, \ldots, \mathbf x_{r-1} \in S$ whereas

is not a constant function. If $f$ is $\mathbb P^m$-centered and is $P$-degenerate of order $m-1$, that is, if

then $f(\cdot)$ is said to be canonical or completely degenerate with respect to $\mathbb P$. If $f(\cdot)$ is not degenerate of any positive order, we say it is nondegenerate or degenerate of order zero.

It's clear that, for all $\varphi \in \mathscr{F}_m :$

and $u_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))$ is a classical $U$-statistic with the $U$- kernel $G_{\varphi, \mathbf{t}, \mathbf{h}_{n, k}}(\textbf{x}, \textbf{y}).$ However, the study of the uniform consistency of $\widehat{r}^{*(m)}_{n}(\varphi, \mathbf{t}; \mathbf{h}_{n, k}(\mathbf{t}))$ to $r^{(m)}(\varphi, \mathbf{t})$ can not be done with a straightforward approach due to the randomness of the bandwidth vector $\mathbf{h}_{n, k}(\mathbf{t})$ which poses some technical problems. To circumvent this, our strategy is first to study the uniform consistency of $\widehat{r}^{*(m)}_{n}(\varphi, \mathbf{t}; \mathbf{h})$, where $\mathbf{h} = (h_1, \ldots, h_m) \in \mathcal{H}_n^{(m)}$ is a multivariate bandwidth that does not depend on $\mathbf{t}$ and $k$. Hence, we study the uniform consistency and the UIB consistency of $u_n(\varphi, \mathbf{t}, \mathbf{h})$ to $\mathbb{E} (u_n(\varphi, \mathbf{t}, \mathbf{h}))$ when $\varphi \in \mathscr{F}_m$ and when $\varphi \equiv 1,$ and we shall consider an appropriate centering factor than the expectation $\mathbb{E}\left(\widehat{r}^{*(m)}_{n}(\varphi, \mathbf{t}; \mathbf{h})\right)$, hence we define :

The second step will be the use a general lemma given in ^[44], adapted to our setting, similar to that of ^[134] (see Subsection 8.1.1) to derive the results for the bandwidth $\mathbf{h}_{n, k}(\mathbf{t}).$

3.3.1.   Uniform consistency and UIB consistency for a multivariate bandwidth

Next, we will give the UIB results for all $\mathbf{t} \in S_\mathcal{X}^m$ and $\mathbf{h} \in \mathcal{H}_n^{(m)}.$ We first start with announcing the result concerning the uniform derivation of the estimate $u_n(\varphi, \mathbf{t}, \mathbf{h})$ with respect to $\mathbb{E}\left(u_n(\varphi, \mathbf{t}, \mathbf{h})\right)$ when the class of functions is bounded.

Theorem 3.8. Suppose that the conditions (H.1.), (C.3.1), (C.4.1), (C.4.4), (C.6.) and (C.7.) are fulfilled, we infer that, as $n\rightarrow \infty$,

The following result covers the uniform derivation of the estimate $u_n(\varphi, \mathbf{t}, \mathbf{h})$ with respect to $\mathbb{E}\left(u_n(\varphi, \mathbf{t}, \mathbf{h})\right)$ when the class of functions is unbounded satisfying general moments condition.}

Theorem 3.9. Suppose that the conditions (H.1.), (C.3.1), (C.4.2), (C.4.4), (C.6.) and (C.7.) are fulfilled. For all $0 < \widetilde b_0 < 1$, we infer that, as $n\rightarrow \infty$,

The following result handles the uniform deviation of the estimate $\widehat{r}^{*(m)}_{n}(\varphi, \mathbf{t}; \mathbf{h})$ with respect to $\widehat{ \mathbb{E}}\left(\widehat{r}^{*(m)}_{n}(\varphi, \mathbf{t}; \mathbf{h})\right)$ in both situations, where the class of functions is bounded or unbounded satisfying a general moment condition.

Theorem 3.10. Suppose that the conditions (H.1.), (C.3.1), (C.4.1), (C.4.4), (C.6.) and (C.7.) (or the following (H.1.), (C.3.1), (C.4.2), (C.4.4), (C.6.) and (C.7.)) are fulfilled. For all $0 < \widetilde b_0 < 1$, we infer, as $n\rightarrow \infty$,

Theorem 3.11. Suppose that the conditions (H.1.), (C.2.1), (C.3.1) and (C.6.) are fulfilled. For all $0 < \widetilde h'_n < 1, \widetilde h'_n \rightarrow 0,$ we infer, as $n\rightarrow \infty$,

Corollary 3.12. Under the assumptions of Theorems 3.10 and 3.11 it follows that, as $n\rightarrow \infty$,

Remark 3.13. As in ^[9,27,43], we will divide the $U$-statistics into different parts: some parts can be approximated by $U$-statistics of independent blocks, while by conditioning on one block, others are empirical processes of independent blocks. To prove the nonlinear terms are negligible, we will also need some symmetrization and maximal inequalities, we refer to ^[10,67].

3.4.   Uniform consistency and UINN consistency of functional conditional U-statistics

Let $0 < \mu^*\leq\nu^* < \infty$ be some constants and $\rho_n^* \in (0, 1)$ is a sequence chosen in such a way that

and

The following result deals with the uniform deviation of the estimate $u_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))$ with respect to $\mathbb{E}\left(u_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))\right)$ when the class of functions is bounded.

Theorem 3.14. Suppose that the conditions (H.1.), (C.3.1), (C.4.1), (C.4.4), (C.6.) and (C.7.) are fulfilled. If in addition assumption (C.8.) holds, we infer that, as $n\rightarrow \infty$,

The following result deals with the uniform deviation of the estimate $u_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))$ with respect to $\mathbb{E}\left(u_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))\right)$ when the class of functions is unbounded satisfying general moments condition.

Theorem 3.15. Suppose that the conditions (H.1.), (C.3.1), (C.4.2), (C.4.4), (C.6.) and (C.7.) are fulfilled. If in addition assumption (C.8.) holds, we infer that, as $n\rightarrow \infty$,

The next results give uniform consistency when the class of functions is bounded or unbounded.

Theorem 3.16. Suppose that the conditions (H.1.), (C.3.1), (C.4.1), (C.4.4), (C.6.) and (C.7.) (or the following (H.1.), (C.3.1), (C.4.2), (C.4.4), (C.6.) and (C.7.)) are fulfilled. If in addition assumption (C.8.) holds, we infer that, as $n\rightarrow \infty$,

Theorem 3.17. Suppose that the conditions (H.1.), (C.2.1), (C.3.1) and (C.6.) are fulfilled. If in addition assumption (C.8.) holds, we infer that, as $n\rightarrow \infty$,

Corollary 3.18. Under the assumptions of Theorems 3.16 and 3.17 it follows that, as $n\rightarrow \infty$,

Remark 3.19. The choice of the parameters $\mu^*$ and $\rho_n^*$ defined in a similar way as in condition (C.8.) affects the rate of convergence of the $k$-NN estimator. We can choose these parameters depending on the small ball probability function $\phi_{t}(h)$.

Remark 3.20. The present work largely extends and completes the work of ^[42,44] in several ways. There are basically no restrictions on the choice of the kernel function in our setup, apart from satisfying some mild conditions that we will give after. The selection of the bandwidth or the number of neighbors, however, is more problematic. It is worth noticing that the choice of bandwidth is crucial to obtain a good rate of consistency, for example, it has a big influence on the size of the estimate's bias. In general, we are interested in the selection of bandwidth and neighbors that produce an estimator that has a good balance between the bias and the variance of the considered estimators. It is then more appropriate to consider the bandwidth and neighbors varying according to the criteria applied and to the available data and location which cannot be achieved by using the classical methods. The interested reader may refer to ^[34,146] for more details and discussion on the subject. In the present section, we have provided a response to this delicate problem in the FDA associated with the dependent data setting. In the present setting, we divide the $U$-statistics into different parts: some parts can be approximated by $U$-statistics of independent blocks, while by conditioning on one block, others are empirical processes of independent blocks. This decomposition is the key tool but makes the proof very involved which is the price of the extension to the dependent framework. This, allows us to use, in a nontrivial way, the techniques used for independent variables and mostly we will be using the results of ^[44]. We highlight that in the present paper, we have used a novel exponential inequality of ^[107] tailored to the dependent framework.

4.   Uniform central limit theorems

4.1.   kNN conditional empirical process

We define the functional conditional empirical process for univariate bandwidth $h_K$ by:

where $\widehat{r}_{n}^{*(1)}(\psi, { t}; h_K)$ designates (2.5) when $m = 1$, and ${r}^{(1)}(\psi, { t})$ refers to the regression function (2.2), with

If, for $\mathbb{P}\psi = \int{\psi}d\mathbb{P}$, where $\mathbb{P}$ is the probability measure and, for each $(\mathbf{ x}, \mathbf{ y})$,

then $\left\{\nu_{n}(\psi \mid { t}) : \psi \in \mathscr{FK}\right\}$ is a random element with values in $l_{\infty}(\mathscr{FK}),$ consisting of all functional $\nu_{\infty}$ on $\mathscr{FK}$ such that

Then, it will be important to investigate the following weak convergence

It is known that the weak convergence to a Gaussian limit with a version of uniformly bounded and uniformly continuous paths (with respect to the $\|\cdot\|_{2}$) is equivalent to the finite-dimensional convergence and the existence of pseudo-metric $d_{\rm p.m}$ on $\mathscr{FK}$ such that $(\mathscr{FK}, d_{\rm p.m})$ is totally bounded pseudo-metric space and

Below, we write $Z \stackrel{{d}}{ = } \mathcal{N}(\mu, \Sigma^{2})$ whenever the random vector $Z$ follows a normal law with vector expectation $\mu$ and matrix variance $\Sigma^{2}$, ${\stackrel{{d}}{\rightarrow }}$ denotes the convergence in distribution. The following theorem is adapted from ^[147] to the setting of the $k$-NN estimators. The main objective of this section is to investigate the central limit theorems for the functional conditional empirical process defined by

Theorem 4.1. Let consider the class of functions $\mathscr{FK}$, suppose that conditions (C.1'.), (C.1.2), (C.2.1), (C.2.2), (C.3'.), (C.5.) and (C.8.) hold. and if the smoothing parameter $k$ satisfies for all $\gamma > 0$

we get for: $\mathfrak{l} \geq 1:\psi_1, \ldots, \psi_\mathfrak{l} \in \mathscr{FK},$

where $\Sigma: = (\sigma_{i, j})_{i, j = 1, \ldots, \mathfrak{l}}$ is the covariance matrix with:

Theorem 4.2. Suppose that the conditions (C.3.1), (C.4.2)–(C.5.1), and (C.8.) hold and for each $\varphi \in \mathscr{F},$

Then, we have

The two previous theorems can be summarized as follows:

Theorem 4.3. Under conditions (C.1'.), (C.1.2), (C.2.), (C.3.1), (C.3'.), (C.4.4) (C.5.1), (C.5.2) and (C.8.), then the process, as $n\rightarrow \infty$,

converges in law to a Gaussian process $\left\{ \mathbb{G}_n(\psi):\psi\in\mathscr{FK}\right\}$ that admits a version with uniformly bounded and uniformly continuous paths with respect to $\Vert\cdot\Vert_2$-norm.

Remark 4.4. We mention that other types of applications can be obtained from Theorem 4.3 including the conditional distribution, conditional density, and conditional hazard function. This, and other applications of interest, will not be considered here due to lack of space.

4.2.   kNN conditional U-processes

In this section, we are interested in studying the weak convergence of conditional $U$-processes under absolute regular observations. Recall the class of functions considered is $\mathscr{F}_{m}\mathscr{K}^m$ given in Section 2. The conditional $U$-process indexed by $\mathscr{F}_{m}\mathscr{K}^m$ :

The $U$-empirical process is defined by

It should be noted that to establish the weak convergence of (4.4) it is first necessary to go through that of (4.6), below. Indeed, we will develop some details that will be used later. Because condition (C.6.) is satisfied, for each $\lambda > 0$, we have

We can write the $U$-statistic as follows

We call the first term of the right side of (4.6) $\mu_n^{(T)}(\varphi, \mathbf{t}; \mathbf{h(t)})$ truncated part and the second $\mu_n^{(R)}(\varphi, \mathbf{t}; \mathbf{h(t)})$ remainder part. First we are interested in $\mu_n^{(T)}(\varphi, \mathbf{t}; \mathbf{h(t)})$. An application of Hoeffding's decomposition gives

where $\{(\mathbf{ X}'_i, \mathbf{ Y}'_i)\}_{i \in \mathbb{N}}$ is a sequence of i.i.d. r.v. with $\mathcal{L}\left(\mathbf{ X}'_i, \mathbf{ Y}'_i\right) = \mathcal{L}\left(\mathbf{ X}_i, \mathbf{ Y}_i\right)$ for each $i$, and $\textbf{X}'$ and $\textbf{Y}'$ are respectively defined as $\textbf{X}$ and $\textbf{Y}$. In view of (4.7), we have

the stationarity assumption and some algebras show that

Therefore,

By the fact that $\pi_{p, m}G_{\varphi, \mathbf{t}, \mathbf{h}}^{(T)}$ is $\mathbb{P}$-canonical, we have to show that

So that to establish the weak convergence of the $U$-process $\left\{\mu_n^{(T)}(\varphi, \mathbf{t}; \mathbf{h(t)})\right\}_{ \mathscr{F}_{m}\mathscr{K}^m}$, it is enough to show

where $\{ \mathbb{G}(\varphi)\}_{m\mathcal{G}^{(1)}}$ is a Gaussian process indexed by $m\mathcal{G}^{(1)}$, and for $2 \leq p \leq m$

We have to prove after, that the remaining part is negligible, in the sense that

Nevertheless, when we have to deal with finite-dimensional convergence, the truncation does not matter. Which means that establishing the finite-dimensional convergence of $\mu_n(\varphi, \mathbf{t}; \mathbf{h(t)})$ is equivalent to establishing that of $\mu_n^{(T)}(\varphi, \mathbf{t}; \mathbf{h(t)}).$

Theorem 4.5. (a) Under conditions (C.1'.), (C.1.2), (C.2.), (C.3'.), (C.5.1), (C.5.2) and if ${r}^{(m)}(\varphi, \mathbf{t})$ is continuous at $\mathbf{t},$ then, as $n\rightarrow \infty$,

where

(b) If, in addition, the smoothing parameter $k$ satisfies the condition (C.8.), then we have, as $n\rightarrow \infty$,

where $\rho^2$ is defined as in (4.10), with

Corollary 4.6. Under conditions (C.1'.), (C.1.2), (C.2.), (C.3'.), (C.5.) and if $n\tilde{h'}^{2\gamma}_{n}\tilde{\phi}(\mathbf{h(t)})\to 0$ as $n \to \infty$, then we infer that:

Theorem 4.7. Under conditions (C.1'.), (C.1.2), (C.2.), (C.3'.), (C.5.1), (C.5.2), (C.8.) and $n\tilde{h'}^{2\gamma}_{n}\tilde{\phi}(\mathbf{h(t)})\to 0$ as $n \to \infty$. Let $\mathscr{F}_m\mathscr{K}^m$ be a measurable VC-subgraph class of functions from $(\mathcal{X}^m, \mathcal{Y}^m)\longrightarrow \mathbb{R}$ such as condition (C.4.2) is satisfied and, if the $\beta$-coefficients of the mixing stationary sequence $\left\{({X}_i, {Y}_i)\right\}_{i \in \mathbb{N}^*}$ fulfill:

for some $r > 1$, then $\left\{U_n^{(m)}(\varphi, \mathbf{t}; \mathbf{h}_{n, k}(\mathbf{t}))\right\}_{\mathscr{F}_m\mathscr{K}^m}$ converges in law to a Gaussian process $\left\{ \mathbb{G}(\varphi)\right\}_{\mathscr{F}_m\mathscr{K}^m}$which has a version with uniformly bounded and uniformly continuous paths with respect to $\Vert\cdot\Vert_2-$norm.

Remark 4.8. It is worth noting what is the price to pay by the nice features of the $k$-NN-based estimators: remembering that $H_{n, k}(X)$ is a random variable (which depends on $(X_1, \ldots, X_n)$, one should expect that additional technical difficulties will appear along the proofs of asymptotic properties. To fix the idea on this point, note that the random elements involved in (2.1), $m = 1$, can not be decomposed as sums of independent variables (as it is the case for instance with kernel-based estimators), and hence its treatment will need more sophisticated probabilistic developments than standard limit theorems for sums of i.i.d. variables. Also, the Hoeffding decomposition can not be applied directly to (2.1), which is the main tool for the study $U$-statistics. The first step in proving Theorem 4.7 is the extension of ^[27,43] to the multivariate bandwidth problem. In addition, we have considered new applications: the set indexed conditional U-statistic, Kendall rank correlation coefficient and time series prediction from a continuous set of past values. Another delicate problem lies in the fact that some maximal inequalities and symmetrisation techniques of ^[10,67] are not applicable directly in our framework, making the proof quit lengthy, in particular, the equicontinuity of the empirical processes.

Remark 4.9. It is straightforward to modify the proofs of our results to show that it remains true when the entropy condition is substituted by the bracketing condition: For some $C_0 > 0$ and $v_0 > 0$,

Refer to p. 270 of ^[190] for the definition of $\mathcal{N}_{[\; ]}\left(\epsilon, \mathscr{F}_m\mathscr{K}^m, L_{2}(\mathbb{P})\right)$.

5.   Some potential applications

Although only four examples will be given here, they stand as archetypes for a variety of problems that can be investigated in a similar way.

5.1.   Set indexed conditional U-statistics

We aim to study the links between $X$ and $Y$, by estimating functional operators associated to the conditional distribution of $Y$ given $X$ such as the regression operator, for $C_{1}\times\cdots \times C_{m}: = \widetilde{\mathbb{C}}$ in a is a class of sets $\mathscr{C}^{m}$,

We define metric entropy with the inclusion of the class of sets $\mathscr{C}$. For each $\varepsilon > 0$, the covering number is defined as :

the quantity $\log(\mathcal{N}(\varepsilon, \mathscr{C}, \mathbb{G}^{(1)}(\cdot\mid x)))$ is called metric entropy with inclusion of $\mathscr{C}$ with respect to $\mathbb{G}^{(1)}(\cdot\mid x)$. The quantity $\log \mathcal{N}(\varepsilon, \mathscr{C}, \mathbb{G}^{(1)}(\cdot\mid x))$ is called metric entropy with inclusion of $\mathscr{C}$ with respect to $\mathbb{G}(\cdot\mid x)$. Estimates for such covering numbers are known for many classes, (see, e.g., ^[81]). We will often assume below that either $\log \mathcal{N}(\varepsilon, \mathscr{C}, \mathbb{G}^{(1)}(\cdot\mid x))$ or $\mathcal{N}(\varepsilon, \mathscr{C}, \mathbb{G}^{(1)}(\cdot\mid x))$ behave like powers of $\varepsilon^{-1}$: we say that the condition ($R_{\gamma}$) holds if

where

for some constants $A, r > 0$. As in ^[164], it is worth noticing that the condition (5.1), $\gamma = 0$, holds for intervals, rectangles, balls, ellipsoids, and for classes which are constructed from the above by performing set operations union, intersection and complement finitely many times. The classes of convex sets in $\mathbb{R}^{d}$ ($d\geq2$) fulfill the condition (5.1), $\gamma = (d-1)/2$. This and other classes of sets satisfying (5.1) with $\gamma > 0$ can be found in ^[81]. As a particular case of (2.5), we estimate $\mathbb{G}^{(m)}(C_{1}\times\cdots \times C_{m}\mid \mathbf{ t})$

One can apply Corollary 3.18 to infer that

Remark 5.1. Another point of view is to consider the following situation, for a compact $\mathbf{ J}\subset\mathbb{R}^{dm}$,

Let $L(\cdot)$ be a distribution in $\mathbb{R}^{d}$ and $\widetilde H_{n, k}(t_{i})$ is the number of neighborhoods associated with $Y_i$s. One can estimate $\mathbf{G}^{(m)}(y_{1}, \ldots, y_{m}\mid \mathbf{ t}) = \mathbf{G}^{(m)}(\mathbf{ y}\mid \mathbf{ t})$ by

One can use Corollary 3.18 to infer that, as $n\rightarrow \infty$,

5.2.   Kendall rank correlation coefficient

To test the independence of one-dimensional random variables $Y_1$ and $Y_2$ ^[127] proposed a method based on the $U$-statistic $K_{n}$ with the kernel function :

Its rejection on the region is of the form $\left\{\sqrt{n} K_{n} > \gamma\right\}$, for more general tests, refer ^[29,32]. In this example, we consider a multivariate case. To test the conditional independence of $\boldsymbol{\xi}, \boldsymbol{\eta}: Y = (\boldsymbol{\xi}, \boldsymbol{\eta})$ given $X$, we propose a method based on the conditional U-statistic :

where $\mathbf{t} = \left(t_{1}, t_{2}\right) \in \mathbb{I} \subset \mathbb{R}^{2}$ and $\varphi(\cdot)$ is Kendall's kernel (5.5). Suppose that $\boldsymbol{\xi}$ and $\boldsymbol{\eta}$ are $d_{1}$ and $d_{2}$-dimensional random vectors respectively and $d_{1}+d_{2} = d$. Furthermore, suppose that $\mathrm{Y}_{1}, \ldots, \mathrm{Y}_{n}$ are observations of $(\boldsymbol{\xi}, \boldsymbol{\eta})$, we are interested in testing :

Let $\mathbf{a} = \left(\mathbf{a}_{1}, \mathbf{a}_{2}\right) \in \mathbb{R}^{d}$ such as $\|\mathbf{a}\| = 1$ and $\mathbf{a}_{1} \in \mathbb{R}^{d_{1}}, \mathbf{a}_{2} \in \mathbb{R}^{d_{2}}$, and $\mathrm{F}(\cdot), \mathrm{G}(\cdot)$ be the distribution functions of $\boldsymbol{\xi}$ and $\boldsymbol{\eta}$ respectively. Suppose $F^{a_{1}}(\cdot)$ and $G^{a_{2}}(\cdot)$ to be continuous for any unit vector $\mathbf{a} = \left(\mathbf{a}_{1}, \mathbf{a}_{2}\right)$ where $\mathrm{F}^{\mathbf{a}_{1}}(t) = \mathbb{P}\left(\mathbf{a}_{1}^{\top} \boldsymbol{\xi} < t\right)$ and $\mathrm{G}^{\mathbf{a}_{2}}(t) = \mathbb{P}\left(\mathbf{a}_{2}^{\top} \boldsymbol{\eta} < t\right)$ and $\mathbf{a}_{1}^{\mathrm{T}}$ means the transpose of the vector $\mathbf{a}_{i}, 1 \leqslant i \leqslant 2$. For $n = 2$, let $Y^{(1)} = \left(\boldsymbol{\xi}^{(1)}, \boldsymbol{\eta}^{(1)}\right)$ and $Y^{(2)} = \left(\boldsymbol{\xi}^{(2)}, \boldsymbol{\eta}^{(2)}\right)$ such as $\boldsymbol{\xi}^{(i)} \in \mathbb{R}^{d_{1}}$ and $\boldsymbol{\eta}^{(i)} \in \mathbb{R}^{d_{2}}$ for $i = 1, 2$, and :

An application of Corollary 3.18 gives, as $n\rightarrow \infty$,

5.3.   Discrimination problems

Now, we apply the results to the problem of discrimination described in Section 3 of ^[185], refer to also to ^[184]. We will use a similar notation and setting. Let $\varphi(\cdot)$ be any function taking at most finitely many values, say $1, \ldots, M$. The sets

then yield a partition of the feature space. Predicting the value of $\varphi(\mathbf Y_{1}, \ldots, \mathbf Y_{k})$ is tantamount to predicting the set in the partition to which $(\mathbf Y_{1}, \ldots, \mathbf Y_{k})$ belongs. For any discrimination rule $g$, we have

where

The above inequality becomes equality if

$g_{0}(\cdot)$ is called the Bayes rule, and the pertaining probability of error

is called the Bayes risk. Each of the above unknown functions $\mathfrak{M}^{j}$'s can be consistently estimated by one of the methods discussed in the preceding sections. Let, for $1\leq j\leq M$,

Set

Let us introduce

The discrimination rule $g_{0, n}(\cdot)$ is asymptotically Bayes' risk consistent, as $n\rightarrow \infty$,

This follows from Corollary 3.18 and the obvious relation

5.4.   Time series prediction from a continuous set of past values

Let $\{Z_n(t), \; t \in \mathbb{R}\}_{n\ge1}$ denote a sequence of processes with value in $\mathbb R$. Let $s$ denote a fixed positive real number. In this model, we suppose that the process is observed from $t = 0$ until $t = t_{\max }$, and assume without loss of generality that $t_{\max } = n T+s < \tau$. The method ensures splitting the observed process into $n$ fixed-length segments. Let us denote each piece of the process by

The response value is therefore $Y_i = Z(i T+s)$, and this can be formulated as a regression problem:

provided that we assume that a function of this kind, $r$, does not depend on $i$ (which is satisfied if the process is stationary, for example). Because of this, when we get to time $\tau$, we can use the following predictor, which is directly derived from our estimator, to predict the value that will be at time $\tau+s$

where $\mathbf z = (z_1, \ldots, z_k) = \{(Z_1(t), \ldots, Z_k(t))$, for $\tau-T \leq t < \tau\}$. Corollary 3.12 provides mathematical support for this nonparametric functional predictor and extends previous results in numerous ways in ^[56,91]. Notice that this modelization encompasses a wide variety of practical applications, as this procedure allows for the consideration of a large number of past process values without being affected by the curse of dimensionality. We believe that our findings will find applications beyond the scope of this work, in particular, because many popular measures of dependence, such as distance covariance and the Hilbert-Schmidt independence criterion can be estimated using $U$-statistics.

In the next section, we provide more details about how some of the methodologies of a number of neighbor choices in the literature can be combined with our results.

6.   The bandwidth selection criterion

Many methods have been established and developed to construct, in asymptotically optimal ways, bandwidth selection rules for nonparametric kernel estimators especially for Nadaraya-Watson regression estimator we quote among them ^{[15,44,45,50,105,109,166]}. This parameter has to be selected suitably, either in the standard finite-dimensional case or in the infinite-dimensional framework to ensure good practical performances. However, according to our knowledge, such studies do not presently exist for treating a such general functional conditional $U$-statistic (unless the real case we could find in the paper of ^[79] a paragraph devoted to the selection of the number $k$). Nevertheless an extension of the leave-one-out cross-validation procedure allows to define, for any fixed $\mathbf{i} = (i_1, \ldots, i_m)\in I(m, n)$ :

where

The Eq (6.1) represents the leave-out-$\left(\mathbf{X}_{\mathbf{i}}, \mathbf{Y}_{\mathbf{i}}\right)$ estimator of the functional regression and also could be considered as a predictor of $\varphi\left(\mathbf{Y}_{\mathbf{i}}\right)$. In order to minimize the quadratic loss function, we introduce the following criterion, we have for some (known) non-negative weight function $\mathcal{W}(\cdot):$

where

Following the ideas developed by ^[166], a natural way for choosing the bandwidth is to minimize the precedent criterion, so let's choose $\widehat{k} \in [k_{1, n}, k_{2, n}]$ minimizing among $k \in [k_{1, n}, k_{2, n}]$ :

we can conclude, by Corollary 3.18, that, as $n\rightarrow \infty$,

The main interest of our results is the possibility of deriving asymptotics for any automatic data-driven parameters. Let $K^\prime(\cdot)$ be a density function in $\mathbb{R}^{d}$ and $H_{n, k^\prime}^\prime(t_{i})$ is the number of neighborhoods associated with $Y_i$s. One can estimate the conditional density $\mathbf{f}^{(m)}(y_{1}, \ldots, y_{m}\mid \mathbf{ t}) = \mathbf{f}^{(m)}(\mathbf{ y}\mid \mathbf{ t})$ by

Hence, the leave-one-out estimator is given by

While the cross-validation procedures described above aim to approximate quadratic errors of estimation, alternative ways for choosing smoothing parameters could be introduced aiming rather to optimize the predictive power of the method. The criterion is given by

7.   Concluding remarks

In this paper, we consider the $k$NN kernel type estimator for conditional $U$-statistics, with the Nadaraya-Watson estimator as a special case, in a functional setting with regular datasets. To obtain our results, we need some regularity on the conditional $U$-statistics and conditional moments, decay rates on the probability of variables belonging to shrinking open balls, and convenient decreasing rates on mixing coefficients. In particular, the conditional moment assumption allows unbounded classes of functions to be considered. The proof of weak convergence adheres to a standard method: convergence of finite dimensions and the intricate equicontinuity of conditional $U$-processes. Approaching independence with a block decomposition technique, and then proving a central limit theorem for independent variables leads to finite-dimensional convergence. The equicontinuity requires more intricate control, and the details are lengthy due to the general and complex framework we have considered and will be presented in the following section. Observe that mixing is a type of asymptotic independence assumption that is commonly used to seek simplicity, but can be implausible when there is a strong dependence between the data. In ^[69] it is argued that $\beta$-mixing is the weakest mixing assumption that allows for a "complete" empirical process theory that incorporates maximal inequalities and uniform central limit theorems. There exist explicit upper bounds for $\beta$-mixing coefficients for Markov chains (cf. ^[113]) and for so-called $V$-geometric mixing coefficients (cf. ^[169]). For several stationary time series models like linear processes (cf. ^[202] for $\alpha$-mixing), ARMA (cf. ^[189]), nonlinear AR (cf. ^[126]). A common assumption in these results is that the observed process or, more often, the innovations of the corresponding process, have a continuous distribution. This is a crucial assumption to handle the relatively complicated mixing coefficients defined over a supremum over two different sigma-algebras. A relaxation of $\beta$-mixing coefficients was investigated by (^[103], Theorem 1) and is specifically designed for the analysis of the EDF, for more details, refer to ^[162]. The application of non-parametric functional concepts to general dependence structure is a relatively underdeveloped field. Notably, the ergodic framework eschews the commonly employed strong mixing condition and its variants for measuring dependence, as well as the extremely involved probabilistic calculations that this condition necessitates. It would be interesting to extend our work to the case of functional ergodic data, but this would require nontrivial mathematics and is well outside the scope of this paper. The primary obstacle lies in the necessity of formulating novel probabilistic results, as the ones employed in our current work, as demonstrated in ^[8], are tailored specifically for $\beta$-mixing samples. Another direction is to consider reducing the predictor's dimensionality by employing a Single Functional Index Model (SFIM) to estimate the regression ^[53]. SFIM has demonstrated its effectiveness in enhancing the consistency of the regression operator estimator. Change-point detection is widely employed to pinpoint positions within a data sequence where a stochastic system undergoes abrupt external influences. This method finds application across various scientific disciplines. The identification of these changes is crucial for exploring their diverse causes and enables appropriate responses. The challenge of detecting disruptions in a sequence of random variables has a rich historical background, refer to ^[26,39,41]. It would be of interest to find applications of our results in this direction.

8.   Mathematical developments

This section is devoted to the proof of our results. The aforementioned notation is also used in what follows. The proof of Theorems are quite involved and will decomposed in several lemmas proved in Section A.

8.1.   Proofs of uniform consistency results

8.1.1.   General Lemma

We present Lemma 8.1 in a general setting, for instance, see ^[44], which could be useful in many other situations than ours; this is a generalization of a result obtained in ^[58]. More generally, this technical tool could be useful for dealing with random bandwidths.

Let $(\textbf{A}_{i}, \textbf{B}_{i})_{n \ge 1}$ be $n$ random vectors valued in $\left(\Omega^m \times \Omega^{\prime m}, \mathscr{A} \times \mathscr{B}\right)$, a general space. Let $S_{\Omega}$ be a fixed subset of $\Omega$ and we note that $G:\mathbb{R}\times\left(S_{\Omega}\times\Omega\right)\rightarrow \mathbb{R}^{+}$ a function such that, $\forall t\in S_{\Omega}$, $G\left(\cdot, (t, \cdot)\right)$ is measurable and $\forall x, x' \in \mathbb{R}:$

$(L_{0})$

We define the pointwise measurable class of functions, for $1\leq m \leq n$:

Let $\left(\textbf{D}_{n, k}(\mathbf{t})\right)_{n \in \mathbb{N}}$ be a sequence of random real vectors (r.r.v.) in such a way that for all $\mathbf{t} = (t_{1}, \ldots, t_{m}) \in S_{\Omega}^m$, $\textbf{D}_{n, k}(\mathbf{t}) = \left(D_{n, k}(t_{1}), \ldots, D_{n, k}(t_{m})\right)$, and $\varphi: \Omega^{\prime m}\rightarrow \mathbb{R}$ be a measurable function belonging to some class of functions $\mathcal{F}_{m}$ and let $\mathfrak{M}^{(m)}: \mathcal{F}_{m}\times S_{\Omega}^m\rightarrow \mathbb{R}$ be a nonrandom function such that,

Now, for all $\mathbf{t}\in S_{\Omega}^m$, $\varphi\in \mathcal{F}_{m}$ and $n\in\mathbb{N}\backslash\left\{0\right\}$ we define

where $\mathbf{h} = (H_{1}, \ldots, H_{m}) \in \mathbb{R}_{+}^m$, and $\widetilde{G} = \prod_{i = 1}^mG(h_i, (x_i, t_{i})), \mathbf{t} = (t_{1}, \ldots, t_{m}) \in S_{\Omega}^m$.

Lemma 8.1. Let $(\mathcal{U}_{n})_{n\in \mathbb{N}}$ be a decreasing positive sequence such that $\lim\limits_{n\rightarrow \infty}\mathcal{U}_{n} = 0$. If, for all increasing sequence $\xi_n\in\left(0, 1\right)$ with $\xi_n-1 = O(\mathcal{U}_{n})$, there exist two sequences of r.r.v. $\left(\textbf{D}_{n, k}^{-}(\xi_n, \mathbf{t})\right)_{n\in \mathbb{N}}$ and $\left(\textbf{D}_{n, k}^{+}(\xi_n, \mathbf{t})\right)_{n\in \mathbb{N}}$ such that

$(L_{1})$

$(L_{2})$

$(L_{3})$

$(L_{4})$

$(L_{5})$

Then, as $n\rightarrow \infty$,

We refer to ^[44] for proof of this lemma.

Proof of Theorem 3.1

In order to establish the convergence rates, the following notation is necessary. For all $t \in S_{\mathcal{X}}$, set

This allows us to write

Now, let us consider the following decomposition

Therefore, the proof of (8.2) is based on the following lemmas.

Lemma 8.2. Under assumptions (C.1.1), (C.3.1), (C.4.1), (C.5.1), (C.5.2'), (C.6.) and (C.7.), we have, as $n\rightarrow \infty$,

and

This lemma gives us the rates of consistency of the stochastic part when the class of functions $\mathscr{F}$ is bounded. The following lemma will give us the result when the class of functions is unbounded.

Lemma 8.3. Under assumptions the (C.1.1), (C.3.1), (C.4.2), (C.5.1), (C.5.2'), (C.6.) and (C.7.), we have, as $n\rightarrow \infty$,

Finally, we only need the following result for the bias term. This lemma can be obtained similar way as in ^[147], where more details are given.

Lemma 8.4. Under the condition (C.2.1), we have, as $n\rightarrow \infty$,

Proof of Theorem 3.3

Similar to ^[44], To prove Theorem 3.3 we need to check the conditions of Lemma 8.1 in the case of $m = 1$. For that, we first identify the variables as follows: $S_{\Omega} = S_{\mathcal{X}}$, ${A}_{i} = {X}_{i}$, $\varphi({B}_{i}) = \varphi({Y}_{i}),$

Choosing $D_{n, k}^{-}(\xi_n, t)$ and $D_{n, k}^{+}(\xi_n, t)$ such that

We denote $h^{-}(t) = D_{n, k}^{-}(\xi_n, t)$, $h^{+}(t) = D_{n, k}^{+}(\xi_n, t)$ and

for all increasing sequence $\xi_n \in (0, 1)$ such that $\xi_n-1 = O(\mathcal{U}_{n}).$ Note that for all $\xi_n\in (0, 1)$, $t \in S_{\mathcal{X}}$ and $k_{1, n}\leq k \leq k_{2.n}$ we have

using the condition (2.12) we can easily deduce that the bandwidths $h^{-}(t)$ and $h^{+}(t)$ both belong to the interval

Checking the conditions ($L_{4}$) and ($L_{5}$)

Let us start with checking ($L_{4}$). The fact that $\xi_n$ is bounded by $1,$ and the local bandwidth $h^{-}(t)$ satisfies the conditions of Theorem 3.1 gives

which is equivalent to

We use the same reasoning to check ($L_{5}$) and we readily obtain

Hence, ($L_{4}$) and ($L_{5}$) are checked.

Checking the condition ($L_{2}$)

To check ($L_{2}$) we show that for all $t\in S_\mathcal{X}$ and $\varepsilon_{0} > 0$,

Let $\varepsilon_{0} > 0$ be fixed. Let $\{t_{1}, \ldots, t_{N_{\varepsilon}(S_\mathcal{X})}\}$ be an $\varepsilon$-net for $S_\mathcal{X},$ for all $t\in S_\mathcal{X}$, we have

Now, we use a lemma similar to ^[124] (see Lemma B.5). For completeness, we give their proof. Making use of Lemma B.5, we infer that

This implies that

In a similar way, we obtain

It follows that

Therefore, by that fact that $k_{i, n}/\ln n\rightarrow \infty$, $i = 1, 2$, we obtain

Checking the condition ($L_{3}$)

We consider the following quantities:

The condition ($L_{3}$) can be written as

Hence, by the fact that $\xi_n \longrightarrow 1$, our claimed result is

The proof of (8.13) is based on the following results

Proof of (8.14)

Using the condition (C.3.1) one has

Now using the condition (C.1.1) we directly obtain

Proof of (8.15)

We have

To prove this, we use Lemma 8.2, which gives

and

Moreover, combining (8.19), (8.20) with the fact that

it follows that

Proof of (8.16)

^[134] use in their proof of part the proof of Lemma 1 in ^[89], on the other, we will use some computations similar to the steps of the proof of Lemma 5.7.0.3 in ^[42]. Let us consider the following quantity:

using the fact that $\frac{ \phi_t(D_n^{-}(\xi_n, t))}{ \phi_t(D_n^{+}(\xi_n, t))} = \xi_n$ and supposing that the condition $\textbf{(C.2.1)}$ holds which means

and assuming that the conditions $(\textbf{C.1.1})$ and $(\textbf{C.3.1})$ to be satisfied, then for all $t \in S_\mathcal{X}$, and $D_n^{-}(\xi_n, t)\; \; \mbox{and}\; \; D_n^{+}(\xi_n, t)$ in $[h_{n, 1}, h_{n, 2}]$, one gets

Keeping in mind the condition $\textbf{(C.3.1)}$ and the fact that $\xi_n \longrightarrow 1$, we obtain

Finally, rewriting (8.23) with $\varphi \equiv 1$ gives us

which is equivalent to

Combining the results of (8.17), (8.22) and (8.24) and the fact that $\xi_n \longrightarrow 1$, implies that

Hence, ($L_{3}$) is checked. Note that ($L_{0}$) is obviously satisfied by (C.3.1), and that ($L_{1}$) is also trivially satisfied by construction of $D_{n}^{-}(\xi_n, t)$ and $D_{n}^{+}(\xi_n, t).$ So one can apply Lemma 8.1, and (8.1) with $m = 1$ is exactly the result of Theorem 3.3.

Preliminaries of the proofs

This part is mainly dedicated to the study of the functional conditional $U$-statistics. Just like in the case of $m = 1$, where $S_\mathcal{X}$ is covered by

for some radius $\varepsilon.$ Hence, for each $\mathbf{t} \in S_\mathcal{X}^m,$ there exists $\boldsymbol{\ell(t)} = (\ell(t_1), \ldots, \ell(t_m))$ where $\forall 1\leq i \leq m, 1 \leq \ell(t_i) \leq N_{\varepsilon}(S_\mathcal{X})$ such that

So for each $\mathbf{t} \in S_\mathcal{X}^m,$ the closest center is $\mathbf{t}_{\boldsymbol{\ell}(\mathbf{t})}$ and the ball with the closest center

The proofs of the UIB consistency for the multivariate bandwidth will follow the same lines as the proofs of the UIB consistency for the univariate smoothing parameter in ^[42,44]. Furthermore, as in the proof of Theorem 3.1, we divide the sequence $\{(\mathbf{X}_i, \mathbf{Y}_i)\}$ into $\upsilon_n$ alternate blocks, here the sizes $a_n, \; b_n$ are different satisfying

and set, for $1\leqslant j \leqslant \upsilon_n-1 :$

Proof of Theorem 3.8

In this section, we consider a bandwidth $\mathbf{h} = (h_1, \ldots, h_m) \in \mathcal{H}_n^{(m)}$. To prove Theorem 3.8, we can write the $U$-statistic for each $\mathbf{t} \in S_\mathcal{X}^m$ as follows

Let us begin with the term $\left|u_{n, 11}(\varphi, \mathbf{t}; \mathbf{h})\right|$, we have

By applying the Telescoping binomial, we get

From condition (C.4.1), we could claim that

similarly, we have

So, (8.28) satisfies :

where

Therefore, we infer that

The transition from Eq (8.30) to (8.31) is done thanks to the fact that the kernel function $K(\cdot)$ is Lipschitz. Uniformly on $\mathbf{t} \in S_\mathcal{X}^m$ and $\mathbf{h} \in \mathcal{H}_n^{(m)}$, we get

by (3.2), where $\mathbf{H}_n = (h_{n, 1}, \ldots, h_{n, m})$ and $\widetilde{\mathbf H}_n = (\widetilde h_n, \ldots, \widetilde h_n)$, with $\widetilde{\mathbf H}_n \leq \mathbf{H}_n$ component by component. The idea is to apply Lemma B.6 on the function

which satisfies for all $\mathbf{t} \in S_\mathcal{X}^m$ :

Notice that the existence of the constant $C_{7}$ on the last right side of the preceding inequality is deduced from the condition (2.11). Now, we can apply Lemma B.6 with $x = \sqrt{\frac{\psi_{S_{\mathcal{X}}}(\varepsilon_{n})}{n\phi(\widetilde h_n)}}-\frac{\varepsilon_{n}}{\widetilde h_n}$, which gives us

such that $c\epsilon_0 > 1$. By developing the computation while respecting the imposed conditions, mainly (C.6.) and (C.7.), we get

The study of the term $u_{n, 12}$ is deduced from the previous one. In fact:

To pass from (8.35) to (8.36), we apply Jensen's inequality in connection with some properties of the absolute value function. Then following the same way already taken, we get

where

Notice that we have

That implies

This gives that

Continue, now with $u_{n, 13}$,

Supposing that the kernel function $G_{\varphi, \mathbf{t}_{\boldsymbol{\ell}}, \mathbf{h}}(\cdot)$ is symmetric, we need to decompose our $U$-statistic according to ^[115] decomposition, we have

Define new classes of functions, for $\mathbf{h} = (h_1, \ldots, h_m) \in \mathcal{H}_n^{(m)}$, ${\boldsymbol{\ell}} \in \left\{1, \ldots, N_{\varepsilon_n}(S_\mathcal{X})\right\}^m$, and $1 \leq p \leq m :$

These classes are VC-type classes of functions with the same characteristics and the envelope function $F_p$ satisfying

Let us start with the linear term of (8.38), which is

From Hoeffding's projection, we have

One can see that

is an empirical process based on a VC-type class of functions contained in $m\left(\mathscr{F}_m\mathscr{K}^m\right)^{(1)}$ with the same characteristics and the elements are defined by:

Hence, the proof of this part is similar to that of the Lemma 8.2 and then:

Pass now to the nonlinear terms. The purpose is to prove that, for $2 \leq p \leq m$:

to do that, we need to decompose the interval $\prod\limits_{j = 1}^m (h_{n, j}, h'_{n, j})$ into smaller intervals. First, let us consider the intervals $\left(h_{n, j}, b_{j}\right)$ for all $j = 1, \ldots, m\; \; \mbox{and}\; \; b_j\in (0, 1)$, we note

where $h_{i_j} = 2^{i-1}h_{n, j}$ and $h'_{i_j} = 2^{i}h_{n, j}$ and we set $L_{j}(n) = \max \left\{i: h'_{i_j}\leq 2b_j\right\},$ and $\mathcal{I}_j = \left\{i_j: 1\leq i_j \leq L_j(n)\right\}.$ We can observe that

and

Now, we set the following new classes, for $\mathbf{h} = (h_1, \ldots, h_m) \in \mathcal{H}_n^{(m)}$ and ${\boldsymbol{\ell}} \in \left\{1, \ldots, N_{\varepsilon_n}(S_\mathcal{X})\right\}^m$, $1 \leq j \leq m$, $1\leq i \leq L(n)$ and $2 \leq p \leq m :$

Thus, to prove (8.39), we need to prove that for $1\leq j\leq m$ and $\boldsymbol{\ell} = \left(\ell_1, \ldots, \ell_m\right)$ :

Notice that for each $\epsilon_0,$ $1\leq j\leq m$ and $\boldsymbol{\ell} = \left(\ell_1, \ldots, \ell_m\right)$, we have

At this stage, we will focus on studying the above equation for $m = 2$ to simplify the proof (the same steps remain valid for $m > 2$). We have

and

For $m = 3$ the formula becomes cumbersome and given by

Let us start by considering the term $\mathbb I$. Suppose that the sequence of independent blocks $\left\{ {\xi}_i = ({\varsigma}_i, {\zeta}_i)\right\}_{i \in \mathbb{N}^*}$ is of size $a_n$. An application of (A.1), shows that

We keep the choice of $\upsilon_{n}$ and $b_n$ such that $\upsilon_{n}{b_n}^r\leq 1,$ which implies that $2\upsilon_n \beta_{b_n} \to 0$ as $n \to \infty$, so the term to consider is the second summand. the key idea is to apply Lemma B.4. We see clearly that the class of functions $\left(\mathscr{F}_m\mathscr{K}^m\right)_{i_j, \boldsymbol{\ell}}$ is uniformly bounded, i.e.,

Moreover, by applying Proposition 2.6 of ^[10] we have for each $(x_i, y_i) \in S_\mathcal{X} \times \mathcal{Y}$ and Rademacher variables $\epsilon_i$ :

where

and

We see that

So, using the fact that $\mathscr{F}_2\mathscr{K}^2$ is a VC-type class of functions satisfying (C.4.4) which implies that the class $\left(\mathscr{F}_2\mathscr{K}^2\right)_{i_j, \boldsymbol{\ell}}$ is also a VC-type class of functions with the same characteristics as $\mathscr{F}_2\mathscr{K}^2$, then,

All the conditions of Lemma B.4 are fulfilled, so a direct application gives for each $\epsilon_0$,

where

Next, let us study the same blocks $\mathbb{II}$, we have

Following the same argument as the blocks $\mathbb I$, we obtain

where

and

Again, using Lemma B.4, we readily obtain

The results for the remaining blocks can be obtained by following the same strategy above. Consequently, we have

where

This completes the proof of the theorem. $\blacksquare$

Proof of Theorem 3.10:

Notice that

Under the imposed hypothesis and the previously obtained results, and for some $c'_1, c'_2$, we get that

Therefore, we can now apply Theorem 3.8 to handle $\mathbf{II}$ and Theorems 3.8 and 3.9 to handle $\mathbf{I},$ depending on whether the class $\mathscr{F}_m$ satisfies (C.4.1) or (C.4.2), we get, for some $c'' > 0$ with probability 1:

Hence, the proof is complete.

Proof of Theorem 3.11:

Under the conditions (C.3.1), we have

Taking in consideration the hypotheses (H.1), (C.2.1), (C.3.1) and (C.6.), we get $\forall \mathbf{h} = (h_1, \ldots, h_m) \in \prod\limits_{j = 1}^m(h_{n, j}, b_j)$ and $\forall \mathbf{t}\in S^m_{\mathcal{X}}:$

where

This completes the proof of the theorem.

Proof of Corollary 3.18:

In this section, we will prove Corollary $3.18$ using Lemma 8.1. Following the same reasoning as the case of the functional regression, we use the notation: $S_{\Omega} = S_{\mathcal{X}}$, $\mathbf{A}_{i} = \mathbf{X}_{i}$, $\varphi(\mathbf{B}_{i}) = \varphi(\mathbf{Y}_{i}),$

Choosing $D_{n, k}^{-}(\xi_n, t)$ and $D_{n, k}^{+}(\xi_n, t)$ such that for all $j = 1, \ldots, m,$

where $\xi_{n, j}$ are increasing sequences that belong to $(0, 1)$, and

We denote $\mathbf{h}^{-}(\mathbf{t}) = \left(h_{1}^{-}(t_{1}), \ldots, h_{m}^{-}(t_{m})\right)\; \; \mbox{and}\; \; \mathbf{h}^{+}(\mathbf{t}) = \left(h_{1}^{+}(t_{1}), \ldots, h_{m}^{+}(t_{m})\right),$ where

We can easily see that, for all $j = 1, \ldots, m$:

Using the condition (2.12) one gets, for all $j = 1, \ldots, m,$ there exist constants $0 < \mu_j \leq \nu_j < \infty$, such that

we put $\rho_{n, j} = \sqrt{\xi_{n, j}}$, $h_{n, j} = \mu_j \phi^{-1}\left(\frac{\rho_{n, j} k_{1, n}}{ n}\right)$ and $h'_{n, j} = \nu_j \phi^{-1}\left(\frac{k_{2, n}}{\rho_{n, j} n}\right),$ thus $\mathbf{h}^{-}(\mathbf{t})$ and $\mathbf{h}^{+}(\mathbf{t})$ belong to the interval

We denote $\widetilde h_n = \min\limits_{1\leq j\leq m} h_{n, j}$ and $\widetilde {h'}_n = \max\limits_{1\leq j\leq m} h'_{n, j}$, therefore,

We also note for all $\mathbf{b} = (b_1, \ldots, b_m) \in (0, 1)^m$:

and

Finally, we can choose constants $0 < \mu^* < \nu^* < \infty$ and a sequence $\left\{\rho_n^*\right\} \in (0, 1)$, while respecting the condition (C.8.), in a way that makes $\widetilde h_n = \mu^* \phi^{-1}\left(\frac{\rho_{n}^* k_{1, n}}{ n}\right),$ $\widetilde {h'}_n = \nu^* \phi^{-1}\left(\frac{k_{2, n}}{\rho_{n}^* n}\right)$ and

It is clear that $(L_0)$ is satisfied due to the condition (C.3.1), and from (8.47) and (8.48), we can easily verify that the construction of $\mathbf{h}^{-}(\mathbf{t})$ and $\mathbf{h}^{+}(\mathbf{t})$ satisfies the condition $(L_1).$

Checking the conditions ($L_{4}$) and ($L_{5}$)

A direct application of Corollary 3.12 gives

which is equivalent to

Applying the same reasoning with $\mathbf{h}^{+}(\mathbf{t})$, we obtain

Thus, the conditions ($L_{4}$) and ($L_{5}$) are checked.

Checking the condition ($L_{2}$)

To check ($L_{2}$) we show that for all $\varepsilon_{0} > 0$ and $\mathbf{t} = (t_1, \ldots, t_m) \in S_\mathcal{X}^m :$

We have

now, using

and

Consequently, we obtain

and

Thus $(L_{2})$ is checked.

Checking the condition ($L_{3}$)

Notice that

The study of (8.56) is similar to the proofs of Theorems 3.10 and 3.11, as we can clearly see that

Let us start with (8.57), we have

Applying the same calculation as in the proof of Theorem 3.10, it follows that:

Now, for (8.58), using the fact that

therefore,

following the same steps as in Theorem 3.11, we can easily conclude that

Consequently, we have

Since $\xi_n \rightarrow 1$, we can also conclude that

which implies that

Finally, by putting $\varphi \equiv 1$ in (8.59), we get

Hence, $(L_3)$ is checked. Now, with all the conditions of Lemma 8.1.1 satisfied, it follows that

which is exactly the desired result, hence, the proof is completed.

8.2.   Proofs of weak convergence results

Preliminaries of the proofs

As mentioned before, a straightforward approach does not work when dealing with random bandwidths. Therefore, we often use some general lemmas (see, for example, ^[44]) to be able to use the results of the non-random bandwidths. In this section, we present the results of ^[147] and ^[43] obtained for some positive bandwidth $h_K$. These results are key instrumental in the proofs. We denote the bias term and the centered variate respectively the following quantities

The decomposition (8.61) plays a key role in our proof. Indeed, following the method adopted by ^[147], we will show that convergences in quadratic mean to $1,$ and that the bias satisfies

Lemma 8.5. Under conditions (C.1.), (C.3.1), and (C.5.1) (or equivalently (C.1'.), (C.3'.) and (C.5.1)), and if $n\phi(h_K)\to \infty$ as $n\to \infty$, then, we have for each $t\in \mathcal{X}$:

Before we present the next result, the following notation is needed :

Set

and

Lemma 8.6. Under conditions (C.1.), (C.2.), (C.3.1) and (C.5.1) (with $p = \infty$ and $\delta > 1$ for condition (C.5.1)), we have for $n\phi(h_K)\underset{n\to \infty}{\longrightarrow} \infty, \; t\in \mathcal{X}$ and positive constants $C_4, C_5$:

whenever $f_1(t) > 0.$ We have

Lemma 8.7. Under conditions (C.1'.), (C.2.), (C.3'.) and (C.5.1) (with $p = \infty$ and $\delta > 1$ for condition (C.5.1)), we have for $n\phi(h_K)\to \infty$ as ${n\to \infty}$ and $t\in \mathcal{X}$:

whenever $f_1(t) > 0,$ and $\kappa_1', \, \kappa_2'$ are constants specified previously.

To unburden the notation a bit and for simplicity, we denote

Lemma 8.8. Under conditions (C.1'.), (C.2.), (C.3'.), (C.5.1) and (C.5.2), if $n\phi(h_K){\to} \infty$ as $n\to \infty,$ then we have for $t\in \mathcal{X}$, as $n\to \infty$,

Lemma 8.9. Under conditions (C.1.), (C.2.), (C.3.1) and (C.5.1), and if

then we have, as $n \rightarrow \infty$,

Proof of Theorem 4.1

Using the Cramér-Wold device, it is sufficient to prove the convergence of one-dimensional distribution in order to prove Theorem 4.1. Indeed, by the linearity of $\nu_n(\psi; k_K\mid { t})$ if suffices to show that

for all $\Phi$ of the form

Therefore, we shall only demonstrate convergence in a single dimension. Remember that we're dealing with

Set

To obtain the desired result, we write

where

and

To obtain the desired results, we follow the strategy of ^[152].

Lemma 8.10. Under the assumptions (C.2.1), (C.2.2), (C.3.2), (C.4.2) and (C.8.), we have

Lemma 8.11. Under the assumptions (C.1'.), (C.2.1.), (C.2.2.), (C.3'.) and (C.5.) and if

we have

Lemma 8.12. Under the assumptions (C.1), (C.3.1), and (C.5.1) (or equivalently (C.1'.), (C.3'.) and (C.5.1)), and if

then we have for each $t\in \mathcal{X}$ :

Lemma 8.13. Under conditions (C.1.), (C.3.1), and (C.5.1) (or equivalently (C.1'.), (C.3'.) and (C.5.1)), and if $n\phi(h_K)\to \infty$ as $n\to \infty$. Then we have for each $t\in \mathcal{X}$:

as $n\to \infty.$

We highlight that this lemma is more general than Lemma 8.5. This result is slightly weaker than the uniform, almost complete convergence with the rate that we obtained in Section 3.14. However, the conditions imposed in this lemma are less restrictive.

Proof of Theorem 4.2

In this section, we will use the same method as in ^[43] and earlier ^[8], that is using the blocking approach which entails breaking down a strictly stationary sequence $(X_1, \ldots, X_n)$, into $2\upsilon_n,$ equal-sized blocks, that each one is of length $n-2\upsilon_na_n$, keeping in mind the notation given in the proof of Lemma 8.2. In order to establish the asymptotic equi-continuity of the conditional empirical process

Let us introduce, for any $\varphi K \in \mathscr{FK}$ and ${t}\in \mathcal{X},$

Then, we have

where for $h_K > 0$, we have

We study the asymptotic equi-continuity of each of the previous terms. For a class of functions $\mathscr{G},$ let $\alpha_n(\cdot)$ be an empirical process based on $({ X}_{1}, { Y}_{1}), \ldots, ({ X}_{n}, { Y}_{n})$ and indexed by $\mathscr{G}$:

and for a measurable function $\varphi(\cdot)$ and ${ t} \in \mathcal{X}$, set

That implies

Again, keeping in mind that $\mathbb{1}_{\left\{D_{n}^{-}\leq H_{n, k}(t)\leq D_{n}^{+} \right\}}\stackrel{a.co}\longrightarrow 1$ when $\frac{k}{n}\rightarrow 0$. We will establish the asymptotic equi-continuity of

which means, for every $\varepsilon > 0$, that

where

The idea is to work with the independent block sequence $\left\{{\xi}_j = ({\zeta}_j, {\varsigma}_j)\right\}_{j = 1}^{\infty}$ instead of working on dependent one, which is possible through (A.1), then we have

where $H_{n, k}(t, {\varsigma})$ is defined by

We choose

Note that $a_n$ in our setting is equivalent to:

Making use of the condition (C.5.1), we get $(\upsilon_n-1)\beta_{a_n}\longrightarrow 0$ as $n\rightarrow 0$, then it's just a matter of the right side term of (8.79). Let us begin with, the blocks being independent, we symmetrize using a sequence $\{\epsilon_j\}_{j\in \mathbb{N}^*}$ of i.i.d. Rademacher variables, i.e., r.v's with

It should be noted that the sequence $\{\epsilon_j\}_{j\in \mathbb{N}^*}$ is independent of the sequence $\left\{\boldsymbol{\xi}_i = ({\varsigma}_i, {\zeta}_i)\right\}_{i \in \mathbb{N}^*}$, thus it remains to establish, for all $\delta > 0$,

Again, using the fact that that $\mathbb{1}_{\left\{D_{n}^{-}\leq H_{n, k}(t, {\varsigma})\leq D_{n}^{+} \right\}}\stackrel{a.co}\longrightarrow 1$ when $\frac{k}{n}\rightarrow 0$ and (A.8), it suffices to show that

Since the $p^{th}$-conditional moment satisfies (C.4.2), we can truncate and we get, for each $\lambda > 0$, as $n \rightarrow \infty$,

Hence,

Then, it suffices to show

We have the following

This is done by using the chaining method. ^[8] gave $b_q = b2^{-q}$, $q = 0, \ldots, q_n$ where $q_n$

and let the class $\mathscr{FK}_q$ of measurable functions of $\mathscr{FK}$

There is a map $\pi_q : \mathscr{FK}\longrightarrow \mathscr{FK}_q$ that takes each $\eta_{n, { t}, \varphi, K} \in \mathscr{FK}$ to its closest function in $\mathscr{FK}_q$ such that

Applying the chaining method

Let $\delta_q$ be in such a way that

Let $r$ be chosen so small in such a way that

Therefore, from (8.84), we readily infer that

By the fact that the terms composing $\nu^{(2)}_n(\eta_{n, { t}, \varphi, K})$ are bounded by $a_n\lambda_n M_n^{1/2(p-1)}$, and by applying the Bernstein's inequality, we obtain

By using (8.83), we have

that means

In view of (8.85), we assume that $\delta < 3$. In a similar way, we have

Finally, by (8.83) it suffices to prove, for each $\delta > 0$,

Making use of the square root trick (Lemma 5.2 ^[102]) and see also ^[137] in similar way as in ^[8], we get

Let us introduce the semi-norm

and the covering number defined for any class of functions $\mathscr{E}$ by

By the latter we can bound $\mathbb{P}\left({\mathbb{A}}_1\right)$, (the calculations are detailed in ^[27]). In the same way, as in ^[27] and before in ^[8], as a result of the independence between the blocks and condition (C.4.3), we apply again Lemma 5.2 in ^[102] and get

Therefore, the theorem is proved.

Proof of Theorem 4.5

Lemma 8.14. Under assumptions of Theorem 4.5, we have

and, if in addition, condition (C.8) is satisfied, then we have

Lemma 8.15. Under the assumptions of Theorem 4.5, we have

where

Proof of Theorem 4.5

As mentioned earlier, the study of the weak convergence of the conditional $U$-process is based on the study of two parts: the truncated part and the remainder part.

Lemma 8.16. Let $\mathscr{F}_{m}\mathscr{K}^m$ be a uniformly bounded class of measurable canonical functions from $\mathcal{X}^m \times \mathcal{Y}^m \longrightarrow \mathbb{R}, \; m\geq 2$. Suppose that there are finite constants $\textbf{a}$ and $\textbf{b}$ such that the $\mathscr{F}_{m}\mathscr{K}^m$ covering number satisfies:

for every $\epsilon > 0$ and every probability measure $Q$. If the mixing coefficient $\beta$ of the stationary sequence $\{{Z}_i = ({X}_i, {Y}_i)\}_{i \in \mathbb{N}^*}$ fulfills

for some $r > 1$, then

Proof of Theorem 4.7

It is known that the weak convergence of an empirical process is obtained from its finite-dimensional convergence and its asymptotic equi-continuity (while respecting certain criteria). Theorem 4.5 gives the finite-dimensional convergence of the conditional U-process $\left\{\mu_n(\varphi, \mathbf{t}, \mathbf{h(t)})\right\}_{\mathscr{F}_m\mathscr{K}^m}$, so what remains to be seen is its asymptotic equi-continuity. We decompose the $U$-process $\mu_n(\varphi, \mathbf{t})$ into two parts truncated and remainder part:

Following the same reasoning to obtain (A.48), we also know that:

Therefore, it suffices to prove the weak convergence of $\mu_n(\varphi, \mathbf{t}; \mathbf{D}_n^{-})$ and $\mu_n(\varphi, \mathbf{t}; \mathbf{D}_n^{+})$ instead of studying $\mu_n(\varphi, \mathbf{t}; \mathbf{h}_{n, k}(t))$. The steps of the proof are similar to ^[43] while taking into account a multivariate bandwidth $\mathbf{h(t)}$ instead of a univariate bandwidth $h_K$. In this section we only show the proof for $\mu^{(T)}_n(\varphi, \mathbf{t}; \mathbf{D}_n^{-})$, the process can be replicated for $\mu^{(T)}_n(\varphi, \mathbf{t}; \mathbf{D}_n^{+})$.

As shown earlier, the truncated part $\mu_n^{(T)}(\varphi, \mathbf{t}, \mathbf{h(t)})$ is decomposed according to the Hoeffding's decomposition:

We shall first investigate the linear term $m\sqrt{n\tilde{\phi}(\mathbf{D}_n^{-})}u_n^{(1)}\left(\pi_{1, m}G_{\varphi, \mathbf{t}, \mathbf{D}_n^{-}}^{(T)}\right)$. Notice that

We can write

We need to introduce a new function

Hence,

The linear term of the process is given by

Therefore, the linear term of the $U$-process $\{\mu_n(\varphi, \mathbf{t}, \mathbf{D}_n^{-})\}_{\mathscr{F}_m\mathscr{K}^m}$ is an empirical process indexed by the class of functions $\mathcal{S}$ defined by

therefore, its weak convergence may be established in a similar way as in the proof of Theorem 4.5. It's clear that $\mathcal{S} \subset m \mathcal{G}^{(1)}$. We consider now the nonlinear part, we have to show that

This is a consequence of the Lemma 8.16. Note that the choice of the number and size of the blocks must be made in such a way that the terms $\rm I-\rm VI$ converge to $0.$ We need to prove that

Again, for clarity purposes, we restrict ourselves to $m = 2$. We have

We will use blocking arguments and treat the resulting terms. We start by considering the first Ⅰ'. We have

Notice that (4.13) readily implies that $\upsilon_n \beta_{b_n}\to 0$ and recall that for all $\varphi \in \mathscr{F}_m$

By the symmetry of the function $F(\cdot)$, it holds that

We use while maintaining order, Chebyshev's inequality and Hoeffding's trick, then

Under (C.6.), we have for each $\lambda > 0$

which tends to $0$ as $n\to\infty$. The terms Ⅱ', Ⅴ' and Ⅵ' are treated in the same way as the first, except that for Ⅱ', Ⅵ' we do not need to apply Hoeffding's trick because our variables $\{{\zeta}_i, {\zeta}_j\}_{i, j \in H_p^{(U)} }$ $\Big($or $\{{\zeta}_i, {\zeta}_j\}_{i, j \in T_p^{(U)} }$ for Ⅵ' $\Big)$ are in the same blocks, and for the term Ⅳ' we deduce its study from those of Ⅰ' and Ⅲ'. Let us consider the term Ⅲ'. As for the truncated part, we have

We also have

Since the Eq (8.94) is still satisfied, the problem is reduced to

we follow the same procedure as in (8.91). The rest has just been shown to be asymptotically negligible, so the process $\left\{\mu_n(\varphi, \mathbf{t}, \mathbf{D}_n^{-})\right\}_{\mathscr{F}_m\mathscr{K}^m}$ converges in law to a Gaussian process which has a version with uniformly bounded and uniformly continuous paths with respect to $\Vert\cdot\Vert_2-$norm. By repeating the same steps, this also holds true, for the process $\left\{\mu_n(\varphi, \mathbf{t}, \mathbf{D}_n^{+})\right\}_{\mathscr{F}_m\mathscr{K}^m}$. Consequently, by (8.93) it follows that the process $\left\{\mu_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))\right\}_{\mathscr{F}_m\mathscr{K}^m}$ also converges in law to a Gaussian process which has a version with uniformly bounded and uniformly continuous paths with respect to $\Vert\cdot\Vert_2-$norm. In a similar way, we treat $\left\{\mu_n(1, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t})) \right\}_{\mathscr{K}^m}$, and about $\mathbb{E}(u_n(\varphi, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))),$ $u_n(1, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t}))$ and $\mathbb{E}(u_n(1, \mathbf{t}, \mathbf{h}_{n, k}(\mathbf{t})))$ the treatment is done as in the proof of Theorem 4.5.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The authors would like to thank the Editor-in-Chief, an Associate-Editor, and four anonymous referees for their constructive remarks, which resulted in a substantial improvement of the work's original form and a more sharply focused presentation.

The second author gratefully acknowledges the funding received towards his PhD from the Algerian government PhD fellowship.

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

A.   Proofs of lemmas

Proof of Lemma 8.2:

Following the same notation as ^[43], refer also to ^[27] and ^[8], the proof of this lemma is based on the blocking approach, which consists of breaking down a strictly stationary sequence $(X_1, \ldots, X_n)$, into $2\upsilon_n,$ equal-sized blocks, that each one is of length $n-2\upsilon_na_n$ that is, for $1\leqslant j\leqslant \upsilon_n$,

The values of $\upsilon_n, a_n$ are given in the following. Another important component in this proof is the sequence of independent blocks $({\xi}_1, \ldots, {\xi}_n)$ satisfying

As in ^[27,43], applying the results of ^[84] on $\beta$-mixing, gives us for any measurable set $A$

Let $t_1, \ldots, t_{N_{\varepsilon}\left(S_{\mathcal{X}}\right)}$ be an $\varepsilon$-net of $S_{\mathcal{X}}$ and $\ell(t) = \arg \min _{\ell \in\left\{1, 2, \ldots, N_{\varepsilon}\left(S_{\mathcal{X}}\right)\right\}}$ $d\left(t, t_\ell\right)$. Observing the following decomposition

Let us start with the term $G_2$. We need to prove that there exists $\eta > 0$ and $b_0 > 0$, such that

The key strategy here is to work with independent block sequence $\left\{{\xi}_j = ({\zeta}_j, {\varsigma}_j)\right\}_{j = 1}^{\infty}$ instead of dependent once, this can be achieved by (A.1). Then we can clearly see that for $1\leq \ell \leq N_{\varepsilon}(S_\mathcal{X})$ :

If we put

then, by condition (C.5.1), it follows that $(\upsilon_n-1)\beta_{a_n}\longrightarrow 0$ as $n\rightarrow 0$. Hence, only the first term in the right-hand side of (A.3) remains to be dealt with. Set

Then, we have $[h_{n, 1}, b_0]\subseteq \bigcup\limits_{j = 1}^{L(n)}\mathcal{H}_{K, j}$. The empirical measure on $\left\{\alpha_n\left(X_i, Y_i\right): i = 1, 2, \ldots, n\right\}$ is defined by:

For the original sequence, we write

For the constructed independent block sequence $\boldsymbol{\xi}$, define

Hence, we have

We consider also the following class of functions for $1\leq j\leq L(n)$ and $1\leq \ell \leq N_{\varepsilon_n}(S_\mathcal{X})$ :

with the envelope function $G^{(\ell)}_{\varphi K, j},$ also keep in mind that $S_\mathcal{X} \subseteq \bigcup\limits_{\ell = 1}^{N_{\varepsilon}(S_\mathcal{X})} B(t_{\ell}, \varepsilon)$. Now that we have all the necessary elements and using the fact that we are now dealing with sums of independent blocks, we can use the standard inequalities of independent setting. We have

Now, to bound the probability in (A.4), we apply Bernstein's inequality with

and

We know that the envelope function $G^{(\ell)}_{\varphi K, j},$ verifies

Hence, by (C.1.1), (C.3.1), (C.4.1) combined with Lemma B.1 and Lemma B.2, we get that

Now, we can apply Bernstein's inequality to the empirical processes, for

Making use of Lemma B.3, we get

where $\mathbf C > 0$. Moreover, from Eq (2.10), and the fact that $L(n) \sim \frac{ \log(b_0/h_{n, 1})}{ {\log(2)}}$ and by choosing

we get from (A.6) and (A.4),

Now, let us study the term $G_1$. By conditions (C.4.1) and (C.3.1), for some constant $\mathbb C > 0$, we have

Using the fact that the kernel $K(\cdot)$ is supposedly Lipschitz, we obtain

where for $1\leq i\leq n :$

Taking into account that $h_K \in [h_{n, 1}, h_{n, 2}]$ and using condition (C.7.), we get for $q\geq 2 :$

where we assume that $C_1$ is the bound of the sequence ${\log{n}}\big/{nh_{n, 1}}$ which converges to $0.$ Hence, by applying a standard inequality (see Corollary A.8 ^[91]), the conditions are satisfied here, uniformly on $t \in S_\mathcal{X}$ and on $h_K,$ on :

in combination with condition (C.6.) and (C.7.), we readily obtain

Finally, we treat $G_3$. Noticing that

similar to the procedures of treating $G_1$ once again, it follows directly that

Hence, by combining (A.2) and (A.8), we so obtain (8.2), as sought.

Proof of (8.3):

Notice that (8.3) is a direct result of (8.2) when the function $\varphi \equiv 1$. This completes the proof of the lemma.

Proof of Lemma 8.10

Let us consider the term $I_1\left(t, H_{n, k}(t)\right)$ such that $\mathbb{1}_{\left\{D_{n}^{-}\leq H_{n, k}(t)\leq D_{n}^{+} \right\}}\stackrel{a.co}\longrightarrow 1$ when $\frac{k}{n}\rightarrow 0$, and making use of condition (C.3.2), we have

We have

The Eq (A.10) gives

Similarly, (A.11) gives us:

The following decomposition will be used to treat (A.12)

For some $\xi_n\in(0, 1)$, recall that $D_n^{-}$ and $D_n^{+}$ are defined in (8.6) and (8.7) respectively. Moreover, observe that

Under the assumptions (C.4.2) and (C.8.), using Hölder's inequality, for $\alpha_1 = \frac{p}{2}$ and $\alpha_2$ such that $\frac{1}{\alpha_1}+\frac{1}{\alpha_2} = 1$, we can write

To obtain the convergence in probability of (A.15), it suffices to use the assumption (C.4.2), (C.8.), and Markov's inequality. Indeed, for all $\varepsilon > 0$, we obtain

for some $\delta_n$ large enough and by condition (C.8.). Now, for the second term of (A.14), Lemma 8.2 gives

Using the fact that $k\left(\phi^{-1}\left(\frac{k}{n}\right)\right)^{2 \gamma} \rightarrow 0$ and by condition (C.2.1), for $u = u_n = \phi^{-1}\left(\frac{k}{n}\right)$, (A.13) gives

On the other side, recall that $\varphi_M(Y): = \varphi(Y) \mathbb{1}_{\left\{F(Y) < M\right\}}$, then, using the fact that the regression function satisfies the Lipschitz condition and under condition (C.4.1), we have

For the second term of the right-hand side of the Eq (8.74), we have

making use of the Lemma 8.2, we readily infer

The proof of $S_2\left(t, {H}_{n, k}(t)\right)$ is complete. Hence, the convergence of $I_1\left(t, H_{n, k}(t)\right)$ to 0 is complete combining the result (8.74) and (A.18), (A.19), (A.20) and (A.23).

Proof of Lemma 8.3:

In this lemma, we suppose that the class of functions $\mathscr{F}$ is not necessarily bounded but satisfying condition (C.4.2). In order to prove Lemma 8.3, we proceed as follows, for an arbitrary $\lambda > 0$ and each function $\varphi \in \mathscr{F}$ :

where we take $\eta_n = n/\left(\log{n}\right)^2.$ So, we write

Let us start with the truncated part. Following the same reasoning as in the proof of Lemma 8.2, we have:

First, let us start with the term $G_{2, T}$, using the same notation as the proof of Lemma 8.2, we have:

where $\tilde{\alpha}_{\upsilon_n}(\varphi^{(T)} K)$ is an empirical process indexed by the class of functions $\mathscr{FK}\mathscr{I}_{\lambda\eta_n}$, where we define the class of functions of $y \in \mathcal{Y}, \; \mathscr{I}_{\lambda\eta_n}$ by

where $p$ being the moment order given in (C.4.2). We consider also the following class of functions, for $1\leq j\leq L(n)$ and $1\leq \ell \leq N_{\varepsilon_n}(S_\mathcal{X})$,

that the envelope function is denoted by $G^{(\ell, T)}_{\varphi K, j}$, then we have

We apply Bernstein's inequality to bound (A.26), so, we have to study the asymptotic behavior of the quantities

and

From condition (C.4.2), we remark that

so thanks to the condition (C.4.2), we get

Furthermore, making use of Lemma B.1 in combination with Lemma B.2 in appendix, and since condition (C.4.3) is satisfied, we infer that

which means that,

Now, we can apply Bernstein's inequality for empirical processes with:

in Lemma B.3, we obtain, for $\mathfrak{C}^\prime > 0$,

where $\mathbf C^\prime > 0$. From Eq (2.10), and fact that $L(n) \sim \frac{ \log(b_0/h_{n, 1})}{ {\log(2)}}$ and by choosing

we get

Next, we prove that the result is valid for the term $G_{1, T}$. By conditions (C.4.2) and (C.3.1), for some constant $\mathbb C' > 0$, we have

where

Observe that

Furthermore, following the same steps as the proof of $G_1$ in the previous lemma's proof, we have:

where, for $1\leq i \leq n,$

We get uniformly on $t\in S_\mathcal{X}$ and on $h_K$ :

So, for $2 \leq q \leq p$, we obtain

The transition from (A.28) to (A.29) is done by using Jensen's inequality used for the concave function $z^a$, for $0 < a\leq 1$. Moreover, from condition (C.7.), we deduce that the quantity ${\varepsilon_n}\big/{h_{n, 1}\phi(h_{n, 1})}$ is bounded, then for $q\geq 2$

where $\mathfrak{C} > 0$. Hence, by applying a standard inequality (see Corollary A.8 ^[91]) that the conditions are satisfied here, uniformly on $t \in S_\mathcal{X}$, we get

Finally, all that is left is to evaluate the term $G_{3, T}$. We have

which means that

Let's take a look at $\left|\mathscr{G}_{1, T}(\varphi K, t; h_K)\right|$ and following the same steps, by the fact that $K(\cdot)$ is a Lipschitz function, we get

where, for $1\leq i \leq n,$

We get for $q\geq 2$ and uniformly on $t\in S_\mathcal{X}$ and on $h_K$ :

The transition from (A.31) to (A.32) is done by using Minkowski's and Jensen's inequalities. Then, following the same way as in treating $\left|\mathscr{G}_{1, T}(\varphi K, t; h_K)\right|$, we get

Therefore, the proof is done for the truncated part. Regarding the remainder term, the idea consists of proving its asymptotic negligibility, that is

which could be derived directly from the proof of the remainder part of the $U$-statistics developed in the sequel.

Proof of Lemma 8.11

We recall that

Therefore, the proof of this lemma is based on the following decomposition:

By the same arguments as those involved in the proof of $I_1(\varphi, t, H_{n, k}(t))$. We get

and

By combining (A.34) and (A.35) and by similar arguments used to treat the $I_{1 1}(\varphi, t, H_{n, k}(t))$ and $I_{1 2}(\varphi, t, H_{n, k}(t))$, we get

We then obtain that $\left|J_1(x)+J_5(x)\right|$ tending to zero as $n$ tends to infinity. We evaluate the term $J_2(x)$ on the right side of (A.33). Let us introduce the following sum

where

and

Thus, the claimed result now is

The asymptotic normality of $J_2(x)$ was proved in Lemma 8.8 by choosing the bandwidth parameter as $u = D_n^{+}$. For $J_3(x)$ and $J_4(x)$, we obtain by using Lemma 8.2 and the fact that $\mathbb{E}\left(\widehat{r}_{n, 1}^{(1)}(1, t, u)\right) = 1$ with $u = D_n^{-}$or $D_n^{+}$

Using Lemma 8.5, we get

Consequently, we have

Hence, the proof is complete.

Proof of Lemma 8.12:

For the proof of this Lemma, it suffices to use the result of ^[58]) in inequality (A.9), Tchebychev's inequality and Lemma 8.2. For $\varepsilon > 0$, we readily infer

Making use of the fact that

We finally obtain

Hence, the proof is complete.$\blacksquare$

Proof of Lemma 8.13:

We have

where

Taking into account condition (C.4.2) and using Hölder's inequality, for $2 < q < p$ such that $\frac{1}{p}+\frac{1}{q} = 1$, we can write for all $i\neq j$:

the later inequality is due to condition (C.1.2). Next, we have

Let us start with the term $V_1$, considering the conditions (C.1.1) and (C.3.1), it follows that

and we have

Hence, combining the later inequality with (A.37) gives us:

whenever $f_1(t) > 0 \; \mbox{and} \; const1 < const2.$ Next, we consider $V_2$. We have

whenever $\omega_n$ satisfies, $\omega_n = o(n)$, and by conditions (C.1.2)–(C.3.1), we readily infer that:

Thus, by the previous equation in combination with (A.39), we have

Making use of (A.37), we obtain

This when combined with (A.38) implies that

where $\omega_n$ is chosen in such a way that the above bound tends to $0$ as $n\to \infty.$ Now, let's consider $V_{2, 2}$. For any two $\sigma$-algebras $\mathcal{A}$ and $\mathcal{B}: \, \alpha(\mathcal{A}, \mathcal{B})\subseteq \beta(\mathcal{A}, \mathcal{B}),$ so by applying Davydov's lemma on strong-mixing sequences, and taking into account condition (C.4.2), we infer

However, $\mathbb{E}\left(\Delta_i(t, h_K(t))\right)$ satisfies (A.37), then we have

Which involves

Once more by using (A.37) and by a simple calculation (reduction of the double sum), we get:

This latter, with the boundary on $V_1$ implies:

Choosing $\omega_n = \left\{\phi(h_K)\right\}^{-(1-2/p)/\delta}$ and making use of condition (C.5.1), we get that:

By the same value chosen for $\omega_n$:

by the fact that $\Psi(h_K)/\phi^2(h_K)$ is assumed to be bounded and $\phi(h_K)\omega_n\to 0$, then the first term of the last inequality tends to $0$ as $n\to \infty$. Furthermore, the second term also tends to $0,$ since $1-(1-2/p)/\delta < 1.$ Hence, the proof is complete.

Proof of Lemma 8.14:

Based on (4.8), we have

Remark that the first term of the last sum is an empirical process indexed by $m\mathcal{G}^{(1)}$, so from Section 4.1, we have

So, it is enough to show, for $2\leq p\leq m,$ that

To avoid ambiguity, we consider the case where $m = 2$ (the other cases are treated in the same way). Then, we have

The projections being $\mathbb{P}$-canonical, then to show (A.43) it suffices to show that

We have

where

We first remark that

Following similar reasoning to obtain (A.41), the following bound suffices to our need

Treating the other terms as in Lemma 2 of ^[203] (refer for detailed proof in the conditional setting to [13, Lemma 3]) we get

Hence, the proof of (8.88) is complete. The statement (8.89) is a direct consequence of (8.88) in connection with condition (C.8).

Proof of Lemma 8.15

For any two constants the sum of the product of these constants by the components of the vector (8.90) is a centered $U$-statistic, i.e., for $C^{(1)}, C^{(2)}\in \mathbb{R}$,

where

So, by the Cramér-Wold theorem the proof of the present lemma is directly deduced from Lemma 8.14.

What remains to be done is to complete the proof of the theorem in question based on the two previous lemmas. Indeed,

and because

then all we have to do is prove that

We have

the latter, the fact that $r^{(m)}(\varphi, \mathbf{t})$ is continuous on $\mathbf{t}$ and $\mathbb{E}\left(u_n(\varphi, \mathbf{t}, \mathbf{h})\right)\to r^{(m)}(\varphi, \mathbf{t})$ leads to the desired result.

Proof of Lemma 8.16:

For the clarity of the exposition, we present the proof for $m = 2$; this case already contains the main idea. As in the proof of Theorem 4.2, we divide the sequence $\{(\mathbf{X}_i, \mathbf{Y}_i)\}$ into $\upsilon_n$ alternate blocks, here the sizes $a_n, \; b_n$ are different satisfying

and set, for $1\leqslant j \leqslant \upsilon_n-1$:

Note that the notation $b_n$ used here and in the proof of Theorem 4.7 denotes the size of the alternative blocks. However, in the proof of Theorem 4.2, it denotes the radius of the nets of the class of functions. Then, we have

We have to treat each of the terms Ⅰ–Ⅵ. The treatment of Ⅴ and Ⅵ is readily achieved through similar techniques used to investigate Ⅰ and Ⅱ, which we omit.

The same type of block but not the same block (Ⅰ):

Suppose that the sequence of independent blocks $\left\{ {\xi}_i = ({\varsigma}_i, {\zeta}_i)\right\}_{i \in \mathbb{N}^*}$ is of size $a_n$. An application of (A.1), shows that

We keep the choice of $b_n$ and $\upsilon_n$ such that :

which implies that $2\upsilon_n \beta_{b_n} \to 0$ as $n \to \infty$, so the term to consider is the second summand. For some $(\xi_{n, 1}, \xi_{n, 2}) \in(0, 1)^2$, we choose $\mathbf{D}_n^{-} = (D_{n, 1}^{-}, D_{n, 2}^{-})$ and $\mathbf{D}_n^{+} = (D_{n, 1}^{+}, D_{n, 2}^{+})$ such that for $j = 1, 2$

then using the results of ^[44], we have $\mathbb{1}_{\left\{D_{n, j}^{-}\leq H_{n, k}(t_j)\leq D_{n, j}^{+} \right\}}\stackrel{a.co}\longrightarrow 1$ when $\frac{k}{n}\rightarrow 0$ and making use of condition (C.3.2), we have :

where $\mathbf{D}_n^{-} = (D_{n, 1}^{-}, D_{n, 2}^{-})$ and $\mathbf{D}_n^{+} = (D_{n, 1}^{+}, D_{n, 2}^{+})$. This implies that

Now, combining Lemma A.1 of ^[67] with Proposition B.8 in the Appendix, we obtain

where $D_{n\mathbf{h}}^{(U_1)}$ is the diameter of $\mathscr{F}_2\mathscr{K}^2$ according to the distance $\widetilde{d}_{nh_K, 2}^{(1)},$ which are defined respectively by

and

Let consider another semi-norm $\widetilde{d}_{nh_K, 2}^{(2)}$

One can see that we have

We readily infer that

where $\lambda_n \to 0.$ Notice that as $\lambda \to 0,$ we have

where $a_n$ and $\lambda_n$ are chosen in such a way that the following relation will be fulfilled

Making use of the triangle inequality, in combination with Hoeffding's trick, for instance, see [8, page 62], we obtain readily that

where $\left\{{\xi}'_i = (\varsigma'_i, \zeta'_i)\right\}_{i \in \mathbb{N}^*}$ are independent copies of $\left\{{\xi}_i = ({\varsigma}_i, {\zeta}_i)\right\}_{i \in \mathbb{N}^*}$. By imposing

we readily infer that

By symmetrizing the expression in the expression in (A.51) and applying again the Proposition B.8 in the Appendix, we get

where

and for $\varphi_1 \widetilde{K}_1, \varphi_2\tilde{K}_2 \in \mathscr{F}_2\mathscr{K}^2$

The fact that

so,

we have the convergence of (A.53) to zero. For the choice of $a_n,$ $b_n$, and $\upsilon_n$, it should be noted that all the values satisfying (A.45), (A.47), (A.50), (A.52) and (A.54) are accepted.

The same blocks (Ⅱ):

Remark that we have

In a similar way as in the preceding proof, it suffices to prove that

Because of computation by [138, p. 53] and the fact that the classes functions are uniformly bounded, we obtain uniformly in $\mathscr{F}_2\mathscr{K}^2$

This implies that we have to prove that

Like for empirical processes, to prove (A.55), it suffices to symmetrize and show that

In a similar way as in (A.49), we infer that

where

and the semi-metric $\widetilde{d}_{nh_K, 2}^{(3)}$ is defined by

Since we are trading uniformly bounded classes of functions, we infer that

Since $a_n^{3/2}(n)^{-1/2}k^{-1}\to 0$, $D_{nh_K}^{(U_3)}\to 0$, we obtain Ⅱ$\to 0$ as $n \to \infty$.

Different types of blocks (Ⅲ)

An application of (A.1), shows that

by the last choice of the parameters $a_n, b_n, \upsilon_n$ and the condition (8.92) imposed on the $\beta$-coefficients, we have

For $p = 1$ and $p = \upsilon_n$, since we have independent exchangeable blocks, we infer that

For $2\leqslant p\leqslant \upsilon_n-1$, we obtain

therefore, it suffices to treat the convergence

By similar arguments as in ^[8], the usual symmetrization gives

where

In a similar way as in (A.49), we infer that

where

Since we have

and by considering the semi-metric

We show that the expression in (A.57) is bounded as follows

by choosing $\gamma_n = n^{-\alpha}$ for some $\alpha > (17r-26)/60r$, we get the convergence to zero of the previous quantity. To bound the second term in the right-hand side of (A.56), we remark that

We now apply the square root trick to the last expression conditionally on $H_{1}^{U}$. We denote by $\mathbb{E}_T$ the expectation with respect to $\sigma\left\{\left({\varsigma}_j, {\zeta}_j\right): j \in T_q, q\geqslant 3\right\}$ and we get by (C.6.), for $2r/(r-1) < s < \infty$, (in the notation in Lemma 5.2 of ^[102])

Since we need $t > 8M_n,$ and $m \to \infty$, by similar arguments as in ^[8] page 69, we get the convergence of (A.57) and (A.59) to zero.

Different types of blocks (Ⅳ)

We have :

Hence, the proof of the lemma is complete.

B.   Auxiliary results

This appendix contains supplementary information that is an essential part of providing a more comprehensive understanding of the paper.

In the sequel, we define $X, X_1, \ldots, X_n$ to be i.i.d. random variables defined on the probability space $\left(\Omega, \mathcal{A}, \mathbb{P}\right)$ and taking values in some measurable space $(\Psi, \mathcal{B}),$ and $\mathscr{F}$ to be a $\mathbb{P}$-measurable class of measurable functions with envelope function $F,$ such that :

We further assume that $\mathscr{F}$ has the following property:

$\bullet$ For any sequence of i.i.d. $\mathcal{X}$-valued random variables $Z_{1}, Z_{2}, \ldots$ it holds that

where $C_{1} > 1$ is a constant depending on $\mathcal{G}$ only.

Lemma B.1 (Theorem 2.14.1 ^[193]). For an empirical process $\alpha_n(f)$ indexed by the class of functions $\mathscr{F}$ with the notation:

and $J(\delta, \mathscr{F})$ meaning

we have, for $p > 1$,

Lemma B.2 (Theorem 3.1 ^[78]). Let $\mathscr{F}$ be a pointwise measurable function class satisfying the above assumptions. If we suppose that the empirical process $\alpha_n(f)$ satisfies:

then for any measurable subset $B\in \mathcal{B}$:

From Theorem 3.2 of ^[78] it follows that a VC-type class of functions satisfies, always, the condition (15.1).

Lemma B.3 (Bernstein type inequality Fact 4.2 ^[78]). Assume that for some $H > 0$ and $p\geq2$ the r.vs $X, X_1, \ldots, X_n$ satisfy:

then for $\mu'_n = \mathbb{E}\left(\left\Vert\sqrt{n}\alpha_n(f)\right\Vert_{\mathscr{F}}\right),$ we have for any $z > 0$:

Lemma B.4 (Proposition 4. ^[9]). Let $X_1, \ldots, X_m$ be i.i.d. random variables with values in some measurable space $(S, \mathcal{S})$. Let $\mathscr{F}$ be a class of symmetric functions $f$ from $S^m$ satisfying some measurability conditions. Suppose that there exists a finite constant $c_0$ such that for each $x_j \in S$ we have:

and that there is a finite constant $b$ such that $\sup_{f\in \mathcal{F}}\left|f(X)\right|\leq b,$ a.s. Then for each $u > 0$:

where the variables $\zeta_{i}$ are a Rademacher variables.

Lemma B.5 (Lemma 6.1 in ^[124], p.186). Let $X_1, \ldots, X_m$ be independent Bernoulli random variables with $\mathbb{P}(X_i{ = 1}) = p$, for all $i = 1, \ldots, n.$ Set $U = X_1+ \cdots + X_m$ and $\mu = pn.$ Then, for any $\omega \geq 1,$ we have

and if $\omega \in(0, 1)$, we have

Lemma B.6. Let $\left\{X_n\right\}_{n \in \mathbb{Z}}$ be a data sequence, along with the kernel function $h(\cdot)$, satisfying Assumptions (A1)–(A3). We then have, there exist absolute constants $C_4, C_5 > 0$ only depending on $\gamma$ and $r$, such that, for any $x \geq 0$ and $T$ sufficiently large,

Proposition B.7. Let $\left\{Z_{n, i}, 1\leqslant i\leqslant n, n\geqslant 1\right\}$ a strong mixing non-stationary sequence of random variables, and let $V$ be a $\sigma\left(Z_{n, i}, 1\leqslant i\leqslant j\right)$-measurable and $W$ be a $\sigma\left(Z_{n, i}, i\geqslant j+m\right)$-measurable. If $\mathbb{E}\left| V\right|^p < \infty$ and $\mathbb{E}\left| W\right|^q < \infty,$ then:

where $p, q, r > 0$:

Proposition B.8 (Proposition 3.6 of ^[10]). Let $\{\mathbf{X}_t : t \in \textbf{T}\}$ be a process satisfying, for $m \geqslant 1$:

and the semi-metric : $\rho(s, t) = \left(\mathbb{E}\left\Vert\mathbf{X}_t-\mathbf{X}_s\right\Vert^{2}\right)^{1/2}.$ There exists a constant $K = K(m)$ such that:

where $D$ being the $\rho$-diameter of $\textbf{T}$.

Remark B.9. In a similar way as in ^[35], Theorem 4.2 can be used to investigate the following problems.

1) (Expectile regression). For $p\in (0, 1)$, let ${\psi}(T-\boldsymbol{\theta}) = (p- \mathbb\{T-\boldsymbol{\theta}\leq 0\})|T-\boldsymbol{\theta}|$, then the zero of $m({\psi}, \cdot)$ with respect to $\boldsymbol{\theta}$ leads to quantities called expectiles by ^[155]. Expectiles, as defined by ^[155], may be introduced either as a generalization of the mean or as an alternative to quantiles. Indeed, classical regression provides us with a high sensitivity to extreme values, allowing for more reactive risk management. Quantile regression, on the other hand, provides the ability to acquire exhaustive information on the effect of the explanatory variable on the response variable by examining its conditional distribution, refer to ^{[3,4,150,151]} for further details on expectiles in functional data setting.

2) (Quantile regression). For $p\in (0, 1)$, let $\boldsymbol{\psi}(T-\boldsymbol{\theta}) = p - \mathbb\{T-\boldsymbol{\theta}\leq 0\}$. Then the zero of $m({\psi}, \cdot)$ with respect to $\boldsymbol{\theta}$ is the conditional $p$-th quantile, initially introduced in ^[129] in the real and linear framework, for more general setting, refer to ^[35].

3) (Conditional winsorized mean). As in ^[117], if we consider $\boldsymbol{\psi}(T-\boldsymbol{\theta}) = -k, T-\boldsymbol{\theta}, k$ if $T-\boldsymbol{\theta} < -k$, $|T-\boldsymbol{\theta}| \leq k$, or $T-\boldsymbol{\theta} > k$, then the zero of $m({\psi}, \cdot)$ with respect to $\boldsymbol{\theta}$ will be the conditional winsorized mean. Notably, this parameter was not considered in the literature on nonparametric functional data analysis involving wavelet estimators. Our paper offers asymptotic results for the conditional winsorized mean when the covariates are functions.