A study of the Fekete-Szegö functional and coefficient estimates for subclasses of analytic functions satisfying a certain subordination condition and associated with the Gegenbauer polynomials

In this paper, we introduce and study a new subclass of normalized analytic functions, denoted by

$ \mathcal F_{\left(\beta,\gamma\right)} \bigg(\alpha,\delta,\mu,H\big(z,C_{n}^{\left(\lambda \right)} \left(t\right)\big)\bigg), $

satisfying the following subordination condition and associated with the Gegenbauer (or ultraspherical) polynomials $ C_{n}^{\left(\lambda\right)}(t) $ of order $ \lambda $ and degree $ n $ in $ t $:

$ \alpha \left(\frac{zG^{'}\left(z\right)}{G\left(z\right)} \right)^{\delta}+\left(1-\alpha\right)\left(\frac{zG^{'} \left(z\right)}{G\left(z\right)}\right)^{\mu} \left(1+\frac{zG^{''}\left(z\right)}{G^{'} \left(z\right)} \right)^{1-\mu} \prec H\big(z,C_{n}^{\left(\lambda\right)} \left(t\right)\big), $

where

$ H\big(z,C_{n}^{\left(\lambda\right)}\left(t\right)\big) = \sum\limits_{n = 0}^{\infty} C_n^{(\lambda)}(t)\;z^n = \left(1-2tz+z^2\right)^{-\lambda}, $

$ G\left(z\right) = \gamma \beta z^{2} f^{''} \left(z\right)+\left(\gamma-\beta \right)zf^{'} \left(z\right)+\left(1-\gamma+\beta\right)f\left(z\right), $

$ 0\leqq \alpha \leqq 1, $ $ 1\leqq \delta \leqq 2, $ $ 0\leqq \mu \leqq 1, $ $ 0\leqq \beta \leqq \gamma \leqq 1 $, $ \lambda \geqq 0 $ and $ t\in \left(\frac{1}{\sqrt{2}}, 1\right] $. For functions in this function class, we first derive the estimates for the initial Taylor-Maclaurin coefficients $ \left|a_{2}\right| $ and $ \left|a_{3}\right| $ and then examine the Fekete-Szegö functional. Finally, the results obtained are applied to subclasses of normalized analytic functions satisfying the subordination condition and associated with the Legendre and Chebyshev polynomials. The basic or quantum (or $ q $-) calculus and its so-called trivially inconsequential $ (p, q) $-variations have also been considered as one of the concluding remarks.