Abstarct: This thesis considers the estimation methods in semiparametric mixed effects regression models for longitudinal data analysis. The model is a combination of mixed effects and semiparametric models. The nonparametric function of the model is estimated using kernel, backfitting and spline methods, then consider the multicollinearity problem and introducea new estimator called the ridge estimator. Also considers the problem of simultaneous variable selection and estimation in the generalized mixed effects model for gaussian and non-gaussian high dimensional data. A penalization type of generalized estimating equation method is proposed while using regression spline to approximate the nonparametric component. In addition, for analyzing multivariate longitudinal data in the presence of atypical observations or outliers, a robust method baxsed on multivariate t-distribution is offered. In each study frxamework, the asymptotic behavior of proposed estimators is studied and for practical implementation, an appropriate iterative algorithm is developed. The performance of the proposed methods are compared through both simulation examples and real data sets.