Abstarct: Aabstract In this thesis, we deal with nonsmooth dynamical system. Nonlinear systems have always been of interest to many researchers in recent years. Given the importance of these systems and their applications in various sciences including mechanics, physics and control, effcient methods should be used to deal with these systems. Most nonsmooth systems are in the form of dynamical systems including nonsmooth or non-differentiable functions. Therefore, in this dissertation, we study this type of nonsmooth dynamical systems, especially including the sign function or discontinuous right hand side. In section 1, we familiar with all types of nonsmooth dynamical systems and some of smoothing methods. In section 2, a wide class of discontinuous dynamical systems is considered where their discontinuouty is because of existence of sign function. For solving the discontinuous system, an optimal control problem is suggested. By solving some numerical examples in mechanical engineering, the effciancy of suggested approach for discontinuous system, is shown. In section 3, a new approach is presented for solving nonsmooth system into a smooth system using Chebyshev interpolating. We than slove the smooth system by Chebyshev pseudospectral method. The effcincy of approach is illustrated to solve two applied nonsmooth dynamical system. In section 4, Chebyshev knotting methods are givan for solving a wide types of optimal control problems including nonsmooth, nonlinear switching optimal control and standard multiphase problems. The methods are baxsed on the Chebyshev approximation. Discontinuity and switches in states, control, cost functional and dynamical constraints of considerd problem are allowed by the consept of pseudo spectral knots. Finally, the optimal control problem is approximated by a mathmatical programming problem which can be solved by sequential quadratic programming and interior point methods.