Abstarct: Atomic force microscope (AFM) has a wide range of application as an important instrument in manufacturing, analysis and imagining of nano surfaces. The main purpose of this thesis is to investigate the linear and nonlinear vibration behavior of rectangular cantilever of AFM. Because the size of AFM cantilever is as small as 100-400 in length, it is very hard to control the exact size during manufacturing. Firstly, the exact solution of torsional and flexural characteristic equation for a rectangular cantilever with different contact position is derived. Consequently, the frequency behavior of system in different contact position and different contact stiffness are analyzed. In this analysis, the influences of some tip properties are investigated. Also a complete model for rectangular cantilever is proposed. In this model the parameters that were neglected before, are investigated. The angle of cantilever with the surface, tip moment of inertia and damping are some parameters that are investigated in this model. The contact damping is discussed in this thesis as well. Secondly, a tangential contact model is derived, by which the tangential contact stiffness is demonstrated to be strongly related to the friction factor and always less than the normal contact stiffness. Finally, the nonlinear behavior of the rectangular cantilever with Hertzian contact is investigated. The nonlinear vibration can be analyzed in a ways similar to the nonlinear problem by using the effective stiffness method proposed.