Abstarct: The atomic force microscope (AFM) is a powerful instrument in the field of nanotechnology, and is now widely used for imaging the surface topography conductors and insulators on an atomic scale. when a tip scans across a sample surface, it induces a dynamic interaction force between the tip and the surface. This dynamic interaction behaviour between the cantilever and sample is complicated, and its analysis can help to increase the resolution of surface images. The AFM typically operates in two modes, according to the type of contact between the tip of the cantilever and the sample, contact mode and tapping mode. In contact mode, AFM cantilever moves up and down vertically with small amplitude (1–5 nm), while in tapping mode the amplitude is approximately high (5-100nm). In this thesis, the vibration behavior of both conventional cantilever and assembled cantilever probe are analyzed with two different objectives. The first objective is to investigate the effects of various parameters such as the geometry of cantilever and tip, contact mechanics and experimental circumstances on the dynamic behavior of AFM microcantilever in the proposed application mode. On the other hand, the investigation of the size dependant vibration behavior of AFM cantilevers and also the study of necessary conditions for applying them in the analysis of the vibration behavior of AFM microcantilever is the second objective. Hence, the analysis have been done baxse on the constitutive equations of different non-classical beam theories such as strain gradient theory, modified couple stress theory and nonlocal beam theory. In order to solve the nonlinear equations in the tapping mode, the perturbation technique have been used. Utilizing this technique, closed-form exxpressions for effective nonlinear frequency and damping factor have been obtained and also a exxpression for the analysis of system frequency response have been presented. For the contact mode, the results show that the vibration behavior of AFM microcantilever are strongly size-dependant especially when the contact stiffness is high and the thickness of the microcantilever approaches the internal material length scale parameter. In this situation, utilizing a non-classical beam theory for analysis the vibration behavior of AFM cantilever are recommended. In the analysis of the nonlinear vibration behavior of conventional microcantilever in the tapping mode, some difference between the results obtained by classical beam theory and those predicted by modified couple stress theory are observed Therefore, the analysis baxse on a non-classical beam theory, in which the effects of the small scale are considered, seems necessary. According to The results indicate that attractive van der Waals interactions lead to a initial softening nonlinearity of the periodic solution response while the repulsive DMT interactions lead to a subsequent hardening nonlinear response. Some factors such as tip position, tip radios and equilibrium tip-sample separation can affect the intensity of either softening or hardening phenomenon.