Abstarct: Interest in heat transfer optimization in a channel with wavy walls has increased considerably over the last decades because of obvious importance of this channels in the industry. These channels are widely used for engineering systems such as heat exchangers, cooling devices, gas turbines, nuclear reactors and electrical chips and etc. In this thesis, Multi Objective Particle Swarm Optimization Algorithm )MOPSO) is used to optimize the heat transfer of a nanofluid flow through the channel with wavy wall. The optimization method is baxsed on the lowest skin- friction coefficient and the highest average Nusselt number. Results are presented for different values of diameter of nanoparticle, wave length and Reynolds number. The design variables of problem are nanoparticle volume fraction, Reynolds number and wave amplitude. In this study, for numerical solution of the incompressible Navier Stokes equations, a program is written using a finite difference approximation algorithm with the Spline Alternating Direction Implicit (SADI) and the Alternating Direction Implicit (ADI). In the present study, using numerical results, the effect of different quantities such as Reynolds number) 100≤Re≤700( , Prandtl number, wavy amplitude of the walls of channel) 0≤a≤0.3 (, nanoparticle volume fraction )0≤φ≤5% ( and diameter of nanoparticle ) 20≤d_p≤100 nm ( has been investigated. The results showed that volume fraction and the wavy wall’s amplitude has a direct effect on the average Nusselt number. Furthermore, an optimum values for Reynolds number, diameter of nanoparticle , nanoparticle volume fraction and wave length is found to maximize the average Nusselt number and minimum the skin- friction coefficient.