Abstarct: Parallel analysis of power quality and protection coordination has become an important part of modern power system studies. Nowadays, the coordination of directional overcurrent relays (DOCRs) is significantly affected by the emerging of distributed generations (DGs). Using voltage sag energy index, a new approach is proposed for optimization of DOCRs coordination in distribution systems in the presence of DGs, Fault current limiters (FCLs), and voltage-sensitive buses. This approach aims to minimize an objective function which comprises three key elements: operating time of relays, voltage sag energy index, and voltage sag duration. The impedance of FCL and the time setting multiplier, current setting and characteristic of DOCRs are the optimization variables. Voltage tolerance curve and voltage sag energy index are also used to show the improvement of voltage sag characteristics and prioritization of answers. Simulation results show that, the operating time of DOCRs and voltage sag characteristics are significantly improved. Selectivity constraints are very crucial in directional overcurrent relays (DOCRs) coordination and could be affected by the change of short circuit current (SCC), which itself is highly dependent on network topology change (NTC). In this thesis, robust and adaptive coordination approaches are used to achieve an acceptable level of power quality (PQ) and selective operation of DOCRs under NTC conditions and to find the optimal setting of DOCRs. baxsed on the simulation results, using robust and adaptive coordination, the DOCRs have selective operation during NTC, and voltage sag improvement is achieved. Although the electric energy supply from distributed generation (DG) resources is increasing, these sources also face limitations. Protection and power quality constraints are the most important issues that limit the maximum DG penetration in power systems. Changing the short circuit level due to DGs' presence –especially synchronous-baxsed (SDG) type– may lead to miscoordination of the protective relays. Also, standard harmonic limits may be violated due to harmonic injection of inverter-baxsed DGs and nonlinear loads. In this thesis, a protection coordination problem is solved for the basic configuration, and the optimal settings are applied. Then, the location, size, and type of DGs are optimized to maximize the DG penetration and the speed of protection system, while reducing power losses. Finally, the protection setting of SDGs is optimized and applied. In these optimizations, the protection coordination constraints and the harmonic limits are considered and met. IEEE 14-bus system and a real sub-transmission system are simulated to evaluate the proposed approach and the new objective function. The obtained results show significant power loss reduction and protection speed improvement for different scenarios and case studies.