Abstarct: With the increasing number of videos being uploaded to the internet and also with the growing rate of crimes and terrorist attacks in recent years, the need for an automatic system aimed at activity recognition is felt more than before. Firstly, Video sharing websites are looking for a way to recognize and remove illegal or violent content automatically. And secondly, human-baxsed video surveillance is not much efficient because of tiredness and distraction as well as its cost. So, a research field for automatic activity recognition was created to address these issues. On account of variations in video background and people's appearance in the video, we need descxriptors that can represent the motion and are also robust to noise and irregular abrupt motions caused by camera motion. Another challenge in this field is to deal with the variations in the way one single activity can be performed. Accordingly we need to obtain a semantic understanding of what is occurring in the video so as to address it. The present thesis proposes a hierarchical approach for activity recognition, which considers the succession of temporal information in the video and is able to compare videos of different lengths. Improved dense trajectory features are used and then encoded by means of improved fisher vector. Temporal information of the video is extracted in two laxyers by employing rank pooling and time-flexible kernel methods. Afterwards activities are separated using a support vector machine. The implementation of the proposed method resulted in 57.97% accuracy on split 1 of hmdb51 dataset and 85.07% accuracy on Olympic sports dataset.