This paper describes an approach to human action recognition based on a probabilistic optimization model of body parts using Hidden Markov Model (HMM). Our method is able to distinguish between similar actions by only considering the body parts having
major contribution to the actions, for example, legs for walking, jogging and running; arms for boxing, waving and clapping. We apply HMMs to model the stochastic movement of the body-parts for action recognition. The HMM construction uses an ensemble of body-part
detectors, followed by grouping of part detections, to perform human identification. Three example-based body part detectors are trained to detect three components of the human body: the head, the legs and the arms. These detectors cope with viewpoint changes
and self-occlusions through the use of ten sub-classifiers that detect body parts over a specific range of viewpoints. Each sub-classifier is a Support Vector Machine (SVM) trained on features selected for the discriminative power for each particular part/viewpoint
combination. Grouping of these detections is performed using a simple geometric constraint model
which yields a viewpoint invariant human detector. We test our approach on three publicly available action datasets: the KTH dataset, the Weizmann dataset and the HumanEva dataset. Our results illustrate that with a simple and compact representation we can achieve robust recognition of human actions comparable to the most complex, state-of-the-art methods.