A probabilistic framework using dynamic Bayesian networks for human activity recognition and tracking

Bayesian inference in its simplest forms is the act of moving from sample data to generalisations with a measurable degree of certainty through the utilisation of probability theory generally and Bayes' theorem specifically. Probabilistic Graphical Models (PGMs) on the other hand provide means to describe complex multivariate systems as joint probability distributions over multidimensional space yet pertaining a graph's unique ability to describe relations between system variables naturally and intuitively. Bayesian Networks (BNs) are a special type of PGMs that focuson modelling conditional joint probability distributions over multivariate domains, Dynamic Bayesian Networks (DBNs) are an extension of BNs to model conditionaljoint probability distributions over sequences of random variable spanning the time domain. The work in this dissertation investigates the potential and power of using Dynamic Bayesian Networks and Bayesian inference techniques for the sole purpose of reasoning over uncertainty in three challenging domains. These are the domains of human activity recognition and detection based multi-object tracking, where recognising human activities and tracking humans plays a fundamental role in the fields of machine learning and computer vision due to its potential in applications such human computer interaction and video surveillance systems, yet they are considered as very challenging time series classification problems, that involves the use of large streams of sensor data to enable action recognition and tracking. Another domain is anomaly detection which involves finding patterns of data that do not conform to what is normal, which can be very advantageous in critical system applications in fields such as network security and bank fraud detection. The first part of this dissertation will go through the theoretical background of Bayesian Networks and Dynamic Bayesian Networks, different aspects will be explored, from representation to parameter estimation and inference techniques. Next a probabilistic modelling and inferencing framework using DBNs and machine learning techniques is proposed, for classifying and predicting human activities from a temporal perspective. This model is first implemented on an aeroplane pilot flying activity domain, through the use of a software simulator. Next, a framework for anomaly detection through means of classification using DBNs is proposed. Afterwards this framework is extended for anomaly detection in the temporal domain using DBNs. These models are then implemented for detecting pilot errors, utilising the previous DBN pilot activity model along with different probabilistic inference techniques. The results are then evaluated and analysed. At the end, a novel Multiple Object Tracking (MOT) algorithm based on the Tracking-by-detection paradigm using DBNs is proposed. The tracker is tested using the MOT15, MOT16, MOT17 challenge benchmarks of video sequences. This algorithm is further enhanced by the introduction of additional affinity measures, in the form of a state of the art feature extractor built using a residual neural network and trained on the MARS person re-identification database. Finally, we improve the tracker performance by the use of a state of the art object detector (YOLOV5), custom trained on Google's Open Images dataset. Results are analysed and evaluated using the MOTChallenge Evaluation Kit, and then compared to other methods.