Opportunistic machine learning methods for effective insider threat detection

The topic of insider threat detection is getting an increased concern from academia, industry, and governments due to the growing number of malicious insider incidents. A malicious insider threat is devised of a set of anomalous behaviours attributed to an insider who exploit their privileges with the intention to compromise the confidentiality, integrity, or availability of the system or data. The existing approaches for detecting insider threats still have a common shortcoming, which is the high number of false alarms (false positives), which deceives the system administrator(s) about suspicious behaviour of many users. To address the shortcoming of false alarms, in this thesis, we formulate an opportunistic approach to detect insider threats with the aim of any-behaviour-all-threat detection. As a preliminary step, we apply feature engineering on the data logs of users' behaviour. This work is conducted on synthetic CMU-CERT data sets which implement a variety of malicious insider threat scenarios. The maturity of data in an organisation is defined into three cases based on the availability of labelled data. We address the different cases of data maturity by proposing, developing, and evaluating machine learning approaches that incorporate techniques to reduce false alarms. The first presents a class imbalance approach, namely CD-AMOTRE, which combines the concept of Class Decomposition (CD) and a novel Artificial Minority Oversampling and Trapper REmoval (AMOTRE) technique. The second builds an adaptive one-class ensemble-based anomaly detection framework which introduces a progressive update method with an outlier aware artificial oversampling procedure. The third proposes a real-time anomaly detection approach, namely Ensemble of Random subspace Anomaly detectors In Data Streams (E-RAIDS). The proposed approaches detect most/all of the malicious insider threats, and achieve the minimum FP over the data sets compared to the existing machine learning approaches.