Anomaly based Intrusion Detection System through Remote Virtual Machine Introspection

Author

Huseyn Huseynov, CUNY City CollegeFollow

Date of Award

2023

Document Type

Dissertation

Department

Engineering

First Advisor

Tarek Saadawi

Keywords

Intrusion Detection, security and privacy, virtualization, artificial immune system, cloud security

Abstract

Research on identifying malicious applications is an important direction in information security, especially when it comes to detection of evasive malware such as keyloggers, trojans, rootkits and their derivatives. Inspired by a biological immune system and based on negative selection algorithm approach to detect various types of malwares is proposed in this paper.

By deeply studying Linux kernel, understanding links behind different internal system processes, examining, and experimenting with hundreds of various keyloggers we propose a single Artificial Intelligence based solution as a comprehensive protection against wide range of malwares. Developed Intrusion Detection System (IDS) can be deployed in the host operating system or launched remotely to introspect Virtual Machines (VMs) protecting them against many types of malicious software such as keyloggers, spyware/adware, rootkits, worms, trojans and other villainous threats. Additional research has been conducted to demonstrate application of proposed IDS in Multi-access edge computing (MEC) environment. As part of the research, there have been developed a custom keylogger with the ability to remotely control its behavior. This replication and continuous tests enabled discovery of 26 systemwide features such as system interrupts, triggered system calls, TCP and UDP signals, socket information, keyboard drivers’ characteristics, size of resident file mappings, and other parameters that are activated based on malware’s actions. Experiments with partially and fully activated features have been continuously conducted specifically against following malicious applications: Logkeys, Rootkit, Stitch, Blueberry, TrojanX, Jynx2, Jynxkit, Umbreon-Rootkit, Vlany.

Developed lightweight and secure Virtual Machine Introspection (VMI) module – KVMonitor has been customized to return data in a JSON-like structure, which in return can be processed by the IDS. Proposed IDS implements Negative Selection Algorithm that can learn to detect multiple variations of malware based on data received from the VMI module. Deployed DEAP Python library as a replacement to custom built Evolutionary Algorithm facilitated generation of random detectors and improved detection accuracy by 30%. Initially developed to detect only keyloggers, IDS now is a cross-platform application with user-friendly graphical interface and ability to successfully detect anomalies triggered by wide range of malware.

To organize real-world experiments, there has been developed a cross-continental testbed environment connecting networks between two labs at Kyushu Institute of Technology in Japan and The City College of New York through secure GRE tunnel. Conducted experiments on this testbed against hundreds of evasive malicious applications demonstrated average anomaly detection rate close to 97%.

Recommended Citation

Huseynov, Huseyn, "Anomaly based Intrusion Detection System through Remote Virtual Machine Introspection" (2023). CUNY Academic Works.
https://academicworks.cuny.edu/cc_etds_theses/1149