DEVELOPMENT OF AN INTELLIGENT FILELESS MALWARE CLASSIFICATION SYSTEM USING OPTIMIZED DEEP LEARNING TECHNIQUE

Subject Area: computer science

Fileless malware is a significant cybersecurity threat because of its ability to operate without traditional file-based signatures which makes it challenging for conventional security techniques to detect. Hence, this study presents the development of an intelligent fileless malware classification system with the use of deep learning and optimization techniques. The system employs the Behaviour-Driven Development (BDD) methodology which enables precise definition and validation of detection scenarios. Data was collected from primary sources like Cyber-Dome testbed infected with fileless malware across multiple operating systems (Windows, Linux, and Mac), and secondary sources such as Kaggle repositories. Feature engineering was performed using the African Vulture Optimization Algorithm (AVOA) to select the most relevant attributes, enhancing model accuracy while reducing computational complexity. A Deep Neural Network (DNN) classifier was trained on the optimized dataset to detect malicious activity. The system was implemented in Python using TensorFlow and tested as a web-based platform. The software tested indicates that the proposed model significantly improves the detection of complex malware behaviours, providing a robust cybersecurity solution.