# NIDS — Detection of Malicious Activities in Cloud Systems via Analysis of Encrypted Traffic Flows

Graduation project — HIAST (Higher Institute for Applied Sciences and Technology)
Networks and Operating Systems Department

## Overview

An unsupervised anomaly detection system for encrypted TLS/HTTPS traffic that identifies malicious activity without decrypting packets — relying solely on observable flow metadata (packet size, direction, and inter-arrival time).

The core motivation: labeled malicious samples are scarce for supervised learning, and traditional Deep Packet Inspection conflicts with encryption's privacy purpose.

## Approach

- **Model:** Stacked LSTM AutoEncoder, trained exclusively on normal traffic (CESNET-TLS22 dataset)
- **Detection:** Reconstruction error thresholding — flows with error above a fixed threshold are flagged as malicious
- **Sequence length:** L=8 packets, chosen via a systematic sequence-length sweep, balancing early detection with reliability
- **Fine-tuning:** Further adapted on CTU-Normal-20 (external traffic source) to reduce the false positive rate from 48.9% to 12.1%

## Final Model Performance

| Metric                                                | Value         |
|---                                                    |---            |
| ROC-AUC (TrickBot)                                    | 0.994         |
| Recall                                                | ≈99.7%        |
| False Positive Rate (CTU-Normal-20, post fine-tuning) | 12.1%         |
| Detection threshold                                   | 0.0155        |

## Repository Structure

```
graduation-project/
├── models/                     
│   ├── data_preprocessing.ipynb
│   ├── CNN_AE.ipynb        
│   ├── CNN_Conditioned.ipynb        
│   ├── CNN_FlagCond.ipynb          
│   ├── DensePPI_AE.ipynb           
│   ├── FlowAE.ipynb                 
│   ├── StackedLSTM_AE.ipynb         
│   └── fine-tuning.ipynb            
│
├── nids-system-code-final/
│   ├── capture-service/             
│   │   └── tests/                  
│   ├── api-service/                 
│   ├── dashboard/                  
│   ├── victim/                    
│   ├── tests/                    
│   └── docker-compose.yml
│
└── report.docx                      
```

## Live System Architecture

Four Docker containers orchestrated via Docker Compose:

- **victim** — Nginx HTTPS server generating TLS traffic
- **capture-service** — Scapy-based live sniffer, flow aggregation, and PyTorch inference (network_mode: host)
- **api-service** — FastAPI REST layer between the database and the dashboard
- **dashboard** — Streamlit dashboard, live-updating every 2 seconds

Each service, including its own `tests/` folder (e.g. `capture-service/tests/`), is defined as an independent service in `docker-compose.yml`.

## Tech Stack

Python, PyTorch, Scapy, FastAPI, Streamlit, SQLite (WAL mode), Docker Compose, scikit-learn, NumPy, Polars

## Dataset Sources

- **CESNET-TLS22** (Luxemburk & Čejka, 2023) — normal traffic, training set
- **CTU-Malware-Capture / Stratosphere** — TrickBot malware samples, evaluation
- **CTU-Normal-20** — external normal traffic source, used for fine-tuning and generalization testing

## Running the System

```bash
cd nids-system-code-final
docker compose up -d --build
```

Dashboard available at `http://<host>:8501`
API available at `http://<host>:8000`

## Testing

**Unit tests** (run inside the `capture-service` container):
```bash
pytest capture-service/tests/unit_test.py
```

**Synthetic load test** (run inside the `capture-service` container):
```bash
python3 capture-service/tests/load_test_synthetic.py
```

**Integration tests** (run from the host, with the full stack running):
```bash
python3 tests/integration_test.py
```

**Security tests** (run from the host, with the full stack running):
```bash
python3 tests/security_test.py
```