# NIDS — Detection of Malicious Activities in Cloud Systems via Analysis of Encrypted Traffic Flows

Graduation project — HIAST (Higher Institute for Applied Sciences and Technology)
Networks and Operating Systems Department

## Overview

An unsupervised anomaly detection system for encrypted TLS/HTTPS traffic that identifies malicious activity without decrypting packets — relying solely on observable flow metadata (packet size, direction, and inter-arrival time).

The core motivation: labeled malicious samples are scarce for supervised learning, and traditional Deep Packet Inspection conflicts with encryption's privacy purpose.

## Approach

- **Model:** Stacked LSTM AutoEncoder, trained exclusively on normal traffic (CESNET-TLS22 dataset)
- **Detection:** Reconstruction error thresholding — flows with error above a fixed threshold are flagged as malicious
- **Sequence length:** L=8 packets, chosen via a systematic sequence-length sweep, balancing early detection with reliability
- **Fine-tuning:** Further adapted on CTU-Normal-20 (external traffic source) to reduce the false positive rate from 48.9% to 12.1%

## Final Model Performance

| Metric                                                | Value         |
|---                                                    |---            |
| ROC-AUC (TrickBot)                                    | 0.994         |
| Recall                                                | ≈99.7%        |
| False Positive Rate (CTU-Normal-20, post fine-tuning) | 12.1%         |
| Detection threshold                                   | 0.0155        |

## Repository Structure

graduation-project/
├── models/ # Training notebooks and architecture-comparison experiments
│ ├── data_preprocessing.ipynb # Preprocessing of CESNET-TLS22 and TrickBot data
│ ├── CNN_AE.ipynb # Baseline CNN AutoEncoder
│ ├── CNN_Conditioned.ipynb # CNN + packet-count conditioning
│ ├── CNN_FlagCond.ipynb # CNN + TCP flag conditioning
│ ├── DensePPI_AE.ipynb # Dense AutoEncoder (Kim & Kim baseline)
│ ├── FlowAE.ipynb # Model based on aggregated flow-level features
│ ├── StackedLSTM_AE.ipynb # Champion model — Stacked LSTM AutoEncoder
│ └── fine-tuning.ipynb # Fine-tuning of the final model (L=8) on CTU-Normal-20
│
├── nids-system-code-final/
│ ├── capture-service/ # Live sniffer + flow aggregator + inference engine
│ │ └── tests/ # unit_test.py, load_test_synthetic.py
│ ├── api-service/ # FastAPI middleware over the detections database
│ ├── dashboard/ # Real-time monitoring dashboard (Streamlit)
│ ├── victim/ # Nginx HTTPS server (traffic generation target)
│ ├── tests/ # integration_test.py, security_test.py
│ └── docker-compose.yml
│
└── report.docx # Full graduation project report (Arabic)


## Live System Architecture

Four Docker containers orchestrated via Docker Compose:

- **victim** — Nginx HTTPS server generating TLS traffic
- **capture-service** — Scapy-based live sniffer, flow aggregation, and PyTorch inference (network_mode: host)
- **api-service** — FastAPI REST layer between the database and the dashboard
- **dashboard** — Streamlit dashboard, live-updating every 2 seconds

Each service, including its own `tests/` folder (e.g. `capture-service/tests/`), is defined as an independent service in `docker-compose.yml`.

## Tech Stack

Python, PyTorch, Scapy, FastAPI, Streamlit, SQLite (WAL mode), Docker Compose, scikit-learn, NumPy, Polars

## Dataset Sources

- **CESNET-TLS22** (Luxemburk & Čejka, 2023) — normal traffic, training set
- **CTU-Malware-Capture / Stratosphere** — TrickBot malware samples, evaluation
- **CTU-Normal-20** — external normal traffic source, used for fine-tuning and generalization testing

## Running the System

```bash
cd nids-system-code-final
docker compose up -d --build
```

Dashboard available at `http://<host>:8501`
API available at `http://<host>:8000`

## Testing

**Unit tests** (run inside the `capture-service` container):
```bash
pytest capture-service/tests/unit_test.py
```

**Synthetic load test** (run inside the `capture-service` container):
```bash
python3 capture-service/tests/load_test_synthetic.py
```

**Integration tests** (run from the host, with the full stack running):
```bash
python3 tests/integration_test.py
```

**Security tests** (run from the host, with the full stack running):
```bash
python3 tests/security_test.py
```