Add README.md

README.md

+# NIDS — Detection of Malicious Activities in Cloud Systems via Analysis of Encrypted Traffic Flows
+
+Graduation project — HIAST (Higher Institute for Applied Sciences and Technology)
+Networks and Operating Systems Department
+
+## Overview
+
+An unsupervised anomaly detection system for encrypted TLS/HTTPS traffic that identifies malicious activity without decrypting packets — relying solely on observable flow metadata (packet size, direction, and inter-arrival time).
+
+The core motivation: labeled malicious samples are scarce for supervised learning, and traditional Deep Packet Inspection conflicts with encryption's privacy purpose.
+
+## Approach
+
+- **Model:** Stacked LSTM AutoEncoder, trained exclusively on normal traffic (CESNET-TLS22 dataset)
+- **Detection:** Reconstruction error thresholding — flows with error above a fixed threshold are flagged as malicious
+- **Sequence length:** L=8 packets, chosen via a systematic sequence-length sweep, balancing early detection with reliability
+- **Fine-tuning:** Further adapted on CTU-Normal-20 (external traffic source) to reduce the false positive rate from 48.9% to 12.1%
+
+## Final Model Performance
+
+| Metric                                                | Value         |
+|---                                                    |---            |
+| ROC-AUC (TrickBot)                                    | 0.994         |
+| Recall                                                | ≈99.7%        |
+| False Positive Rate (CTU-Normal-20, post fine-tuning) | 12.1%         |
+| Detection threshold                                   | 0.0155        |
+
+## Repository Structure
+
+graduation-project/
+├── models/ # Training notebooks and architecture-comparison experiments
+│ ├── data_preprocessing.ipynb # Preprocessing of CESNET-TLS22 and TrickBot data
+│ ├── CNN_AE.ipynb # Baseline CNN AutoEncoder
+│ ├── CNN_Conditioned.ipynb # CNN + packet-count conditioning
+│ ├── CNN_FlagCond.ipynb # CNN + TCP flag conditioning
+│ ├── DensePPI_AE.ipynb # Dense AutoEncoder (Kim & Kim baseline)
+│ ├── FlowAE.ipynb # Model based on aggregated flow-level features
+│ ├── StackedLSTM_AE.ipynb # Champion model — Stacked LSTM AutoEncoder
+│ └── fine-tuning.ipynb # Fine-tuning of the final model (L=8) on CTU-Normal-20
+│
+├── nids-system-code-final/
+│ ├── capture-service/ # Live sniffer + flow aggregator + inference engine
+│ │ └── tests/ # unit_test.py, load_test_synthetic.py
+│ ├── api-service/ # FastAPI middleware over the detections database
+│ ├── dashboard/ # Real-time monitoring dashboard (Streamlit)
+│ ├── victim/ # Nginx HTTPS server (traffic generation target)
+│ ├── tests/ # integration_test.py, security_test.py
+│ └── docker-compose.yml
+│
+└── report.docx # Full graduation project report (Arabic)
+
+
+## Live System Architecture
+
+Four Docker containers orchestrated via Docker Compose:
+
+- **victim** — Nginx HTTPS server generating TLS traffic
+- **capture-service** — Scapy-based live sniffer, flow aggregation, and PyTorch inference (network_mode: host)
+- **api-service** — FastAPI REST layer between the database and the dashboard
+- **dashboard** — Streamlit dashboard, live-updating every 2 seconds
+
+Each service, including its own `tests/` folder (e.g. `capture-service/tests/`), is defined as an independent service in `docker-compose.yml`.
+
+## Tech Stack
+
+Python, PyTorch, Scapy, FastAPI, Streamlit, SQLite (WAL mode), Docker Compose, scikit-learn, NumPy, Polars
+
+## Dataset Sources
+
+- **CESNET-TLS22** (Luxemburk & Čejka, 2023) — normal traffic, training set
+- **CTU-Malware-Capture / Stratosphere** — TrickBot malware samples, evaluation
+- **CTU-Normal-20** — external normal traffic source, used for fine-tuning and generalization testing
+
+## Running the System
+
+```bash
+cd nids-system-code-final
+docker compose up -d --build
+```
+
+Dashboard available at `http://<host>:8501`
+API available at `http://<host>:8000`
+
+## Testing
+
+**Unit tests** (run inside the `capture-service` container):
+```bash
+pytest capture-service/tests/unit_test.py
+```
+
+**Synthetic load test** (run inside the `capture-service` container):
+```bash
+python3 capture-service/tests/load_test_synthetic.py
+```
+
+**Integration tests** (run from the host, with the full stack running):
+```bash
+python3 tests/integration_test.py
+```
+
+**Security tests** (run from the host, with the full stack running):
+```bash
+python3 tests/security_test.py
+```
\ No newline at end of file