# Smart-IDPS
### Smart Distributed Intrusion Detection and Prevention System for Cloud Environments

A distributed, real-time network security monitoring system built on a multi-node OpenStack cluster, leveraging Apache Hadoop and Apache Spark for large-scale traffic analysis, and a two-stage machine learning pipeline for intrusion detection.

---

## 📌 Overview

Smart-IDPS monitors network traffic across a distributed cloud environment in real time, detects malicious behavior (DDoS, Brute Force, DoS), classifies the type of attack, and enables both automatic and manual incident response — including IP blocking — through a live analyst dashboard.

The project was developed as a graduation thesis and demonstrates the practical necessity of distributed computing (Hadoop + Spark) for intrusion detection at scale, backed by benchmark comparisons against traditional single-node approaches.

---

## 🏗️ System Architecture

The system runs on a **three-node OpenStack cluster**:

| Node | IP | Services |
|------|-----|----------|
| **k1** | 10.0.0.15 | HDFS NameNode, Kafka, Zookeeper, Spark Master |
| **k2** | 10.0.0.124 | FastAPI backend, React dashboard, Spark Worker, Filebeat, CICFlowMeter |
| **k3** | 10.0.0.18 | PostgreSQL, HDFS DataNode, Spark Worker |

### Data Flow (Pipeline)

```
pcap files → pcap-watcher → CICFlowMeter → CSV → traffic_dripper
     → Filebeat → Kafka → Spark Streaming → FastAPI → PostgreSQL → React Dashboard
```

### Machine Learning Model

A **two-stage hierarchical classifier**:

1. **Stage 1 — Binary Detection**: Logistic Regression determines whether traffic is `Benign` or an `Attack`.
2. **Stage 2 — Multiclass Attack Typing**: Random Forest (trained via Spark ML) classifies the attack type — `DDoS`, `Brute Force`, or `DoS`.

---

## ⚙️ Tech Stack

**Infrastructure & Orchestration**
- OpenStack (KVM), Ubuntu 24.04 LTS
- Docker, Docker Compose

**Big Data & Streaming**
- Apache Kafka, Apache Zookeeper
- Apache Spark (Streaming + MLlib)
- Hadoop HDFS

**Machine Learning & Data**
- CSE-CIC-IDS2018 dataset
- CICFlowMeter
- Filebeat

**Backend & Storage**
- FastAPI (REST + WebSocket)
- PostgreSQL (psycopg2, ThreadedConnectionPool)

**Frontend**
- React, Vite, React Router

**Testing**
- Pytest (19 unit tests, run inside Docker)
- Apache JMeter 5.6.3

**External Services**
- AbuseIPDB (IP reputation lookup, 24h caching)

---

## 📂 Repository Structure

```
smart-idps/
├── k1-node/          # HDFS NameNode, Kafka, Zookeeper, Spark Master configs
├── k2-node/           # FastAPI backend, React dashboard, Spark Worker
├── k3-node/           # PostgreSQL, HDFS DataNode, Spark Worker
├── ml-pipeline/       # Training scripts, feature engineering, model artifacts
├── report/            # Academic thesis documentation (Arabic)
└── README.md
```

> **Note:** Large binary artifacts (trained models, `.pcap` captures, datasets, `node_modules/`) are excluded from version control via `.gitignore` due to size constraints. See [Model & Dataset Access](#-model--dataset-access) below.

---

## 🚀 Getting Started

### Prerequisites
- Docker & Docker Compose installed on all three nodes
- Internal network connectivity between nodes (`10.0.0.x` — **do not use floating IPs**, they are NAT addresses and will break inter-node communication)

### Setup

1. Clone the repository on each node:
   cd ids-project
   ```

2. Navigate to the relevant node directory (`k1-node`, `k2-node`, or `k3-node`) and start its services:
   ```bash
   docker-compose up -d
   ```
   > If you change a `docker-compose.yml` file, use `docker-compose up -d --force-recreate` — `restart` does **not** apply config changes.

3. Verify HDFS DataNode connectivity:
   ```bash
   docker exec namenode hdfs dfsadmin -report | grep "Live datanodes"
   ```

4. Access the dashboard via the k2 node's FastAPI/React endpoint.

- **Unit tests**: 19 pytest cases (health/auth, alerts, verdict model, helper functions), executed inside the k2 Docker container:
  ```bash
  docker exec -it <container_name> pytest
  ```
-

## 📊 Key Results

| Metric | Result |
|--------|--------|
| Overall detection accuracy (Stage 1 + Stage 2) | 98% |
| Model comparison | Random Forest is 3.84× faster than GBT in Stage 2 |
| Distributed vs. single-node | scikit-learn fails at full scale (16M rows); Spark succeeds |

> **Known limitation**: The two-stage classifier occasionally confuses DoS and DDoS due to the absence of a multi-source traffic indicator among the 22 selected features — addressed in the thesis as a justification for human-in-the-loop retraining.

---

## 🔐 Security Notes

- Authentication uses **Opaque Session Tokens** (not JWT).
- Monitoring follows a **pull-based** model.
- Flows are **not** labeled by attacker IP, to avoid leaking ground truth into the model.
- IP blocking enforcement is handled inside `create_alert`, returning an `ignored` status for already-blocked IPs.

---

## 🗺️ Future Work

- iptables-level IP blocking and OpenStack Security Group integration
- Automatic blocking based on repeat-offense thresholds, with TTL-based temporary blocks
- MITRE ATT&CK mapping
- MLflow experiment tracking
- Role-Based Access Control (RBAC)
- Automated publishing of retrained models pending quality validation

---

## 👤 Author

**Areej Mohammad** — Graduation Thesis Project

---

## 📄 License

This project is developed for academic purposes as part of a university graduation thesis.