Update README

README.md

-# نظام أمني ذكي موزّع لكشف التسلل في البيئات السحابية
-### Smart Distributed Intrusion Detection System (Smart-IDPS)
+# Smart-IDPS
+### Smart Distributed Intrusion Detection and Prevention System for Cloud Environments
 
-نظام كشف تسلل (IDS) موزّع، مبني على Apache Hadoop وApache Spark، يعمل على عنقود (Cluster) من ثلاث عقد فعلية، ويستخدم نموذج تعلّم آلي هرمي ثنائي المرحلة (Random Forest) لتصنيف حركة الشبكة إلى طبيعية أو مهاجمة، وتحديد نوع الهجوم عند اكتشافه.
+A distributed, real-time network security monitoring system built on a multi-node OpenStack cluster, leveraging Apache Hadoop and Apache Spark for large-scale traffic analysis, and a two-stage machine learning pipeline for intrusion detection.
 
 ---
 
-## 1. لمحة عامة عن المشروع
+## 📌 Overview
 
-يهدف المشروع إلى بناء نظام قادر على:
-- تجميع سجلات حركة الشبكة من سيرفرات افتراضية متعددة عبر Apache Kafka
-- تخزينها بشكل موزّع ومتسامح مع الأعطال عبر Hadoop HDFS
-- تحليلها وتصنيفها في الوقت الفعلي عبر Apache Spark ونموذج تعلّم آلي مدرَّب مسبقاً
-- إثبات الجدوى الهندسية لاستخدام بيئة موزّعة (Spark على عنقود من 3 عقد) مقارنة بالمعالجة المركزية (Single-Node)، من خلال تجارب مقارنة موثّقة (زمن التدريب، الاستدلال، الدقة، وقابلية التوسّع)
+Smart-IDPS monitors network traffic across a distributed cloud environment in real time, detects malicious behavior (DDoS, Brute Force, DoS), classifies the type of attack, and enables both automatic and manual incident response — including IP blocking — through a live analyst dashboard.
+
+The project was developed as a graduation thesis and demonstrates the practical necessity of distributed computing (Hadoop + Spark) for intrusion detection at scale, backed by benchmark comparisons against traditional single-node approaches.
+
+---
+
+## 🏗️ System Architecture
+
+The system runs on a **three-node OpenStack cluster**:
+
+| Node | IP | Services |
+|------|-----|----------|
+| **k1** | 10.0.0.15 | HDFS NameNode, Kafka, Zookeeper, Spark Master |
+| **k2** | 10.0.0.124 | FastAPI backend, React dashboard, Spark Worker, Filebeat, CICFlowMeter |
+| **k3** | 10.0.0.18 | PostgreSQL, HDFS DataNode, Spark Worker |
+
+### Data Flow (Pipeline)
+
+```
+pcap files → pcap-watcher → CICFlowMeter → CSV → traffic_dripper
+     → Filebeat → Kafka → Spark Streaming → FastAPI → PostgreSQL → React Dashboard
+```
+
+### Machine Learning Model
+
+A **two-stage hierarchical classifier**:
+
+1. **Stage 1 — Binary Detection**: Logistic Regression determines whether traffic is `Benign` or an `Attack`.
+2. **Stage 2 — Multiclass Attack Typing**: Random Forest (trained via Spark ML) classifies the attack type — `DDoS`, `Brute Force`, or `DoS`.
 
 ---
 
-## 2. بنية العنقود (Cluster Architecture)
+## ⚙️ Tech Stack
 
-| العقدة | الدور | المكوّنات المستضافة |
-|---|---|---|
-| **k1** | العقدة الرئيسية (Master) | Spark Master, Hadoop NameNode, Kafka, Zookeeper |
-| **k2** | عقدة عاملة (Worker) | Spark Worker, Hadoop DataNode,  |
-| **k3** | عقدة عاملة (Worker) | Spark Worker, Hadoop DataNode, PostgreSQL |
+**Infrastructure & Orchestration**
+- OpenStack (KVM), Ubuntu 24.04 LTS
+- Docker, Docker Compose
 
-كل عقدة تُدار عبر ملف `docker-compose.yml` مستقل ضمن مجلدها الخاص (`k1/`, `k2/`, `k3/`)، وتتصل ببعضها عبر شبكة داخلية باستخدام عناوين IP الثابتة للأجهزة.
+**Big Data & Streaming**
+- Apache Kafka, Apache Zookeeper
+- Apache Spark (Streaming + MLlib)
+- Hadoop HDFS
+
+**Machine Learning & Data**
+- CSE-CIC-IDS2018 dataset
+- CICFlowMeter
+- Filebeat
+
+**Backend & Storage**
+- FastAPI (REST + WebSocket)
+- PostgreSQL (psycopg2, ThreadedConnectionPool)
+
+**Frontend**
+- React, Vite, React Router
+
+**Testing**
+- Pytest (19 unit tests, run inside Docker)
+- Apache JMeter 5.6.3
+
+**External Services**
+- AbuseIPDB (IP reputation lookup, 24h caching)
 
 ---
 
-## 3. تدفّق البيانات (Data Flow)
+## 📂 Repository Structure
+
+```
+smart-idps/
+├── k1-node/          # HDFS NameNode, Kafka, Zookeeper, Spark Master configs
+├── k2-node/           # FastAPI backend, React dashboard, Spark Worker
+├── k3-node/           # PostgreSQL, HDFS DataNode, Spark Worker
+├── ml-pipeline/       # Training scripts, feature engineering, model artifacts
+├── report/            # Academic thesis documentation (Arabic)
+└── README.md
+```
+
+> **Note:** Large binary artifacts (trained models, `.pcap` captures, datasets, `node_modules/`) are excluded from version control via `.gitignore` due to size constraints. See [Model & Dataset Access](#-model--dataset-access) below.
+
+---
+
+## 🚀 Getting Started
+
+### Prerequisites
+- Docker & Docker Compose installed on all three nodes
+- Internal network connectivity between nodes (`10.0.0.x` — **do not use floating IPs**, they are NAT addresses and will break inter-node communication)
 
-آلية التشغيل (Setup & Run)
+### Setup
 
-### المتطلبات الأساسية
-- Docker وDocker Compose على كل عقدة من الثلاث
-- اتصال شبكي بين العقد الثلاث (نفس الشبكة الداخلية)
-- تحديث عناوين IP في ملفات `docker-compose.yml` لتطابق عناوين الأجهزة الفعلية الحالية
+1. Clone the repository on each node:
+   cd ids-project
+   ```
+
+2. Navigate to the relevant node directory (`k1-node`, `k2-node`, or `k3-node`) and start its services:
+   ```bash
+   docker-compose up -d
+   ```
+   > If you change a `docker-compose.yml` file, use `docker-compose up -d --force-recreate` — `restart` does **not** apply config changes.
+
+3. Verify HDFS DataNode connectivity:
+   ```bash
+   docker exec namenode hdfs dfsadmin -report | grep "Live datanodes"
+   ```
+
+4. Access the dashboard via the k2 node's FastAPI/React endpoint.
+
+- **Unit tests**: 19 pytest cases (health/auth, alerts, verdict model, helper functions), executed inside the k2 Docker container:
+  ```bash
+  docker exec -it <container_name> pytest
+  ```
+-
+
+## 📊 Key Results
+
+| Metric | Result |
+|--------|--------|
+| Overall detection accuracy (Stage 1 + Stage 2) | 98% |
+| Model comparison | Random Forest is 3.84× faster than GBT in Stage 2 |
+| Distributed vs. single-node | scikit-learn fails at full scale (16M rows); Spark succeeds |
+
+> **Known limitation**: The two-stage classifier occasionally confuses DoS and DDoS due to the absence of a multi-source traffic indicator among the 22 selected features — addressed in the thesis as a justification for human-in-the-loop retraining.
+
+---
+
+## 🔐 Security Notes
+
+- Authentication uses **Opaque Session Tokens** (not JWT).
+- Monitoring follows a **pull-based** model.
+- Flows are **not** labeled by attacker IP, to avoid leaking ground truth into the model.
+- IP blocking enforcement is handled inside `create_alert`, returning an `ignored` status for already-blocked IPs.
+
+---
+
+## 🗺️ Future Work
+
+- iptables-level IP blocking and OpenStack Security Group integration
+- Automatic blocking based on repeat-offense thresholds, with TTL-based temporary blocks
+- MITRE ATT&CK mapping
+- MLflow experiment tracking
+- Role-Based Access Control (RBAC)
+- Automated publishing of retrained models pending quality validation
+
+---
+
+## 👤 Author
+
+**Areej Mohammad** — Graduation Thesis Project
+
+---
 
-### خطوات التشغيل
+## 📄 License
 
-**1. على العقدة الرئيسية (k1):**
-```bash
+This project is developed for academic purposes as part of a university graduation thesis.
\ No newline at end of file
-cd k1/
-sudo docker compose up -
\ No newline at end of file