I’ve been building and debugging a custom AI-driven SOC platform from the ground up using Elasticsearch, FastAPI, Python detection pipelines, and a live investigation dashboard.

This started as a simple SOC dashboard experiment, but quickly evolved into a real security engineering project involving backend architecture, detection engineering, incident processing, API authentication, and live frontend synchronization.

What has been implemented so far:

  • Installed and configured Elasticsearch on a VPS
  • Built a custom Python detection engine querying Elasticsearch directly
  • Created live brute-force and suspicious activity detections
  • Built a FastAPI-based incident ingestion backend
  • Added token-authenticated /incident APIs
  • Implemented incident normalization pipelines
  • Added MITRE ATT&CK mapping enrichment
  • Created a dynamic Risk Scoring Engine
  • Connected incidents to a live frontend investigation dashboard
  • Replaced static/mock data with real-time incident ingestion
  • Built live sync workflows between backend and frontend
  • Fixed PM2 crashes and backend restart failures
  • Solved Elasticsearch 8.x ↔ Python client incompatibilities
  • Recovered broken Python virtual environments
  • Built reproducible environments using pip freeze > requirements.txt

A big part of this project wasn’t just coding — it was solving real infrastructure and engineering problems:

  • curl: (52) Empty reply from server
  • FastAPI route failures
  • PM2 process crashes
  • Elasticsearch authentication issues
  • API version mismatches
  • Python dependency conflicts
  • backend/frontend integration bugs
  • incident rendering issues
  • live sync failures

Current architecture:

Elasticsearch→ Custom Detection Engine→ Incident Normalization→ MITRE ATT&CK Mapping→ Risk Scoring Engine→ FastAPI Incident API→ Live Incident Investigation

Dashboard

The platform is now successfully ingesting and displaying real security incidents live from Elasticsearch into the investigation UI.

Some of the commands/workflows used during the build:

python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn elasticsearch requests
uvicorn main:app --host 0.0.0.0 --port 80..
pip freeze > requirements.txt
curl http://localhost:80../api/incident

Brute-force incident testing:

curl -X POST http://localhost:80../incident \
-H "Authorization: Bearer Token.. \
-H "Content-Type: application/json" \
-d '{
"title":"RDP Bruteforce",
"severity":"critical",
"category":"unauthorized_access",
"ip":"192.168.1.0"
}'

Slowly turning this into a real AI-powered security operations platform, one layer at a time.