Over the past few days, I designed and implemented a custom Security Operations Center (SOC) platform integrating Elasticsearch, FastAPI, automated detection pipelines, and dynamic incident investigation workflows.

The goal of this project was to move beyond static dashboards and build a real-world security operations architecture capable of ingesting alerts, detecting threats, correlating security events, and preparing the foundation for autonomous incident response.
Core Architecture
The platform currently consists of:
- Elasticsearch for log indexing and security event storage
- Custom Python detection engine for real-time threat detection
- FastAPI backend for authenticated incident ingestion APIs
- Dynamic incident investigation dashboard
- Nginx reverse proxy infrastructure
- Secure API token authentication
- Real-time incident feed pipeline
Detection Pipeline
A custom detection engine continuously queries Elasticsearch for suspicious activity such as failed authentication attempts and brute-force behavior.
When suspicious activity is detected, the engine automatically:
- Correlates matching events
- Extracts attacker IPs and metadata
- Creates structured incidents
- Sends incidents to the FastAPI backend
- Makes incidents immediately available to the investigation dashboard
This creates a fully automated detection-to-investigation workflow.
Challenges & Engineering Lessons
One of the most valuable parts of the project was solving real-world infrastructure and dependency issues, including:
- Elasticsearch client/server version mismatches
- Python virtual environment isolation
- API authentication design
- Reverse proxy configuration
- Incident ingestion architecture
- Dependency management and package pinning
- Secure backend communication
- Production-style debugging workflows
The project evolved from a simple dashboard into a modular security engineering platform.
Challenges that I faced :
- Installed and configured Elasticsearch on a VPS
- Debugged curl: (52) Empty reply from server
- Fixed Elasticsearch 8.x ↔ Python client 9.x incompatibility
- Rebuilt FastAPI incident ingestion routes
- Added Bearer token authentication for /incident
- Recovered broken Python virtual environments
- Fixed PM2 process crashes and backend restart issues
- Created a custom Elasticsearch detection engine in Python
- Connected live detections to a FastAPI incident API
- Synced real incidents into the frontend investigation dashboard
- Replaced mock incident data with live backend ingestion
- Generated reproducible environments using pip freeze > requirements.txt


Current pipeline:
Elasticsearch → Detection Engine → FastAPI → Incident Investigation UI
Finally, seeing live security incidents appear inside the platform in real time.
Current Capabilities
The platform currently supports:
- Real-time incident ingestion
- Token-protected alert APIs
- Dynamic incident visualization
- Elasticsearch correlation
- Security alert pipelines
- Incident investigation workflows
- Severity and category mapping
- Centralized incident feeds