Skip to content

kick707/uti

BranchesTags

Repository files navigation

Ultimate Threat Intelligence

A comprehensive, automated threat intelligence platform that combines real-time SSL certificate monitoring with multi-source threat feed aggregation. The system enriches all collected data with geolocation, WHOIS, DNS, and behavioural analysis before indexing to Elasticsearch for security analytics and threat hunting.

Overview

The platform operates in two modes:

  1. Real-Time Certificate Monitoring – Continuously tracks global SSL/TLS certificate issuance via CertStream to detect suspicious domains and potential phishing infrastructure.
  2. Scheduled Threat Feed Collection – Aggregates indicators of compromise (IOCs) from multiple open-source and commercial threat intelligence feeds.

All collected data is enriched with contextual metadata and stored in Elasticsearch for correlation, analysis, and visualisation via Kibana.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Data Sources                              │
├─────────────────────────────────────────────────────────────┤
│  • CertStream (Real-time SSL certificates)                  │
│  • OpenPhish, PhishStats, PhishingDB (Phishing feeds)       │
│  • Cyware TAXII, MISP (Threat intelligence platforms)       │
│  • TOR Project (Exit node lists)                            │
│  • C2 Tracker (Command & control infrastructure)            │
│  • TweetFeed (Social media threat intelligence)             │
│  • IP/Domain Blacklists (Community blocklists)              │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                  Collection Layer                            │
├─────────────────────────────────────────────────────────────┤
│  certstream_monitor.py  │  collectors/… (scheduled)         │
│  └─ lib/threat_intel.py (enrichment library)                │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                  Enrichment Services                         │
├─────────────────────────────────────────────────────────────┤
│  • DNS (SOA, SPF, DMARC, DKIM, BIMI, A records)             │
│  • WHOIS (Registration data, nameservers)                   │
│  • Geolocation (IP-API, IPInfo, IPRegistry)                 │
│  • SSL/TLS Certificate Parsing                              │
│  • Threat Scoring (Phishing risk assessment)                │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                Storage & Analytics                           │
├─────────────────────────────────────────────────────────────┤
│  Elasticsearch Cloud  │  Elasticsearch Local  │  Kibana      │
└─────────────────────────────────────────────────────────────┘

Project Structure

uti/
├── README.md                       # Project documentation
├── LICENSE                         # MIT licence
├── .env.example                    # Environment variables template
├── .gitignore                      # Git exclusions
├── requirements.txt                # Python dependencies
│
├── docker-compose.yml              # Full stack orchestration
├── Dockerfile                      # Container for certstream_monitor
│
├── certstream_monitor.py           # Main application: real-time SSL monitoring
├── lib/
│   └── threat_intel.py             # Core enrichment library
├── config/
│   └── scoring.py                  # Threat scoring configuration
│
├── collectors/
│   ├── phishing/                   # Phishing feed collectors
│   ├── infrastructure/             # Infrastructure & blacklist collectors
│   └── platforms/                  # Threat intelligence platform collectors
└── README.md

Core Components

Main Application – certstream_monitor.py

The long-running service that ingests live certificate transparency events, enriches SAN entries, and indexes enriched documents into both Elastic Cloud and a local Elasticsearch node.

Operational Flow:

  1. Connect to the CertStream WebSocket feed (ws://certstream:8080/).
  2. Receive a live stream of newly issued certificates and parse SAN entries.
  3. Enrich each domain with threat intelligence:
    • DNS Analysis – SOA, SPF, DMARC, DKIM, BIMI, A records
    • WHOIS Context – registrar metadata, creation/expiry dates, registrant email
    • SSL/TLS Details – issuer, validity window, certificate chain, SNI-aware checks
    • Geolocation – country, city, ASN, ISP from IP-API/IPInfo/IPRegistry
    • Infrastructure Detection – tunnelling services, cloud hosting footprints
    • Behavioural Scoring – phishing risk, email security posture, string heuristics
  4. Index enriched documents into Elasticsearch Cloud (es1) and local Elasticsearch (es2).
  5. Automatically reconnect and resume on transient network failures.

Detection capabilities include:

  • Newly registered phishing domains and typosquats.
  • Tunnelling and anonymisation services (ngrok, localhost.run, Tor, etc.).
  • Cloud-hosted infrastructure impersonating brands.
  • Language heuristics to identify targeted demographics.
  • Email security posture (SPF/DMARC/DKIM/BIMI) scoring.

Core Library – lib/threat_intel.py

The enrichment hub (~1,500 lines) powering every collector. Key capabilities include:

Elasticsearch integration

  • es1 / es2 clients for Elastic Cloud and local clusters.
  • get_index_name() and get_data() helpers for dynamic index selection and document construction.
  • Bulk helpers and retry logic that feed the pipeline.

DNS & WHOIS utilities

  • get_soa_record(), get_spf_record(), get_dmarc_record(), get_dkim_record(), get_bimi_record(), get_a_record(), convert_soa_email().
  • get_whois(), get_registrar(), get_name_servers() for ownership context.

IP geolocation fallbacks

  • get_ip_info_ip_api() (primary service with built-in throttling).
  • get_ip_info_ipinfo() and get_ip_details_ipregistry() fallbacks.
  • get_ip_data() orchestration that merges best-effort attributes.

SSL/TLS analysis

  • SSLUtils helper for SNI-aware certificate retrieval and PEM/X509 conversion.
  • Certificate metadata extraction inside get_data() (issuer, validity, serial).

Domain heuristics & string analysis

  • detect_language(), is_tunnelling_service(), is_cloud_provider().
  • count_special_characters(), count_numeric_characters(), count_consonant_characters(), count_vowel_characters().
  • count_dots_in_domain(), count_hyphens_in_fqdn_url(), calculate_consonant_sequence(), calculate_vowel_sequence(), count_syllables(), word_count().

Enrichment extras

  • get_fav_icon() and get_screenshot() (ScreenshotMachine integration).
  • postIP() for AbuseIPDB submissions.
  • Automatic rate limiting, request timers, and circuit breakers for API hiccups.

Configuration bindings

  • Reads .env for ELASTIC_PASSWORD, CLOUD_ID, ES_LOCAL_PASSWORD, ES_LOCAL_HOST, and optional API keys (IPINFO_API_KEY, IPREGISTRY_API_KEY, SCREENSHOTMACHINE_KEY, IMGBB_API_KEY, ABUSE_IPDB_API_KEY, ALIEN_VAULT_OTX_API_KEY).
  • Gracefully downgrades functionality when keys are absent.

Threat scoring

  • Pulls weights from config/scoring.py and annotates documents with phishing/TOR/blacklist scores.

Threat Scoring – config/scoring.py

Defines the scalar weights that feed the overall risk score (phishing index, TOR index, blacklist penalties, etc.). Adjust these values to tune alerting thresholds.

Threat Feed Collectors

All collectors live under collectors/ and are designed to run as modules (python -m collectors.phishing.openphish). Each collector writes rotating logs to ./logs/, enriches IOCs via lib.threat_intel, and indexes into Elasticsearch.

Phishing Feeds (collectors/phishing/)

  • openphish.py – Fetches OpenPhish active URLs (suggested cadence: daily at 08:00). Enrichment: full domain analysis + risk scoring → index sk_v5_geo_fish1_{MM.YYYY}.
  • phishingdb.py – Pulls the MitchellKrogza phishing list (daily). Enrichment: WHOIS/DNS/IP context → index sk_v5_geo_fish4_{MM.YYYY}.
  • phishstats.py – Multi-threaded ingestion of PhishStats CSV exports (every 2 hours). Recomputes scores based on screenshot/email indicators → index sk_v5_geo_fish3_score_{MM.YYYY}.
  • tweetfeed.py – Polls TweetFeed for same-day phishing sightings (hourly). Enrichment: domain + IP context → index sk_v5_geo_fish5_{MM.YYYY}.

Infrastructure Intelligence (collectors/infrastructure/)

  • blacklists.py – Aggregates community IP/domain blocklists (daily). Handles file downloads, deduplication, geolocation, and bulk indexing → index sk_v5_geo_blacklist_{MM.YYYY}.
  • tor_nodes.py – Tracks TOR exit nodes from multiple sources (every 6 hours). Marks anonymisation infrastructure and enriches IP metadata → index sk_v5_geo_torexit_{MM.YYYY}.
  • c2_tracker.py – Ingests the mthcht C2 Tracker project (daily). Uses 20 worker threads and 5,000-document bulk chunks to index 50+ C2 frameworks → index sk_v5_geo_ta_ttps_c2_ips_{MM.YYYY}.

Threat Intelligence Platforms (collectors/platforms/)

  • cyware_taxii.py – Connects to Cyware CTIX via TAXII 2.1 (suggested every 4 hours). Discovers collections, paginates STIX 2.1 objects, and indexes indicators + TTPs with context.
  • misp.py – Pulls attributes from a MISP instance (suggested every 4 hours). Supports tag filters, event context, and enriches network indicators prior to indexing.

Docker Infrastructure

The compose stack provisions the complete environment:

Service Role
setup Bootstraps Elasticsearch certificates and users
es01 Single-node Elasticsearch (TLS enabled)
kibana Kibana UI
web Nginx reverse proxy (optional)
certstream CertStream server (0rickyy0/certstream-server-go)
domrat Application container (runs certstream_monitor)

Persistent volumes:
certs (TLS material), esdata01 (Elasticsearch data), kibanadata (Kibana state)

Network:
Custom bridge network domrat.

Quick start

docker build -t ultimate-threat-intel .
docker compose up --build

# Monitor health
docker compose logs -f setup es01 kibana domrat

# Smoke-test a collector
docker compose exec domrat python -m collectors.phishing.openphish

# Shutdown (add -v to purge volumes)
docker compose down

Installation & Setup (Local Python)

git clone https://github.com/kick707/uti.git
cd uti

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
# Edit .env to set credentials and API keys

# Optional: verify Elasticsearch connectivity
python -c "from lib.threat_intel import es1; print(es1.ping())"

Run collectors as modules:

python -m collectors.phishing.openphish
python -m collectors.infrastructure.c2_tracker
python -m collectors.platforms.cyware_taxii

Credentials

Required (Core Functionality)

Configure these to unlock the core collectors (Elasticsearch, MISP, Cyware).

Elasticsearch

ELASTIC_PASSWORD=your_elastic_cloud_password
CLOUD_ID=your_elastic_cloud_id
ES_LOCAL_PASSWORD=your_local_elasticsearch_password  # Optional if local ES disabled
ES_LOCAL_HOST=https://es01:9200                      # Optional override

MISP (collectors/platforms/misp.py)

MISP_URL=http://your-misp-instance:8880
MISP_KEY=your_misp_api_key

Cyware TAXII (collectors/platforms/cyware_taxii.py)

TAXII_USERNAME=your_taxii_username
TAXII_PASSWORD=your_taxii_password

Optional Integrations

IPINFO_API_KEY=your_ipinfo_api_key              # IP geolocation fallback
IPREGISTRY_API_KEY=your_ipregistry_api_key      # Alternative geolocation
ABUSE_IPDB_API_KEY=your_abuseipdb_api_key       # Threat reporting
SCREENSHOTMACHINE_KEY=your_screenshotmachine_key
IMGBB_API_KEY=your_imgbb_api_key
ALIEN_VAULT_OTX_API_KEY=your_alienvault_otx_api_key

Cron / Scheduling Reference

Adapt the schedule to your environment:

# Phishing feeds
0 8 * * *   /opt/uti/.venv/bin/python -m collectors.phishing.openphish
0 8 * * *   /opt/uti/.venv/bin/python -m collectors.phishing.phishingdb
0 */2 * * * /opt/uti/.venv/bin/python -m collectors.phishing.phishstats
0 * * * *   /opt/uti/.venv/bin/python -m collectors.phishing.tweetfeed

# Infrastructure intelligence
0 9 * * *   /opt/uti/.venv/bin/python -m collectors.infrastructure.blacklists
0 */6 * * * /opt/uti/.venv/bin/python -m collectors.infrastructure.tor_nodes
0 10 * * *  /opt/uti/.venv/bin/python -m collectors.infrastructure.c2_tracker

# Threat platforms
0 */4 * * * /opt/uti/.venv/bin/python -m collectors.platforms.cyware_taxii
0 */4 * * * /opt/uti/.venv/bin/python -m collectors.platforms.misp

Data Flow

SSL Certificate → CertStream → certstream_monitor.py
                               ↓
                        lib/threat_intel.py
                     (DNS/WHOIS/IP/SSL enrichment)
                               ↓
                       config/scoring.py weights
                               ↓
                 ┌─────────────┴─────────────┐
                 ↓                           ↓
        Elasticsearch Cloud          Elasticsearch Local
                 ↓                           ↓
                 └─────────────┬─────────────┘
                               ↓
                             Kibana
                (Dashboards, alerts, investigations)

API Rate Limits

Service Limit Handling
IP-API.com 45 requests/minute Automatic throttling + fallback to IPInfo
IPInfo.io 50,000 requests/month Secondary enrichment provider
IPRegistry.co 100,000 requests/month Tertiary enrichment provider
AbuseIPDB 1,000 requests/day Used only for explicit reporting
ScreenshotMachine Plan dependent Optional feature activated via API key

Security Considerations

Implemented

  • Credentials stored in environment variables; .env ignored by git.
  • Elasticsearch TLS enabled with automated certificate provisioning.
  • Separate API keys for each integration.
  • Rotating log handlers to avoid leaking secrets.

Recommendations

  1. Rotate credentials every 90 days.
  2. Apply IP allowlists for Elasticsearch endpoints.
  3. Use least-privilege API keys (read-only when possible).
  4. Enable Elastic audit logging and monitor for anomalies.
  5. Review threat scoring weights quarterly.
  6. Monitor collector execution metrics and logs.

Performance Optimisation

  • Elasticsearch tuning – Adjust bulk.flush_interval, indices.memory.index_buffer_size, and thread_pool.write.queue_size when sustained ingestion exceeds default thresholds.
  • Collector concurrency – ThreadPoolExecutor is used in select collectors (e.g., PhishStats, C2 tracker); tune worker counts based on CPU/network capacity.
  • Bulk chunk sizing – Collectors use 5,000-document chunking for Elasticsearch bulk indexing; adjust if you encounter memory pressure.

Monitoring & Logging

  • All collectors log to ./logs/ with rotating file handlers.
  • Use docker compose logs or Kibana to monitor pipeline health.
  • Metrics to track: indexing throughput, enrichment success rate, API failures, cluster health.

Troubleshooting

Symptom Quick Check
Elasticsearch ping fails python -c "from lib.threat_intel import es1; print(es1.info())"
Missing dependencies pip install -r requirements.txt --upgrade
IP-API rate limit exceeded Review logs for fallback activation (logs/*.log)
CertStream disconnects Monitor docker compose logs certstream; auto-reconnect is enabled
Collector not indexing Check logs/<collector>.log for exceptions and review .env credentials

Contributing

Contributions are welcome. Please:

  • Follow PEP 8.
  • Include docstrings for complex logic.
  • Add tests or validation steps for new behaviour.
  • Update documentation when introducing new collectors or configuration flags.
  • Never commit credentials or secrets.

Acknowledgments

  • CertStream – SSLMate for real-time certificate transparency
  • Threat Feeds – OpenPhish, PhishStats, PhishingDB, TweetFeed
  • Community – C2 Tracker (mthcht), DAN.ME.UK (TOR lists)
  • Platforms – Cyware, MISP Project
  • Infrastructure – Elastic Stack

Support

License

MIT © 2025 kick707 – See the LICENSE file for details.

Developed for cybersecurity professionals. Use responsibly.

About

Ultimate Threat Intelligence

Resources

Readme

License

MIT license
Activity

Stars

4 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages