Skip to content
Brand Protect Docs

The Python Microservice

The Python 3.14 microservice (services/) performs the heavy network, AI, and document-generation work. It is a FastAPI application (services/src/main.py) exposing routes under /api, with OpenTelemetry instrumentation, on port 8000.

Package layout

Section titled “Package layout”
services/src/
├── main.py                    # FastAPI app, /health, instrumentation
├── api/routes.py              # authenticated /v1 endpoints
├── orchestrator/              # pipeline runner
│   ├── __init__.py            # start_scan: 8-stage orchestration
│   ├── phases.py              # run_phase helper (thread + timeout)
│   ├── enrichment.py          # background domain/social enrichment
│   ├── state.py               # ScanState / cancellation
│   └── keywords.py            # keyword merging
├── domain_intel/              # generation, discovery, RDAP/WHOIS,
│   │                          # severity, auditing, SSL, infra, tech detection
│   └── similarity/            # ssdeep/tlsh/visual pHash
├── social/                    # handle enumeration, availability,
│   │                          # generation, dorking, identity matching, profiles
│   ├── availability.py        # (Maigret probe)
│   └── identity_matching.py   # embedding-based matching
├── scoring/                   # signals, weights, score, threats,
│   │                          # campaigns, brand_profile, keywords, learner
│   └── learner.py             # Bayesian weight updates
├── ai/
│   ├── core/                  # provider abstraction + settings
│   └── mcp/                   # MCP stdio tool servers (audit, keywords)
├── reports/
│   ├── pdf_report.py          # WeasyPrint full + summary PDFs
│   └── takedown.py            # abuse-report email builder
├── messaging/redis_client.py  # pub/sub publisher
├── learning/                  # adaptive weight learning
├── config/                    # settings.py, logging.py (JSON)
├── core/models.py             # pydantic domain/social models
└── utils/cancellation.py

HTTP surface

Section titled “HTTP surface”

See Internal API for the full endpoint reference. The authenticated endpoints are:

Method Path Purpose
GET /health Liveness (unauthenticated)
POST /v1/scan/start Run the full pipeline
POST /v1/domain/enrich Enrich one owned domain
POST /v1/social/enrich Enrich one social handle
POST /v1/brand/keyword_expand LLM keyword inference
POST /v1/report/generate Full + summary PDFs (base64)
POST /v1/campaign/narratives LLM campaign narratives
POST /v1/takedown/generate Takedown email draft

All authenticated endpoints accept 202 and run heavy work in BackgroundTasks. Auth is a shared INTERNAL_API_TOKEN bearer token.

The pipeline runner

Section titled “The pipeline runner”

orchestrator/__init__.py::start_scan drives the eight stages. Phases 2–4 run in parallel via asyncio.gather; each phase runs in a worker thread via run_phase with a timeout and publishes start/completion status to Redis. Resume support lets a terminal scan restart from any stage.

Messaging

Section titled “Messaging”

messaging/redis_client.py publishes:

  • Progress — scan_db_updates and per-scan scan_<id> event streams.
  • Findings — scan_findings (initial batch + deltas).
  • Profiles — domain_profiles, social_profiles.
  • Social findings — social_findings.

The Rails scan_db_consumer subscribes to these to persist data and broadcast to the browser. See Messaging.

AI provider abstraction

Section titled “AI provider abstraction”

ai/core/settings.py loads the AI configuration with the admin panel as the source of truth:

  • If a PlatformSetting("ai_provider") is configured, its enabled + config win.
  • Environment variables are fallbacks only — they fill config gaps (e.g. the API key per provider) but never auto-enable a provider.
  • _env_api_key() resolves AI_API_KEY first, then ANTHROPIC_API_KEY (anthropic), GOOGLE_API_KEY (google), else OPENAI_API_KEY.

When a provider is unavailable or a call fails, the service never blocks a scan or report — it falls back to deterministic generation and logs a warning (report_ai_provider_unavailable, ai_insights_skipped_no_provider, report_ai_deterministic_fallback, ai_audit_skipped_not_enabled).

Reliability properties

Section titled “Reliability properties”
  • Deterministic fallbacks — reports, takedowns, and insights never depend on a model call succeeding.
  • Graceful degradation — optional dependencies (pyjarm, shodan, censys, securitytrails, PIL, ssdeep/tlsh) are skipped when unavailable.
  • Bounded retries — enrichment retries once on total failure; keyword expansion has configurable retries.
  • Structured logging — JSON logs with OpenTelemetry trace/span IDs.

Next

Section titled “Next”