Enrichment

Enrichment (stages 2 and 3) gathers evidence about every discovered domain. Stage 2 — record information — collects registration data; stage 3 — fingerprinting — profiles the host. Both run in parallel with handle enumeration and stream results as deltas as each finding is enriched.

Record information (stage 2)

record_info.py with whois_fallback.py:

RDAP is the primary source — structured registration data with registrant and abuse contacts.
WHOIS is the fallback when RDAP is unavailable, parsed into the same RegistrationData shape.
Includes abuse-contract resolution for takedown contacts.

Fingerprinting (stage 3)

fingerprinting/pipeline.py coordinates several probes:

Probe	Module	Data
DNS	—	A/AAAA/MX/NS/TXT records
HTTP	—	content, title, headers, redirects, HTML fingerprint (`html_lsh` via ssdeep/tlsh)
TLS/SSL	`ssl_info.py`	certificate, SANs, issuer, validity
Screenshot	—	page screenshot stored under the screenshots volume
Tech detection	`tech_detection.py`	framework/tech fingerprinting
Infrastructure	`infra_intel.py`	IP, ASN, hosting provider, co-hosted domains, JARM fingerprint, open ports

Optional data sources

Infrastructure and threat enrichment use optional API keys and degrade gracefully when absent:

Shodan (SHODAN_API_KEY) — host intelligence.
SecurityTrails (SECURITYTRAILS_API_KEY) — historical DNS.
Censys (CENSYS_API_KEY) — certificate/asset intelligence.
JARM (JARM_ENABLED) — TLS fingerprinting (pyjarm; skipped when unavailable).

Enrichment of owned domains

Beyond scans, the platform enriches owned domains and social handles in the background as soon as they are added to the brand profile:

EnrichDomainJob → POST /v1/domain/enrich → builds a domain_profiles snapshot used as the baseline in later scans.
EnrichSocialHandleJob → POST /v1/social/enrich → builds a social_profiles snapshot.

These run with per-resource cancellation, a concurrency semaphore (2 workers), and a one-shot retry on total failure. Results carry an enrichment_generation; the Rails consumer drops stale updates.

Configuration

Variable	Purpose
`SOCIAL_ENRICH_TIMEOUT`	timeout for social enrichment (default 120)
`JARM_ENABLED`	toggle JARM fingerprinting
`SHODAN_API_KEY`, `SECURITYTRAILS_API_KEY`, `CENSYS_API_KEY`	optional infra sources
`AVATARS_DIR` / `SCREENSHOTS_DIR`	where media is stored