visidata: enhance IOC plugins with improved lookups and validation

Expand iplib, iptype, and ioc plugins with better caching, throttling,
and lookup logic. Update validation script and showcase journal accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
tobias
2026-03-07 22:49:49 +01:00
parent 559fa38c04
commit 49db614262
6 changed files with 608 additions and 44 deletions

View File

@@ -36,13 +36,14 @@ vd --visidata-dir "$PWD" --config "$PWD/visidatarc" --play showcase_ioc.vdj
What it showcases:
- custom types: `IP`, `Domain`, `URL`, `Hash`
- IP membership expressions: `src_ip * network`
- IP network fields: `src_ip.type`, `src_ip.mask`, `src_ip.range`, `src_ip.broadcast`, `src_ip.identity`, `src_ip.hostcount`, `src_ip.rfc_type`
- URL parsing fields: `url.host`, `url.parts.path`, `url.domain`
- hash classification: `file_hash.kind`
- IP lookups: `src_ip.ipinfo.*`, `src_ip.asn.*`, `src_ip.geo.*`, `src_ip.country()`
- provider visibility: `src_ip.geo.source`, `src_ip.asn.source`, `domain.dns.source`
- domain/network intel: `domain.dns.*`, `domain.rdap.*`
- domain/network intel: `domain.dns.*`, `domain.rdap.*`, `domain.resolveip`, `domain.resolveips`, `domain.resolveipv4`, `domain.resolveipv6`
- hash intel: `file_hash.mb.*` (MalwareBazaar)
- VirusTotal lookups: `src_ip.vt.*`, `file_hash.vt.*`, `domain.vt.*`, `url.vt.*`
- VirusTotal lookups: `src_ip.vt.*`, `file_hash.vt.*`, `domain.vt.*`, `url.vt.*` (plus `hash.vt.name`, `hash.vt.names`, `hash.vt.score`, `domain.vt.ip`, `domain.vt.ips`)
- local plugin command: `tke-hidecol`
Lookup notes:
@@ -82,15 +83,24 @@ Membership test:
Lookup objects expose both normalized fields and raw response data:
- `ipcol.type` (`ipv4`/`ipv6`/`cidr4`/`cidr6`), `ipcol.family`, `ipcol.is_cidr`
- `ipcol.mask`, `ipcol.netmask`, `ipcol.identity`, `ipcol.broadcast`, `ipcol.range`, `ipcol.hostcount`, `ipcol.address_count`
- `ipcol.rfc_type` (classification: e.g. `global`, `private`, `documentation`, `shared`, `link-local`, ...)
- `ipcol.ipinfo.country`
- `ipcol.ipinfo.data.<any_json_field>`
- `ipcol.asn.asn`, `ipcol.asn.name`, `ipcol.asn.country`
- `ipcol.asn.data.<any_json_field>`
- `ipcol.vt.verdict` (e.g. `"3/94"`), `ipcol.vt.malicious`, `ipcol.vt.total`, `ipcol.vt.category` (alias: `ipcol.vt.type`)
- `ipcol.vt.verdict` (e.g. `"3/94"`), `ipcol.vt.score`, `ipcol.vt.malicious`, `ipcol.vt.total`, `ipcol.vt.category` (alias: `ipcol.vt.type`)
- `ipcol.vt.data.<any_json_field>`
- `ipcol.geo.*` (best-available geo: prefers MaxMind mmdb, else free HTTP providers)
- `ipcol.maxmind.*` (offline-only MaxMind lookup; empty if no mmdb)
Type shortcuts on table-like sheets:
- `;i` -> `type-ip`
- `;d` -> `type-domain`
- `;u` -> `type-url-ioc`
- `;h` -> `type-hash`
#### Caching
All lookup providers cache results in a local sqlite+pickle DB (default `~/.visidata_cache.db`).
@@ -124,6 +134,28 @@ Pure-Python library used by `iptype.py` for:
This file intentionally does **not** import VisiData so it can be validated outside the VisiData runtime.
### VT schema (`*.vt`)
`ip.vt`, `domain.vt`, `url.vt`, and `hash.vt` expose a normalized shape for quick querying across free + premium responses, while still preserving full raw JSON:
Common fields:
- `verdict` (`"malicious/total"`)
- `score` / `confidence` (`malicious/total` float)
- `malicious`, `suspicious`, `harmless`, `undetected`, `timeout`, `total`
- `category` / `categories`
- `reputation`, `votes_harmless`, `votes_malicious`
- `tags`, `last_analysis_date`, `last_modification_date`
- `results` (normalized engine results map), `stats`, `data` (full raw API response)
Object-specific conveniences:
- `ip.vt`: `asn`, `as_owner`, `country`, `continent`, `network`
- `domain.vt`: `ip` (best/last known), `ips` (all extracted A/AAAA)
- `url.vt`: URL-level verdict/score plus direct raw access via `url.vt.attrs.*`
- `hash.vt`: `name` (best malware name), `names` (all extracted names), plus verdict/score
Raw passthrough:
- Any VT `attributes` field is also available via `obj.vt.<attribute_name>` and `obj.vt.attrs.<attribute_name>`.
## Config: `visidatarc`
This repos `visidatarc` is intended to be installed as VisiDatas `config.py`: