visidata: enhance IOC plugins with improved lookups and validation
Expand iplib, iptype, and ioc plugins with better caching, throttling, and lookup logic. Update validation script and showcase journal accordingly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -36,13 +36,14 @@ vd --visidata-dir "$PWD" --config "$PWD/visidatarc" --play showcase_ioc.vdj
|
||||
What it showcases:
|
||||
- custom types: `IP`, `Domain`, `URL`, `Hash`
|
||||
- IP membership expressions: `src_ip * network`
|
||||
- IP network fields: `src_ip.type`, `src_ip.mask`, `src_ip.range`, `src_ip.broadcast`, `src_ip.identity`, `src_ip.hostcount`, `src_ip.rfc_type`
|
||||
- URL parsing fields: `url.host`, `url.parts.path`, `url.domain`
|
||||
- hash classification: `file_hash.kind`
|
||||
- IP lookups: `src_ip.ipinfo.*`, `src_ip.asn.*`, `src_ip.geo.*`, `src_ip.country()`
|
||||
- provider visibility: `src_ip.geo.source`, `src_ip.asn.source`, `domain.dns.source`
|
||||
- domain/network intel: `domain.dns.*`, `domain.rdap.*`
|
||||
- domain/network intel: `domain.dns.*`, `domain.rdap.*`, `domain.resolveip`, `domain.resolveips`, `domain.resolveipv4`, `domain.resolveipv6`
|
||||
- hash intel: `file_hash.mb.*` (MalwareBazaar)
|
||||
- VirusTotal lookups: `src_ip.vt.*`, `file_hash.vt.*`, `domain.vt.*`, `url.vt.*`
|
||||
- VirusTotal lookups: `src_ip.vt.*`, `file_hash.vt.*`, `domain.vt.*`, `url.vt.*` (plus `hash.vt.name`, `hash.vt.names`, `hash.vt.score`, `domain.vt.ip`, `domain.vt.ips`)
|
||||
- local plugin command: `tke-hidecol`
|
||||
|
||||
Lookup notes:
|
||||
@@ -82,15 +83,24 @@ Membership test:
|
||||
|
||||
Lookup objects expose both normalized fields and raw response data:
|
||||
|
||||
- `ipcol.type` (`ipv4`/`ipv6`/`cidr4`/`cidr6`), `ipcol.family`, `ipcol.is_cidr`
|
||||
- `ipcol.mask`, `ipcol.netmask`, `ipcol.identity`, `ipcol.broadcast`, `ipcol.range`, `ipcol.hostcount`, `ipcol.address_count`
|
||||
- `ipcol.rfc_type` (classification: e.g. `global`, `private`, `documentation`, `shared`, `link-local`, ...)
|
||||
- `ipcol.ipinfo.country`
|
||||
- `ipcol.ipinfo.data.<any_json_field>`
|
||||
- `ipcol.asn.asn`, `ipcol.asn.name`, `ipcol.asn.country`
|
||||
- `ipcol.asn.data.<any_json_field>`
|
||||
- `ipcol.vt.verdict` (e.g. `"3/94"`), `ipcol.vt.malicious`, `ipcol.vt.total`, `ipcol.vt.category` (alias: `ipcol.vt.type`)
|
||||
- `ipcol.vt.verdict` (e.g. `"3/94"`), `ipcol.vt.score`, `ipcol.vt.malicious`, `ipcol.vt.total`, `ipcol.vt.category` (alias: `ipcol.vt.type`)
|
||||
- `ipcol.vt.data.<any_json_field>`
|
||||
- `ipcol.geo.*` (best-available geo: prefers MaxMind mmdb, else free HTTP providers)
|
||||
- `ipcol.maxmind.*` (offline-only MaxMind lookup; empty if no mmdb)
|
||||
|
||||
Type shortcuts on table-like sheets:
|
||||
- `;i` -> `type-ip`
|
||||
- `;d` -> `type-domain`
|
||||
- `;u` -> `type-url-ioc`
|
||||
- `;h` -> `type-hash`
|
||||
|
||||
#### Caching
|
||||
|
||||
All lookup providers cache results in a local sqlite+pickle DB (default `~/.visidata_cache.db`).
|
||||
@@ -124,6 +134,28 @@ Pure-Python library used by `iptype.py` for:
|
||||
|
||||
This file intentionally does **not** import VisiData so it can be validated outside the VisiData runtime.
|
||||
|
||||
### VT schema (`*.vt`)
|
||||
|
||||
`ip.vt`, `domain.vt`, `url.vt`, and `hash.vt` expose a normalized shape for quick querying across free + premium responses, while still preserving full raw JSON:
|
||||
|
||||
Common fields:
|
||||
- `verdict` (`"malicious/total"`)
|
||||
- `score` / `confidence` (`malicious/total` float)
|
||||
- `malicious`, `suspicious`, `harmless`, `undetected`, `timeout`, `total`
|
||||
- `category` / `categories`
|
||||
- `reputation`, `votes_harmless`, `votes_malicious`
|
||||
- `tags`, `last_analysis_date`, `last_modification_date`
|
||||
- `results` (normalized engine results map), `stats`, `data` (full raw API response)
|
||||
|
||||
Object-specific conveniences:
|
||||
- `ip.vt`: `asn`, `as_owner`, `country`, `continent`, `network`
|
||||
- `domain.vt`: `ip` (best/last known), `ips` (all extracted A/AAAA)
|
||||
- `url.vt`: URL-level verdict/score plus direct raw access via `url.vt.attrs.*`
|
||||
- `hash.vt`: `name` (best malware name), `names` (all extracted names), plus verdict/score
|
||||
|
||||
Raw passthrough:
|
||||
- Any VT `attributes` field is also available via `obj.vt.<attribute_name>` and `obj.vt.attrs.<attribute_name>`.
|
||||
|
||||
## Config: `visidatarc`
|
||||
|
||||
This repo’s `visidatarc` is intended to be installed as VisiData’s `config.py`:
|
||||
|
||||
Reference in New Issue
Block a user