# VisiData Config + Plugins This folder contains a VisiData `config.py` (symlinked from `visidatarc`) plus a small set of local plugins under `plugins/`. ## Install The installer links (or copies) the config and plugins into VisiData’s per-user directory. ```bash ./install.sh --link # default, symlinks into place ./install.sh --copy # copies into place ./install.sh --deps # installs optional Python deps into $VD_DIR/plugins-deps ``` On VisiData 3.3, `$VD_DIR` defaults to: - macOS: `~/Library/Preferences/visidata` - Linux: `${XDG_CONFIG_HOME:-~/.config}/visidata` ## Plugins Plugins are installed into `$VD_DIR/plugins/` and imported via the top-level `plugins` package. ## Showcase Demo This repo includes a self-contained sample dataset + command log to demonstrate the local IOC/IP features: - `showcase_ioc.tsv` (sample IOC rows) - `showcase_ioc.vdj` (replay file) Run it interactively from this repo root: ```bash vd --visidata-dir "$PWD" --config "$PWD/visidatarc" --play showcase_ioc.vdj ``` What it showcases: - custom types: `IP`, `Domain`, `URL`, `Hash` - IP membership expressions: `src_ip * network` - URL parsing fields: `url.host`, `url.parts.path`, `url.domain` - hash classification: `file_hash.kind` - IP lookups: `src_ip.ipinfo.*`, `src_ip.asn.*`, `src_ip.geo.*`, `src_ip.country()` - provider visibility: `src_ip.geo.source`, `src_ip.asn.source`, `domain.dns.source` - domain/network intel: `domain.dns.*`, `domain.rdap.*` - hash intel: `file_hash.mb.*` (MalwareBazaar) - VirusTotal lookups: `src_ip.vt.*`, `file_hash.vt.*`, `domain.vt.*`, `url.vt.*` - local plugin command: `tke-hidecol` Lookup notes: - VT columns require `options.tke_vt_api_key` (or `VT_API_KEY` / `VIRUSTOTAL_API_KEY` / `~/.virustotal_api_key`). - IPInfo/ASN/Geo columns use free providers and may be rate-limited; `options.tke_ipinfo_token` improves reliability. - To keep replays practical with strict throttling, some heavy lookup columns are intentionally limited to a subset of rows. ### `plugins/hidecol.py` Adds a command to hide columns that are empty or constant across all rows. - Command: `tke-hidecol` - Menu: `Column -> Hide -> empty/superfluous columns` ### `plugins/iptype.py` Adds a custom IP datatype that supports: - IPv4 + IPv6 addresses - CIDR networks (e.g. `192.168.7.0/24`) - Correct sorting (numeric, by version) - Membership test operator: `ip * net` (and `net * ip`) - Normalized lookup/enrichment properties, accessible as attributes in expressions #### Type + Command - Type converter: `ip(...)` - Type name: `IP` - Command: `type-ip` (sets `cursorCol.type=ip`) #### Operations Membership test: - `ipcol * "192.168.7.0/24"` -> `True`/`False` - `"192.168.7.0/24" * ipcol` -> `True`/`False` #### Attributes (on `IP` typed cells) Lookup objects expose both normalized fields and raw response data: - `ipcol.ipinfo.country` - `ipcol.ipinfo.data.` - `ipcol.asn.asn`, `ipcol.asn.name`, `ipcol.asn.country` - `ipcol.asn.data.` - `ipcol.vt.verdict` (e.g. `"3/94"`), `ipcol.vt.malicious`, `ipcol.vt.total`, `ipcol.vt.category` (alias: `ipcol.vt.type`) - `ipcol.vt.data.` - `ipcol.geo.*` (best-available geo: prefers MaxMind mmdb, else free HTTP providers) - `ipcol.maxmind.*` (offline-only MaxMind lookup; empty if no mmdb) #### Caching All lookup providers cache results in a local sqlite+pickle DB (default `~/.visidata_cache.db`). #### Lookup Providers + Keys Options (set in `config.py` / `visidatarc`): - `options.tke_cache_db_path="~/.visidata_cache.db"` - `options.tke_lookup_cache_ttl=86400` - `options.tke_lookup_error_ttl=300` - `options.tke_lookup_timeout=10` - `options.tke_ipinfo_token="..."` (optional; ipinfo can work without it) - `options.tke_ipapi_key="..."` (optional) - `options.tke_vt_api_key="..."` (required for VT lookups unless using `~/.virustotal_api_key`) - `options.tke_maxmind_mmdb_path="/path/to/GeoLite2-City.mmdb"` (optional) Env var equivalents: - `IPINFO_TOKEN`, `IPAPI_KEY` - `VT_API_KEY` or `VIRUSTOTAL_API_KEY` (also supports `~/.virustotal_api_key`) - `MAXMIND_MMDB_PATH` or `GEOIP_MMDB_PATH` MaxMind (offline “free” GeoLite2) support: - Place a `GeoLite2-City.mmdb` / `GeoLite2-Country.mmdb` file in `$VD_DIR/`, or set `options.tke_maxmind_mmdb_path`. ### `plugins/iplib.py` Pure-Python library used by `iptype.py` for: - Normalized info classes (`IPInfo`, `ASNInfo`, `VTInfo`, `GeoInfo`) - `JSONNode` wrapper (`.data.`) for safe attribute-style access into raw dict/list JSON - Parsing/normalization helpers for each provider’s response shape This file intentionally does **not** import VisiData so it can be validated outside the VisiData runtime. ## Config: `visidatarc` This repo’s `visidatarc` is intended to be installed as VisiData’s `config.py`: - `$VD_DIR/config.py` (VisiData 3.3 default) - and also `~/.visidatarc` as a legacy fallback It currently contains: - display/date format options - a sqlite+pickle caching decorator and a set of general-purpose helpers (aggregators, timestamp parsing, “dirty” JSON parsing, etc) ### Are the `visidatarc` functions superseded? Partially: - The **IP-centric lookups and normalized attribute access** are now primarily handled by `plugins/iptype.py` on typed values (e.g. `ipcol.geo.country_code`). - Many other helpers in `visidatarc` (aggregators like `avgdiff`, parsing/time conversion helpers, etc.) are still independent and useful. ### Keeping old + new side-by-side (without duplicating code) Yes. The cleanest pattern in VisiData is: 1. Put shared logic into a module under `plugins/` (so it’s on `sys.path` via `$VD_DIR`). 2. In `visidatarc`, import and expose thin wrappers (or just import the module and use `module.func(...)` in expressions). Concretely: - `plugins/iplib.py` already holds parsing/normalization shared by the IP type. - If you have legacy functions in `visidatarc` that overlap with the new IP lookups, refactor those functions into a shared module (e.g. `plugins/lookups.py`) and have both `visidatarc` and `plugins/iptype.py` call into it. This keeps backward-compatible names available while ensuring caching/auth/provider behavior is implemented in one place.