300 Commits
Author SHA1 Message Date
tke 9a33508b1d feat(security): add giac_lookup tool and update README catalog 2026-09-25 15:31:29 +02:00
tke 25780cd6e8 fix(proxy): unit file carries DISPLAY + DBUS_SESSION_BUS_ADDRESS
Lockout notifications need a session path; systemd user services start with an
empty Environment, which made notify-send silently unreachable. Installer now
writes the current DISPLAY and the user-bus address (XDG runtime dir) into the
unit; deploy + daemon-reload + restart applied live and verified (7 listeners,
routed profiles answer 200).
2026-09-24 13:12:18 +02:00
tke be0e2d84cc feat(proxy): desktop lockout notification + 'proxy-bridge reauth' command
Notification path (bridge.js): entering auth lockout fires ONE desktop
notification per episode (notify-send, critical urgency), only when a session
is detectable (DBUS_SESSION_BUS_ADDRESS or DISPLAY) — headless stays silent.
Message explains the 407 cause and points at 'proxy-bridge reauth'.

Recovery path (proxyctl.js): 'proxy-bridge reauth' prompts masked, verifies the
new password with a SINGLE upstream CONNECT (407 -> NOT stored), stores via
secret-tool on success, then writes a reauth-stamp.json the bridge watches
(1s poll): clears in-memory authState and reloads keyring creds — no restart.

Shared one-shot verifier extracted to verify-creds.js (install.js + proxyctl.js
use the same code; also fixes proxyctl 'setup' shelling to the retired
/opt/setup.js — 'reauth' is its replacement for the password-expiry flow).

Tests: +3 bridge (notify once-per-episode incl. remediation hint; headless
silent; stamp clears lockout without restart) +2 proxyctl (verified stored +
stamped; rejected never stored/stamped). Suite: 28/28.
2026-09-24 13:10:42 +02:00
tke e49b4af14a chore(proxy): archive former /opt/proxy-bridge git history as bundle - /opt is now deploy-only (untracked); provenance lives in .installed-from. Bundle preserves 4 historical commits for forensics. 2026-09-24 08:29:53 +02:00
tke a23fa00dfb feat(proxy): apt proxy audit + doctor mode; reinstall-from-clone = update
apt-audit.js scans EVERY file in apt.conf.d for Acquire::*::Proxy, classifying:
- creds: embedded userinfo (plaintext leak + stale-cred hammering risk)
- overrides: sorts after 80proxy-bridge so apt last-wins REPLACES the bridge URL
- external: points at a non-bridge proxy
Installer Legacy section alerts + fixes (backup kept: *.disabled-by-proxy-bridge).
New 'install.js doctor [--fix]': audits apt, bashrc, snap, service drop-ins,
service state, keyring presence; exit 1 when issues found.

Update semantics: deploy writes .installed-from (git commit + file hashes);
runtime JSONs are never created if present and never overwritten; tests prove
3 consecutive installs leave runtime state byte-identical. Downgrade protection
comes from the visible marker + diff reporting, never silent clobbering.

Tests: 6 audit + 2 installer (update-semantics, doctor) — suite now 23/23.
2026-09-23 16:07:53 +02:00
tke a626392eb1 Add sensible exlcudes for restic backup 2026-09-23 15:50:52 +02:00
tke 753fac9fe9 fix(proxy): installer — no prompts during dry-run; never pin service to agent-bundled node
- askConfirm short-circuits non-interactively in --dry-run (prints what would be asked)
- systemd node resolution order: existing unit's pinned binary > newest nvm node >
  PATH lookup excluding hermes-agent runtimes > process.execPath. On the live box
  this keeps the unit byte-identical (true idempotency for the service section).
2026-09-23 15:15:33 +02:00
tke 3f9be30784 feat(proxy): idempotent interactive installer; retire credential-embedding legacy scripts
install.js (node, replaces install_proxy.sh + setup.js + update_*.sh + get_proxy.sh):
- sections: preflight, deploy, config import/create, credentials, legacy migration,
  systemd unit, system-wide proxy (bashrc/apt/docker/snapd/ollama), verify
- idempotent: re-run converges; tests prove a second non-interactive run shows
  zero CHANGED
- imports existing /opt configs non-destructively; asks before changing
- credentials: prompted when keyring entry missing; verified with ONE CONNECT
  round-trip before storing; 407 responses are never stored (anti-lockout);
  unreachable upstream stores nothing
- legacy migration: removes bashrc #PROXY_A93JK2 line + plaintext-cred files,
  rewrites cred-embedding apt/snap/service drop-ins to the bridge URL
- flags: --dry-run, --non-interactive (fails listing missing info), --no-systemd

Tests cover dry-run-no-op, idempotency, verify-store-on-200, refuse-on-407.
2026-09-23 15:14:19 +02:00
tke c514f32661 feat(proxy): credential-failure backoff prevents corp proxy account lockout
After 3 consecutive 407s within 120s (defaults; env-tunable), the bridge stops
sending Proxy-Authorization to that upstream for 300s, then retries once with
refreshed keyring creds. A successful request clears the failure state. Requests
during the lockout fail fast with the upstream's 407 instead of retrying stale
credentials — the exact pattern that locks accounts.

Tests: 407-stub lockout engages on 4th attempt; success mid-stream resets the
counter (re-lock needs a fresh 3); missing creds never send an auth header.
Harness: awaitProxyResponse() decouples client-side settle from server FIN
timing (previous once(end) pattern wedged under node --test).
2026-09-23 14:57:18 +02:00
tke 0b8c5001fc fix(proxy): port HTTPS-upstream TLS-wrap from deployed bridge; unify CONNECT establishment
- parseProxyConfig handles PAC 'HTTPS host:port' (tls.connect + CONNECT inside TLS) in
  both the CONNECT and plain-HTTP handlers — ports live fix d8b9db2 into the repo
- CONNECT: split tunnel establishment into establishDirect / sendConnectToProxy,
  removing the duplicate DIRECT 'connect' listener that raced onUpstreamData
- env-overridable PROFILE_FILE / REMOTES_FILE / PAC_DIR (test hermeticity)
- tests: +DIRECT echo exactly-once, +HTTPS upstream via local PAC dir + self-signed
  TLS stub (no internet needed); 8/8 pass
2026-09-23 14:49:07 +02:00
tke f434d58ef9 Update Firefox extension sync modes 2026-08-21 19:07:59 +02:00
tke 348a275480 feat(setup): add vllm support and default model limits to update-models.sh 2026-08-17 09:37:23 +02:00
tke 52e0c00cd8 fix(proxy): handle blocked CONNECT resets 2026-07-01 09:37:41 +02:00
tobiasandClaude Fable 5 f540e08437 fix(shell): repair broken aliases, modernize, make bash/zsh + linux/mac portable
- drop hardcoded /home/tke/.http_proxy sourcing; both proxy installers
  (update_bashrc_proxy.sh #PROXY_A93JK2, install_proxy.sh
  #PROXY_BRIDGE_CONFIG) inject their own bashrc lines
- fix 'set $_Z_OWNER=' (set positional params instead of the variable)
- prefer zoxide over z.sh, load only one jumper instead of both
- dockertags: dead Docker Hub v1 API -> v2, handle library/ prefix
- rot13: handle uppercase
- ppxml: drop dead duplicate with typo'd xmllin command
- exa (unmaintained) -> eza, fallback lsd -> ls; single ll definition
- guard linux-only bits (locate -A, ntfsmount, usbguard, apt+fzf, mc)
  and optional tools with command -v; batcat support for ubuntu
- unsetproxy: grep -P is GNU-only, unset vars explicitly instead
- quote ${workdir:-$(pwd)} mounts; dedupe mv/cp/pip/python
- guard shopt autocd (macOS bash 3.2)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 16:56:39 +02:00
tobiasandClaude Fable 5 a88cc3146c docs: add verified build instructions; fix bincmp ssdeep API
- README "Building From Source" section covering all Go and Rust tools;
  every command tested (rustc, go build, goipgrep make build/test)
- bincmp: ssdeep.HashFromFile does not exist in pinned ssdeep v0.4.0,
  switch to FuzzyFilename; fix two go vet warnings
- ignore in-place build outputs for bincmp/gopname/goinfo

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 14:34:11 +02:00
tobiasandClaude Fable 5 1cbf8afb4a chore: cleanup — untrack binaries, consolidate Go dirs, dedupe tools
- Untrack and delete compiled binaries (tarsum, gosoft.exe, rust uniq/uniq2);
  ignore build outputs (dist/, bin/, *.exe, *.test, .ruff_cache/)
- Merge tools/go/ and projects/go-tools/go/ into projects/go-tools/<name>/
- Fix goipgrep .gitignore: bare 'ipgrep' pattern was ignoring cmd/ipgrep/,
  so the main entrypoint was never tracked; now anchored to /ipgrep
- Archive duplicate implementations to archive/experimental/{rust,go}/
  (uniq, between, tarsum rewrites); canonical versions stay in tools/
- Update README tool catalog to match new layout

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 13:42:45 +02:00
tobiasandClaude Opus 4.7 401b3e1781 feat(proxy): add destination overrides and fix PAC helpers
Add a hot-reloaded override table (overrides.json) that forces specific
hosts/IPs to a chosen profile regardless of the active profile, applied
across all bridge ports. Matching supports exact host, wildcard, domain
suffix, single IP, and CIDR. Managed via `proxy-bridge override
list|add|remove|clear` and surfaced in status/config.

Also fix two PAC sandbox helpers: localHostOrDomainIs (unqualified-host
case) and isInNet (was a no-op stub, now does real IPv4 subnet matching).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:58:11 +02:00
tke 7f73746427 Update proxy bridge setup 2026-05-22 13:01:54 +02:00
tke 203f2bf189 feat(proxy): secure refactor and system-wide integration
- Removed hardcoded corporate proxy URL from all scripts.
- Updated bridge.js to load configuration from /opt/proxy-bridge/config.json.
- Updated setup.js to interactively configure upstream proxy and credentials.
- Enhanced install_proxy.sh to automatically configure APT, Bash, and system services.
- Purged sensitive URL from git history and verified zero leakage.
2026-05-22 12:23:16 +02:00
tke ae5d503268 feat: expand tool catalog and improve 'what' search recall
- Add 32 new tool and dockerfile entries to README.md catalog.
- Increase 'what' shortlist limit to 100 for better search recall.
- Update 'what' default model to gemma4 and improve robust JSON parsing.
2026-05-18 13:21:23 +02:00
tke ac3245b78f proxy-bridge: add keyring-based HTTP CONNECT proxy bridge 2026-04-27 14:27:27 +02:00
tke 71bffc77ae proxy: add ollama to restart services 2026-04-27 14:25:36 +02:00
tke 177783908d feat: Add standalone Docker image pull and save tool
Add dockerpullsave.py - a Python utility that downloads Docker images
directly from registries and saves them as tar archives without requiring
Docker daemon to be running.

Features:
- Supports multiple registries (Docker Hub, GHCR, ECR, private registries)
- Handles authentication via bearer tokens and registry discovery
- Multi-architecture manifest list support (defaults to linux/amd64)
- Robust digest-based image selection for consistent pulls
- Progress bar for large layer downloads with Docker-style formatting
- Proper Docker image structure generation for 'docker load' compatibility

Usage examples:
  ./dockerpullsave.py ubuntu:latest
  ./dockerpullsave.py ghcr.io/oras-project/oras:v1.1.0
  ./dockerpullsave.py alpine@sha256:f271e74b17ced...

Output: Creates {image}_{tag}.tar file ready for 'docker load -i'

This tool is particularly useful for:
- Air-gapped environments where Docker daemon cannot access registries
- Batch image downloading and offline distribution
- Registry migration and backup scenarios
- Security scanning workflows requiring image inspection
2026-04-02 12:49:47 +02:00
tke c6a849a1fa Add SQLite table to JSONL export tool 2026-03-08 12:38:08 +01:00
tobiasandClaude Opus 4.6 d344c84473 scripts: add opencode model listing sync helper
Queries lmstudio and ollama /v1/models endpoints to keep opencode.json
model entries up to date, preserving manual overrides.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 22:50:13 +01:00
tobiasandClaude Opus 4.6 47e72e40b3 visidata: enhance IOC plugins with improved lookups and validation
Expand iplib, iptype, and ioc plugins with better caching, throttling,
and lookup logic. Update validation script and showcase journal accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 22:49:49 +01:00
tke 2a811fb331 Rewrite what around README catalog and Ollama
Remove the JSON tool database and move tool metadata into a compact README catalog.
Make what README-driven and Ollama-only, with shortlist generation and JSON-repair retry handling.
Pull qwen3.5:2b and ministral-3:3b, compare them on fixed repository queries, and set ministral-3:3b as the default model.
Tighten README wording so similar tools like domgrep/geturls and sparsecmp/scatterhash rank correctly.
2026-03-07 20:39:24 +01:00
tke 017079848d Restructure repo layout and document conventions
Move legacy systemscripts into scripts/display and scripts/setup.
Rehome stray top-level tools into their domain folders.
Archive narrow experiments and outdated codegrab leftovers.
Remove empty legacy directories and stale root files.
Expand macOS metadata ignores and update the README with the refined repository structure.
2026-03-07 18:54:32 +01:00
tke 67fbbf5a36 Add eslogger completion and Go tools 2026-03-07 10:32:56 +01:00
tobias 0ff19b5765 visidata: ignore local input history file
Prevent accidental commits of interactive VisiData input history by ignoring input_history.jsonl in the local config directory.
2026-02-21 23:20:42 +01:00
tobias 8d031b80eb tools: add registry-aware docker pull helper
Add a standalone script that fetches image manifests/layers from common registries and assembles a docker-loadable tarball for offline transfer workflows.
2026-02-21 23:20:42 +01:00
tobias 62a405a0a7 visidata: add replayable IOC showcase and usage docs
Provide a sample dataset and cmdlog that exercise typed IOC enrichment while keeping heavy lookups scoped for practical throttled runs, and document how to run it.
2026-02-21 23:20:42 +01:00
tobias 2723964a93 visidata: add IOC types with cached, throttled lookups
Centralize provider caching and rate-limit handling, then add Domain/URL/Hash IOC types and safer VT/IPInfo key resolution so lookups stay reliable on free-tier APIs.
2026-02-21 23:20:42 +01:00
tobias d63af75618 goipgrep: refactor into module; pure-Go ping/resolve; cache+CI; drop binary 2026-02-21 23:20:42 +01:00
tobias c8131f85b2 visidata: document config and plugins 2026-02-21 23:20:42 +01:00
tobias a797b48593 visidata: improve IP lookups (cached, keys, maxmind) 2026-02-21 23:20:42 +01:00
tobias 8cbdc9f41f visidata: add IP type with CIDR + lookups 2026-02-21 23:20:42 +01:00
tobias c45c729ad4 visidata: make installer idempotent and use v3.3 VD_DIR 2026-02-21 23:20:42 +01:00
tobias bc837a4f62 visidata: add installer and optional deps list 2026-02-21 23:20:42 +01:00
tobias bf7cf21a77 visidatarc: v3.3-safe config + helper fixes 2026-02-21 23:20:42 +01:00
tobias da872a0de1 visidata: fix hidecol and add menu entry 2026-02-21 23:20:42 +01:00
tobias 7f1b6bb788 Remove .DS_Store and ignore it 2026-02-21 23:20:42 +01:00
tobias 92c7fc69c6 Restructure repository: organize tools by purpose, create what search tool
- Move single-file tools to tools/ organized by category (security, forensics, data, etc.)
- Move multi-file projects to projects/ (go-tools, puzzlebox, timesketch, rust-tools)
- Move system scripts to scripts/ (proxy, display, setup, windows)
- Organize config files in config/ (shell, visidata, applications)
- Move experimental tools to archive/experimental
- Create 'what' fuzzy search tool with progressive enhancement (ollama->fzf->grep)
- Add initial metadata database for intelligent tool discovery
- Preserve git history using 'git mv' commands
2026-02-21 23:20:42 +01:00
Tobias Kessels 8d7b3c9270 Moved and renamed screensettings scripts 2025-10-01 12:25:09 +02:00
Tobias Kessels e56601ce23 Add cool chars to README, lsd alias, and Ubuntu telemetry disable script 2025-10-01 09:52:58 +02:00
tobias 2290adf3b9 update shell aliases 2025-08-24 16:20:10 +02:00
tobias 1cf631d73d Fix alias for fdfind 2025-08-24 16:20:10 +02:00
tabledevil a215ff6929 Add goinfo and gosoft 2025-03-25 08:16:04 +01:00
Tobias Kessels b32be5e2c2 port ipgrep to golang 2025-02-07 14:06:39 +01:00
Tobias Kessels 568bcead28 Add usage info to gobetween 2024-08-01 16:22:47 +02:00