287 Commits

Author SHA1 Message Date
tobias f540e08437 fix(shell): repair broken aliases, modernize, make bash/zsh + linux/mac portable
- drop hardcoded /home/tke/.http_proxy sourcing; both proxy installers
  (update_bashrc_proxy.sh #PROXY_A93JK2, install_proxy.sh
  #PROXY_BRIDGE_CONFIG) inject their own bashrc lines
- fix 'set $_Z_OWNER=' (set positional params instead of the variable)
- prefer zoxide over z.sh, load only one jumper instead of both
- dockertags: dead Docker Hub v1 API -> v2, handle library/ prefix
- rot13: handle uppercase
- ppxml: drop dead duplicate with typo'd xmllin command
- exa (unmaintained) -> eza, fallback lsd -> ls; single ll definition
- guard linux-only bits (locate -A, ntfsmount, usbguard, apt+fzf, mc)
  and optional tools with command -v; batcat support for ubuntu
- unsetproxy: grep -P is GNU-only, unset vars explicitly instead
- quote ${workdir:-$(pwd)} mounts; dedupe mv/cp/pip/python
- guard shopt autocd (macOS bash 3.2)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 16:56:39 +02:00
tobias a88cc3146c docs: add verified build instructions; fix bincmp ssdeep API
- README "Building From Source" section covering all Go and Rust tools;
  every command tested (rustc, go build, goipgrep make build/test)
- bincmp: ssdeep.HashFromFile does not exist in pinned ssdeep v0.4.0,
  switch to FuzzyFilename; fix two go vet warnings
- ignore in-place build outputs for bincmp/gopname/goinfo

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 14:34:11 +02:00
tobias 1cbf8afb4a chore: cleanup — untrack binaries, consolidate Go dirs, dedupe tools
- Untrack and delete compiled binaries (tarsum, gosoft.exe, rust uniq/uniq2);
  ignore build outputs (dist/, bin/, *.exe, *.test, .ruff_cache/)
- Merge tools/go/ and projects/go-tools/go/ into projects/go-tools/<name>/
- Fix goipgrep .gitignore: bare 'ipgrep' pattern was ignoring cmd/ipgrep/,
  so the main entrypoint was never tracked; now anchored to /ipgrep
- Archive duplicate implementations to archive/experimental/{rust,go}/
  (uniq, between, tarsum rewrites); canonical versions stay in tools/
- Update README tool catalog to match new layout

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 13:42:45 +02:00
tobias 401b3e1781 feat(proxy): add destination overrides and fix PAC helpers
Add a hot-reloaded override table (overrides.json) that forces specific
hosts/IPs to a chosen profile regardless of the active profile, applied
across all bridge ports. Matching supports exact host, wildcard, domain
suffix, single IP, and CIDR. Managed via `proxy-bridge override
list|add|remove|clear` and surfaced in status/config.

Also fix two PAC sandbox helpers: localHostOrDomainIs (unqualified-host
case) and isInNet (was a no-op stub, now does real IPv4 subnet matching).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:58:11 +02:00
tke 7f73746427 Update proxy bridge setup 2026-05-22 13:01:54 +02:00
tke 203f2bf189 feat(proxy): secure refactor and system-wide integration
- Removed hardcoded corporate proxy URL from all scripts.
- Updated bridge.js to load configuration from /opt/proxy-bridge/config.json.
- Updated setup.js to interactively configure upstream proxy and credentials.
- Enhanced install_proxy.sh to automatically configure APT, Bash, and system services.
- Purged sensitive URL from git history and verified zero leakage.
2026-05-22 12:23:16 +02:00
tke ae5d503268 feat: expand tool catalog and improve 'what' search recall
- Add 32 new tool and dockerfile entries to README.md catalog.
- Increase 'what' shortlist limit to 100 for better search recall.
- Update 'what' default model to gemma4 and improve robust JSON parsing.
2026-05-18 13:21:23 +02:00
tke ac3245b78f proxy-bridge: add keyring-based HTTP CONNECT proxy bridge 2026-04-27 14:27:27 +02:00
tke 71bffc77ae proxy: add ollama to restart services 2026-04-27 14:25:36 +02:00
tke 177783908d feat: Add standalone Docker image pull and save tool
Add dockerpullsave.py - a Python utility that downloads Docker images
directly from registries and saves them as tar archives without requiring
Docker daemon to be running.

Features:
- Supports multiple registries (Docker Hub, GHCR, ECR, private registries)
- Handles authentication via bearer tokens and registry discovery
- Multi-architecture manifest list support (defaults to linux/amd64)
- Robust digest-based image selection for consistent pulls
- Progress bar for large layer downloads with Docker-style formatting
- Proper Docker image structure generation for 'docker load' compatibility

Usage examples:
  ./dockerpullsave.py ubuntu:latest
  ./dockerpullsave.py ghcr.io/oras-project/oras:v1.1.0
  ./dockerpullsave.py alpine@sha256:f271e74b17ced...

Output: Creates {image}_{tag}.tar file ready for 'docker load -i'

This tool is particularly useful for:
- Air-gapped environments where Docker daemon cannot access registries
- Batch image downloading and offline distribution
- Registry migration and backup scenarios
- Security scanning workflows requiring image inspection
2026-04-02 12:49:47 +02:00
tke c6a849a1fa Add SQLite table to JSONL export tool 2026-03-08 12:38:08 +01:00
tobias d344c84473 scripts: add opencode model listing sync helper
Queries lmstudio and ollama /v1/models endpoints to keep opencode.json
model entries up to date, preserving manual overrides.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 22:50:13 +01:00
tobias 47e72e40b3 visidata: enhance IOC plugins with improved lookups and validation
Expand iplib, iptype, and ioc plugins with better caching, throttling,
and lookup logic. Update validation script and showcase journal accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 22:49:49 +01:00
tke 2a811fb331 Rewrite what around README catalog and Ollama
Remove the JSON tool database and move tool metadata into a compact README catalog.
Make what README-driven and Ollama-only, with shortlist generation and JSON-repair retry handling.
Pull qwen3.5:2b and ministral-3:3b, compare them on fixed repository queries, and set ministral-3:3b as the default model.
Tighten README wording so similar tools like domgrep/geturls and sparsecmp/scatterhash rank correctly.
2026-03-07 20:39:24 +01:00
tke 017079848d Restructure repo layout and document conventions
Move legacy systemscripts into scripts/display and scripts/setup.
Rehome stray top-level tools into their domain folders.
Archive narrow experiments and outdated codegrab leftovers.
Remove empty legacy directories and stale root files.
Expand macOS metadata ignores and update the README with the refined repository structure.
2026-03-07 18:54:32 +01:00
tke 67fbbf5a36 Add eslogger completion and Go tools 2026-03-07 10:32:56 +01:00
tobias 0ff19b5765 visidata: ignore local input history file
Prevent accidental commits of interactive VisiData input history by ignoring input_history.jsonl in the local config directory.
2026-02-21 23:20:42 +01:00
tobias 8d031b80eb tools: add registry-aware docker pull helper
Add a standalone script that fetches image manifests/layers from common registries and assembles a docker-loadable tarball for offline transfer workflows.
2026-02-21 23:20:42 +01:00
tobias 62a405a0a7 visidata: add replayable IOC showcase and usage docs
Provide a sample dataset and cmdlog that exercise typed IOC enrichment while keeping heavy lookups scoped for practical throttled runs, and document how to run it.
2026-02-21 23:20:42 +01:00
tobias 2723964a93 visidata: add IOC types with cached, throttled lookups
Centralize provider caching and rate-limit handling, then add Domain/URL/Hash IOC types and safer VT/IPInfo key resolution so lookups stay reliable on free-tier APIs.
2026-02-21 23:20:42 +01:00
tobias d63af75618 goipgrep: refactor into module; pure-Go ping/resolve; cache+CI; drop binary 2026-02-21 23:20:42 +01:00
tobias c8131f85b2 visidata: document config and plugins 2026-02-21 23:20:42 +01:00
tobias a797b48593 visidata: improve IP lookups (cached, keys, maxmind) 2026-02-21 23:20:42 +01:00
tobias 8cbdc9f41f visidata: add IP type with CIDR + lookups 2026-02-21 23:20:42 +01:00
tobias c45c729ad4 visidata: make installer idempotent and use v3.3 VD_DIR 2026-02-21 23:20:42 +01:00
tobias bc837a4f62 visidata: add installer and optional deps list 2026-02-21 23:20:42 +01:00
tobias bf7cf21a77 visidatarc: v3.3-safe config + helper fixes 2026-02-21 23:20:42 +01:00
tobias da872a0de1 visidata: fix hidecol and add menu entry 2026-02-21 23:20:42 +01:00
tobias 7f1b6bb788 Remove .DS_Store and ignore it 2026-02-21 23:20:42 +01:00
tobias 92c7fc69c6 Restructure repository: organize tools by purpose, create what search tool
- Move single-file tools to tools/ organized by category (security, forensics, data, etc.)
- Move multi-file projects to projects/ (go-tools, puzzlebox, timesketch, rust-tools)
- Move system scripts to scripts/ (proxy, display, setup, windows)
- Organize config files in config/ (shell, visidata, applications)
- Move experimental tools to archive/experimental
- Create 'what' fuzzy search tool with progressive enhancement (ollama->fzf->grep)
- Add initial metadata database for intelligent tool discovery
- Preserve git history using 'git mv' commands
2026-02-21 23:20:42 +01:00
Tobias Kessels 8d7b3c9270 Moved and renamed screensettings scripts 2025-10-01 12:25:09 +02:00
Tobias Kessels e56601ce23 Add cool chars to README, lsd alias, and Ubuntu telemetry disable script 2025-10-01 09:52:58 +02:00
tobias 2290adf3b9 update shell aliases 2025-08-24 16:20:10 +02:00
tobias 1cf631d73d Fix alias for fdfind 2025-08-24 16:20:10 +02:00
tabledevil a215ff6929 Add goinfo and gosoft 2025-03-25 08:16:04 +01:00
Tobias Kessels b32be5e2c2 port ipgrep to golang 2025-02-07 14:06:39 +01:00
Tobias Kessels 568bcead28 Add usage info to gobetween 2024-08-01 16:22:47 +02:00
Tobias Kessels b482023e37 Add gobetbeen.go as a golang replacement for between 2024-08-01 16:13:12 +02:00
Tobias Kessels 5d1aa21129 add csv2json.go to converta csv with headers to jsonlines 2024-08-01 14:23:32 +02:00
Tobias Kessels a625f8184c Replace uniqrs with gouniq
replaced binary unqirs with a plattform independent golang rewrite.
gouniq works removes duplicate lines from an unsorted text file thus maintaining the original order of lines
2024-08-01 10:51:37 +02:00
Tobias Kessels 020ae8cc6a Add split_linewise.py
split_linewise.py splits large textfiles into smaller chunks, like 500MB for Uploading into splunk while also making sure each line stays intakt
2024-08-01 09:44:57 +02:00
tke d34af356b1 Add dissect and sort alias 2024-05-24 08:20:20 +02:00
tke bd087aa539 Auth only when needed 2024-05-21 16:13:07 +02:00
tke 0eb88dc52e Retrieve valid tags beforehand 2024-05-21 11:47:06 +02:00
tke 2c860375d4 added binary Search for sep signature hit 2024-05-15 12:02:30 +02:00
tke 7270d00f19 Merge remote-tracking branch 'refs/remotes/origin/master' 2024-05-08 16:31:17 +02:00
tke 34614c5aed Fixed issue with tke-hide-col plugin 2024-05-08 16:28:55 +02:00
tke 66690116c6 improve errorhandling in get_ntp 2024-05-08 16:28:55 +02:00
tke 1fb18d583a Update libarchive error handling; improve error message formatting 2024-05-08 16:28:55 +02:00
tke 74ed496b20 Added Libarchive-Version tarsums 2024-05-08 16:28:55 +02:00