27 KiB
gists
This repository is a personal toolbox: small utilities, shell config, Docker build recipes, forensic helpers, network scripts, and a few larger side projects. It is not a single application, so the useful documentation is a map of what lives where and which files matter.
How To Read This Repo
tools/is the main home for standalone utilities grouped by purpose.projects/holds larger multi-file projects that deserve their own subtree.scripts/holds machine-management helpers.config/holds shell, desktop, and VisiData configuration.dockerfiles/holds container recipes and small Compose setups.archive/holds older or experimental material that is kept for reference, not as the primary entrypoint.
This README explains the parts that need context. Boilerplate and standard metadata files such as go.mod, go.sum, .gitignore, GitHub Actions files, and obvious launcher/config assets are not documented line-by-line unless they carry important behavior.
Refined Repository Structure
The old restructure_git.sh captured the right direction but in the wrong form. The useful part is the structure itself, not a one-shot move script. The intended layout for this repo is:
- root: only discovery files and broad entrypoints such as
README.md,.gitignore,what, and small repo-wide metadata. tools/: single-purpose utilities that can usually be understood and run on their own.projects/: anything multi-file, stateful, tested, or large enough to deserve its own local README or build flow.scripts/: machine-setup and environment-management scripts, grouped by operational area such as proxy, display, setup, or platform.config/: reusable configuration, completions, themes, desktop entries, and tool-specific setup.dockerfiles/: container recipes and Compose stacks.archive/: historical material kept for reference, not the default place to start.
The practical cleanup rules are:
- keep the root shallow; do not leave new standalone utilities at the top level if they fit under
tools/,scripts/, orconfig/ - move one-file tools by function, not by language
- move multi-file codebases by project, even if they are small
- keep generated artifacts, captured output, and binaries out of active source directories unless they are intentionally distributed
- use
archive/experimental/for things worth keeping but not worth advertising - keep macOS Finder metadata out of the repo;
.DS_Store, AppleDouble files, and__MACOSX/bundles are ignored
Applied to the current tree, the remaining rough edges are:
- a few tracked binaries still sit beside source and may eventually deserve a dedicated
bin/ordist/convention - some material under
archive/experimental/may still be worth deleting rather than merely archiving
Top-Level Files
what: README-driven repository search helper. It uses one local Ollama model and searches only the catalog below.README.md: this guide..gitignore: standard repository ignore rules.
Tool Catalog
Format: path | goal | usage. This section is intentionally compact so what can pass it to a small local model without dragging the whole repository into context.
Active Tools
what| goal: search this repository's tool catalog with Ollama only | usage:./what "query"or./what -l
Security
tools/security/scan_vt.py| goal: check file hashes against VirusTotal | usage:python3 tools/security/scan_vt.py sample.bintools/security/imphash.py| goal: calculate PE import hashes for malware triage | usage:python3 tools/security/imphash.py file.exetools/security/scapy_arp.py| goal: scan a local network with ARP requests | usage:python3 tools/security/scapy_arp.pytools/security/simple_portscan.py| goal: do a lightweight TCP port scan | usage:python3 tools/security/simple_portscan.py hosttools/security/smtpbanner.py| goal: grab SMTP banners from remote servers | usage:python3 tools/security/smtpbanner.py hosttools/security/testpw.py| goal: test password candidates against a target workflow | usage:python3 tools/security/testpw.py ...tools/security/vt_download.py| goal: download malware samples or data from VirusTotal-related workflows | usage:python3 tools/security/vt_download.py ...tools/security/vt_ip.py| goal: enrich IP addresses with VirusTotal intel | usage:python3 tools/security/vt_ip.py 8.8.8.8tools/security/vt_pdns.py| goal: query passive DNS style data from VirusTotal workflows | usage:python3 tools/security/vt_pdns.py domain.tldtools/security/certwipe| goal: wipe disks with secure-erase oriented steps | usage:tools/security/certwipe /dev/sdX
Forensics
tools/forensics/chechsqlite.py| goal: inspect SQLite databases for password or hash style columns | usage:python3 tools/forensics/chechsqlite.py sample.dbtools/forensics/extractfolder.py| goal: bulk-extract or sort files from a folder workflow | usage:python3 tools/forensics/extractfolder.py input_dirtools/forensics/sqlite32jsonl.py| goal: export each user table from a SQLite database to one JSONL file with robust filename handling | usage:python3 tools/forensics/sqlite32jsonl.py sample.db -o outdirtools/forensics/process_leak.py| goal: inspect process-leak style artifacts | usage:python3 tools/forensics/process_leak.py artifacttools/forensics/mailunpack| goal: extract mail attachments inside a constrained container workflow | usage:tools/forensics/mailunpack message.emltools/forensics/showgm.sh| goal: open image GPS EXIF coordinates in Google Maps | usage:tools/forensics/showgm.sh image.jpgtools/forensics/showosm.sh| goal: open image GPS EXIF coordinates in OpenStreetMap | usage:tools/forensics/showosm.sh image.jpg
Data And Text
tools/data/domgrep.py| goal: extract domain names specifically from URLs or mixed text input | usage:cat urls.txt | python3 tools/data/domgrep.pytools/data/geturls.py| goal: extract full raw URLs from text when you want links rather than domains | usage:python3 tools/data/geturls.py file.txttools/data/unum.py| goal: inspect Unicode code points and names | usage:echo "text" | python3 tools/data/unum.pytools/data/quickchardet.py| goal: guess file encoding quickly | usage:python3 tools/data/quickchardet.py file.txttools/data/json_save.py| goal: normalize or save JSON fragments from text streams | usage:python3 tools/data/json_save.py ...tools/data/kv_parse.py| goal: parse key-value formatted text | usage:python3 tools/data/kv_parse.py input.txttools/data/vba_chr_decode.py| goal: decode VBAChr(...)obfuscation patterns | usage:python3 tools/data/vba_chr_decode.py macro.txttools/data/concat.py| goal: concatenate structured text inputs in a repeatable way | usage:python3 tools/data/concat.py file1 file2tools/data/split_linewise.py| goal: split text into line-based chunks | usage:python3 tools/data/split_linewise.py input.txttools/data/uniq.py| goal: remove duplicate lines while preserving first occurrence order | usage:python3 tools/data/uniq.py file.txttools/data/urldecode.py| goal: URL-decode strings from stdin or files | usage:python3 tools/data/urldecode.pytools/data/between| goal: print text between delimiters | usage:tools/data/between START END < file.txttools/data/csv_get| goal: extract selected CSV fields quickly | usage:tools/data/csv_get file.csv columntools/data/csv2dot| goal: turn CSV relationships into Graphviz dot edges | usage:tools/data/csv2dot
Hashing And Archives
tools/hashing/libarchivesum.py| goal: hash files inside archives without full extraction | usage:python3 tools/hashing/libarchivesum.py archive.ziptools/hashing/scatterhash.py| goal: hash very large files by sparse sampling when you need a fingerprint rather than a comparison | usage:python3 tools/hashing/scatterhash.py huge.imgtools/hashing/hashzip.py| goal: hash ZIP contents or metadata for comparison | usage:python3 tools/hashing/hashzip.py sample.ziptools/hashing/tarsum.py| goal: compute tar-oriented checksums in Python | usage:python3 tools/hashing/tarsum.py archive.tartools/hashing/sparsecmp.sh| goal: compare very large files or block devices by sampling chunks at fixed offsets | usage:tools/hashing/sparsecmp.sh source target 100tools/hashing/trunc_by_hash.py| goal: find the byte length where a rolling hash matches a target digest | usage:python3 tools/hashing/trunc_by_hash.py HASH file.bin
Network And Cloud
tools/network/ipgrep| goal: extract IP or MAC indicators from text | usage:cat file.txt | tools/network/ipgreptools/network/fritzshark.sh| goal: inspect or capture FritzBox traffic workflows | usage:tools/network/fritzshark.shtools/network/fritzshark2.sh| goal: alternate FritzBox traffic workflow | usage:tools/network/fritzshark2.shtools/network/get_ntp.py| goal: query NTP information from remote systems | usage:python3 tools/network/get_ntp.py hosttools/network/get_stp.sh| goal: inspect spanning-tree data on a network | usage:tools/network/get_stp.sh devicetools/cloud/cloudsend.py| goal: upload files to Nextcloud or OwnCloud shares | usage:python3 tools/cloud/cloudsend.py filetools/cloud/cloudsend.sh| goal: shell wrapper for cloud share upload workflows | usage:tools/cloud/cloudsend.sh filetools/cloud/docker_pull.py| goal: download image layers from a container registry withoutdocker pull| usage:python3 tools/cloud/docker_pull.py ubuntu:latesttools/cloud/speech.py| goal: run cloud-backed speech or transcription tasks | usage:python3 tools/cloud/speech.py inputtools/cloud/vqa3.py| goal: classify images with a local or model-backed VQA workflow | usage:python3 tools/cloud/vqa3.py image.jpgtools/cloud/youtube_resolve.sh| goal: resolve direct media URLs from YouTube-like inputs | usage:tools/cloud/youtube_resolve.sh URL
Formats, System, And Text Experiments
tools/formats/convert2pdf.sh| goal: convert documents into PDF form | usage:tools/formats/convert2pdf.sh input.docxtools/formats/flatpdf.sh| goal: flatten or normalize PDFs for downstream handling | usage:tools/formats/flatpdf.sh input.pdftools/formats/openflattenpdf.sh| goal: flatten a PDF through PostScript and open the result | usage:tools/formats/openflattenpdf.sh input.pdftools/formats/rename.mime.py| goal: rename or sort files by MIME type | usage:python3 tools/formats/rename.mime.pytools/system/backup_docker.sh| goal: back up a Docker Compose stack | usage:tools/system/backup_docker.sh docker-compose.ymltools/system/restore_docker.sh| goal: restore a saved Docker workflow | usage:tools/system/restore_docker.shtools/system/watchgrowth.sh| goal: watch a file or directory grow over time | usage:tools/system/watchgrowth.sh pathtools/system/ltop.py| goal: show the most frequent lines from a stream liketop| usage:tail -f log | python3 tools/system/ltop.pytools/system/noerr| goal: run a command with stderr suppressed | usage:tools/system/noerr some commandtools/system/wipe.sh| goal: perform destructive wipe or cleanup steps | usage:tools/system/wipe.sh targettools/text/probability.py| goal: run a small text probability experiment | usage:python3 tools/text/probability.pytools/text/depth| goal: inspect text depth or nesting characteristics | usage:tools/text/depth input.txt
CTF Helpers
tools/ctf/filtertext.py| goal: filter challenge text to useful fragments | usage:python3 tools/ctf/filtertext.py input.txttools/ctf/getjs.py| goal: extract JavaScript from challenge pages | usage:python3 tools/ctf/getjs.py page.htmltools/ctf/guess.py| goal: brute-force or guess through a challenge search space | usage:python3 tools/ctf/guess.py ...tools/ctf/ps_.py| goal: run a CTF-specific parsing or post-processing step | usage:python3 tools/ctf/ps_.py ...tools/ctf/search.py| goal: search challenge artifacts for signals | usage:python3 tools/ctf/search.py inputtools/ctf/submit_flag.sh| goal: submit flags to a challenge endpoint | usage:tools/ctf/submit_flag.sh FLAGtools/ctf/transpose.py| goal: transpose text or matrix-like challenge data | usage:python3 tools/ctf/transpose.py input
Go Tools And Small Projects
tools/go/bincmp/gobincmp.go| goal: compare files or directories with fuzzy hashing | usage:go run tools/go/bincmp/gobincmp.go left righttools/go/gopname/pname.go| goal: demo process-title renaming withgspt| usage:go run tools/go/gopname/pname.gotools/go/tarsum/tarsum.go| goal: print a SHA-256 checksum for a tar file | usage:go run tools/go/tarsum/tarsum.go archive.tarprojects/go-tools/go/goipgrep/| goal: production-grade IP and MAC extractor with ping, DNS, and lookup support | usage:projects/go-tools/go/goipgrep/scripts/build.shprojects/go-tools/go/csv2json/csv2json.go| goal: convert CSV input to JSON | usage:go run projects/go-tools/go/csv2json/csv2json.goprojects/go-tools/go/gobetween/gobetween.go| goal: extract text between delimiters in Go | usage:go run projects/go-tools/go/gobetween/gobetween.goprojects/go-tools/go/goinfo/goinfo.go| goal: inspect file or system information in Go | usage:go run projects/go-tools/go/goinfo/goinfo.goprojects/go-tools/go/gosoft/gosoft.go| goal: enumerate installed software from multiple package sources | usage:go run projects/go-tools/go/gosoft/gosoft.goprojects/go-tools/go/gouniq/gouniq.go| goal: remove duplicate lines in Go | usage:go run projects/go-tools/go/gouniq/gouniq.go < file.txtprojects/rust-tools/between.rs| goal: Rust version of between-delimiter extraction | usage:rustc projects/rust-tools/between.rs && ./betweenprojects/rust-tools/uniq.rs| goal: Rust uniq implementation preserving first occurrences | usage:rustc projects/rust-tools/uniq.rs && ./uniq file.txtprojects/rust-tools/uniq2.rs| goal: alternate Rust uniq implementation | usage:rustc projects/rust-tools/uniq2.rs && ./uniq2 file.txtprojects/puzzlebox/| goal: solve voxel and puzzlebox search problems with several solver variants | usage:python3 projects/puzzlebox/solve.pyprojects/timesketch/deploy_timesketch.sh| goal: deploy a Timesketch environment | usage:projects/timesketch/deploy_timesketch.sh
Admin And Setup Scripts
scripts/proxy/get_proxy.sh| goal: print current proxy settings | usage:scripts/proxy/get_proxy.shscripts/proxy/update_apt_proxy.sh| goal: write apt proxy configuration | usage:scripts/proxy/update_apt_proxy.sh host portscripts/proxy/update_bashrc_proxy.sh| goal: add shell proxy exports to a bash config | usage:scripts/proxy/update_bashrc_proxy.sh host portscripts/proxy/update_service_proxy.sh| goal: apply proxy settings to service units | usage:scripts/proxy/update_service_proxy.sh servicescripts/display/3_screen_setup.sh| goal: apply a fixed three-monitorxrandrlayout | usage:scripts/display/3_screen_setup.shscripts/display/notebook_extended.sh| goal: apply a laptop-plus-external-display layout | usage:scripts/display/notebook_extended.shscripts/display/reset_screens.sh| goal: reset screen outputs to a known state | usage:scripts/display/reset_screens.shscripts/display/single_fullhd.sh| goal: force a single full-HD laptop display mode | usage:scripts/display/single_fullhd.shscripts/display/toggle_display.sh| goal: toggle an external display workflow | usage:scripts/display/toggle_display.shscripts/display/toggle_touchpad| goal: toggle touchpad state on or off | usage:scripts/display/toggle_touchpadscripts/setup/automountctl| goal: manage automount-related setup | usage:scripts/setup/automountctlscripts/setup/disable_ubuntu_telemetry.sh| goal: disable Ubuntu telemetry packages and endpoints | usage:sudo scripts/setup/disable_ubuntu_telemetry.shscripts/setup/mount_container| goal: mount or unmount LUKS container files listed in.containersmanifests | usage:scripts/setup/mount_container mountscripts/setup/share.sh| goal: run a local sharing workflow | usage:scripts/setup/share.shscripts/setup/terminal-logs.sh| goal: configure or collect terminal logging | usage:scripts/setup/terminal-logs.shscripts/setup/update-models.sh| goal: sync lmstudio and ollama model listings into opencode.json | usage:scripts/setup/update-models.sh [lmstudio|ollama]scripts/windows/Get-ZimmermanTools.ps1| goal: download Zimmerman forensic tools on Windows | usage:powershell -File scripts/windows/Get-ZimmermanTools.ps1scripts/windows/getscreen.psm1| goal: provide PowerShell screen-capture helpers | usage:Import-Module scripts/windows/getscreen.psm1scripts/windows/sbom.ps1| goal: generate or inspect SBOM-related data in PowerShell | usage:powershell -File scripts/windows/sbom.ps1
Config And Reference Entry Points
config/visidata/| goal: install and use the local VisiData config plus plugins | usage:cd config/visidata && ./install.sh --linkconfig/install.sh| goal: bootstrap local environment configuration | usage:config/install.shconfig/z.sh| goal: provide a shell directory-jump helper | usage:source config/z.shconfig/shell/completions/eslogger.zsh| goal: add Zsh completion for Apple'seslogger| usage:source config/shell/completions/eslogger.zsh
Archived Or Narrow Tools
archive/experimental/ctf_primefac.py| goal: factor a hard-coded challenge integer withprimefac| usage:python3 archive/experimental/ctf_primefac.pyarchive/experimental/screen2.js| goal: capture a specific webpage screenshot with PhantomJS | usage:phantomjs archive/experimental/screen2.jsarchive/experimental/screenshot.js| goal: capture screenshots for multiple URLs with PhantomJS | usage:phantomjs archive/experimental/screenshot.js URL ...archive/experimental/usbreset.c| goal: reset a USB device from Linux userspace | usage:gcc archive/experimental/usbreset.c -o usbresetarchive/experimental/sep_test.sh| goal: binary-search Docker image tags for a malware detection change | usage:archive/experimental/sep_test.sh image start_tag end_tagarchive/experimental/flm.py| goal: keep an old experimental script available for salvage | usage:python3 archive/experimental/flm.pyarchive/experimental/fuzz.sh| goal: keep an old shell fuzzing experiment available for salvage | usage:archive/experimental/fuzz.sharchive/experimental/hydrogentest.py| goal: keep an old experiment available for salvage | usage:python3 archive/experimental/hydrogentest.pyarchive/experimental/kv.py| goal: keep an old key-value parsing experiment available for salvage | usage:python3 archive/experimental/kv.pyarchive/experimental/lpic.sh| goal: keep an old system experiment available for salvage | usage:archive/experimental/lpic.sharchive/experimental/matplottest.py| goal: keep an old plotting experiment available for salvage | usage:python3 archive/experimental/matplottest.pyarchive/experimental/rootshell.c| goal: keep a dangerous historical C example archived rather than active | usage:do not run; reference only
tools/: Standalone Utilities
Security
tools/security/scan_vt.py: VirusTotal file lookup helper.tools/security/imphash.py: PE import-hash calculator for malware triage.tools/security/scapy_arp.py: ARP-based host discovery on local networks.tools/security/simple_portscan.py: lightweight port scanner.tools/security/smtpbanner.py: banner grabber for SMTP targets.tools/security/testpw.py: password-testing helper.tools/security/vt_download.py,vt_ip.py,vt_pdns.py: VirusTotal-related download and enrichment helpers.tools/security/certwipe: disk wiping helper with secure-erase focused intent.
Forensics
tools/forensics/chechsqlite.py: inspects SQLite databases for password/hash-like fields and consistency issues.tools/forensics/extractfolder.py: folder extraction/helper script for bulk processing.tools/forensics/sqlite32jsonl.py: exports each user table in a SQLite database to a separate JSONL file with argparse help and logging.tools/forensics/mailunpack: containerizedmunpackwrapper for extracting mail attachments safely.tools/forensics/process_leak.py: process-memory or artifact triage helper.tools/forensics/showgm.sh,showosm.sh: extract GPS EXIF data from images and open the location in Google Maps or OpenStreetMap.
Data / Text Processing
tools/data/domgrep.py: extracts domains from URL-ish input.tools/data/geturls.py: URL extraction helper.tools/data/unum.py: Unicode inspection tool that prints code points, categories, and names.tools/data/quickchardet.py: quick character-encoding detection.tools/data/json_save.py,kv_parse.py,vba_chr_decode.py: small transformation helpers for structured or obfuscated data.tools/data/concat.py,split_linewise.py,uniq.py,between,csv_get: general command-line text and record wrangling tools.tools/data/urldecode.py: URL-decoding helper.
Hashing / Archives
tools/hashing/libarchivesum.py: hashes files inside archives without extracting them first.tools/hashing/scatterhash.py: sparse hashing for large files.tools/hashing/hashzip.py: ZIP-oriented hashing helper.tools/hashing/sparsecmp.sh: spot-check large files or block devices by comparing chunks at intervals.tools/hashing/tarsum.py: tar hashing utility in Python.tools/hashing/trunc_by_hash.py: finds the byte offset where a stream first matches a target hash so a file can be truncated deterministically.
Networking / Cloud
tools/network/ipgrep: network indicator extractor for IPs and MACs; there is a fuller Go project version underprojects/go-tools/go/goipgrep/.tools/network/fritzshark.sh,fritzshark2.sh: FritzBox traffic / packet capture helpers.tools/network/get_ntp.py,get_stp.sh: NTP and STP inspection scripts.tools/cloud/cloudsend.py,cloudsend.sh: Nextcloud/OwnCloud share upload helpers.tools/cloud/docker_pull.py: registry client that downloads container images without requiringdocker pull.tools/cloud/speech.py: cloud-backed speech or transcription helper.tools/cloud/vqa3.py: CLIP-based image classification experiment.tools/cloud/youtube_resolve.sh: YouTube/media URL resolution helper.
Formats / System / Misc
tools/formats/convert2pdf.sh,flatpdf.sh,rename.mime.py: document conversion and MIME-based file organization helpers.tools/formats/openflattenpdf.sh: flatten a PDF by round-tripping it through PostScript, then open the result for viewing.tools/system/backup_docker.sh,restore_docker.sh: Docker/Compose backup and restore workflows.tools/system/watchgrowth.sh: live file growth watcher.tools/system/ltop.py:top-style frequency counter for streamed lines.tools/system/noerr: tiny wrapper that runs a command with stderr suppressed.tools/system/wipe.sh: destructive cleanup / wiping helper.tools/ctf/: small CTF-solving utilities such as text filtering, JS extraction, guessing, search, transpose, and flag submission helpers.tools/text/probability.py,tools/text/depth: text-analysis experiments.
Go Tools Under tools/go/
tools/go/bincmp/: fuzzy binary and directory comparison usingssdeep.tools/go/gopname/: process-title renaming demo usinggspt.tools/go/tarsum/: Go tar checksum utility; currently behaves as a raw SHA-256 of the tar file contents.
projects/: Larger Multi-File Work
projects/go-tools/go/goipgrep/: the most complete standalone project in the repo. It extracts IPs and MACs from text and can optionally do ping checks, reverse DNS, and IP info lookups. Read its local README first.projects/go-tools/go/csv2json/,gobetween/,goinfo/,gosoft/,gouniq/: smaller Go experiments and utilities.projects/puzzlebox/: puzzle-solving and voxel-based search experiments with several solver variants.projects/rust-tools/: small Rust utilities and built binaries.projects/timesketch/deploy_timesketch.sh: deployment helper for Timesketch.
config/: Shell, Desktop, and VisiData
config/shell/: shell prompt, aliases, input settings, dircolors, and local completions.config/shell/completions/eslogger.zshis the local Zsh completion for Apple’seslogger.config/applications/: desktop launchers.config/install.sh: environment/bootstrap installer.config/z.sh,config/zupdate.sh: directory-jump helper and update script.config/visidata/: a real subsystem, not just dotfiles. It contains installation logic, avisidatarc, local plugins, validation scripts, sample datasets, and a dedicated README. If you use VisiData in this repo, start there.
dockerfiles/: Container Recipes
dockerfiles/build_firefox.sh,build_kali.sh: convenience builders.dockerfiles/*.dockerfileand subdirectoryDockerfiles: one-off environments for Firefox, Kali, Plaso, Rekall, RegRipper, Volatility, FlatPDF, and several security tools.dockerfiles/elk/: ELK stack setup with a short README about required kernel tuning.dockerfiles/pdf-analysis/: PDF analysis container with Didier Stevens tools, peepdf, and related utilities; see its local README and command help.dockerfiles/droppy/,openhab/: small Compose-based service setups.dockerfiles/logstash/: Logstash configs for different ingestion cases.
scripts/
scripts/proxy/: proxy environment propagation for apt, bash, and services.scripts/display/: display and touchpad toggles, including namedxrandrscreen-layout presets.scripts/setup/: host setup helpers such as automounting, sharing, terminal logging, encrypted container mounting, Ubuntu telemetry disabling, and model listing sync for opencode.scripts/windows/: PowerShell helpers, including Zimmerman tools bootstrap and SBOM-related work.
archive/: Reference Material
archive/collected/: old collected command and tool lists plus a small README.archive/experimental/: scripts kept for reference or later salvage, including older PhantomJS screenshot helpers, one-off CTF code, USB reset code, and narrow test scripts.archive/awk/andarchive/binaries/: older helpers that were kept but de-emphasized.
Where To Start
- If you want a single-file utility, start in
tools/. - If you want a more complete tool with tests or a build flow, check
projects/. - If you want local environment setup or terminal customizations, check
config/. - If you want containers, check
dockerfiles/. - If you are unsure, run
./what -lor ask./what "query".