gists/README.md

gists

This repository is a personal toolbox: small utilities, shell config, Docker build recipes, forensic helpers, network scripts, and a few larger side projects. It is not a single application, so the useful documentation is a map of what lives where and which files matter.

How To Read This Repo

• tools/ is the main home for standalone utilities grouped by purpose.
• projects/ holds larger multi-file projects that deserve their own subtree.
• scripts/ holds machine-management helpers.
• config/ holds shell, desktop, and VisiData configuration.
• dockerfiles/ holds container recipes and small Compose setups.
• archive/ holds older or experimental material that is kept for reference, not as the primary entrypoint.

This README explains the parts that need context. Boilerplate and standard metadata files such as go.mod, go.sum, .gitignore, GitHub Actions files, and obvious launcher/config assets are not documented line-by-line unless they carry important behavior.

Refined Repository Structure

The old restructure_git.sh captured the right direction but in the wrong form. The useful part is the structure itself, not a one-shot move script. The intended layout for this repo is:

• root: only discovery files and broad entrypoints such as README.md, .gitignore, what, and small repo-wide metadata.
• tools/: single-purpose utilities that can usually be understood and run on their own.
• projects/: anything multi-file, stateful, tested, or large enough to deserve its own local README or build flow.
• scripts/: machine-setup and environment-management scripts, grouped by operational area such as proxy, display, setup, or platform.
• config/: reusable configuration, completions, themes, desktop entries, and tool-specific setup.
• dockerfiles/: container recipes and Compose stacks.
• archive/: historical material kept for reference, not the default place to start.

The practical cleanup rules are:

• keep the root shallow; do not leave new standalone utilities at the top level if they fit under tools/, scripts/, or config/
• move one-file tools by function, not by language
• move multi-file codebases by project, even if they are small
• keep generated artifacts, captured output, and binaries out of active source directories unless they are intentionally distributed
• use archive/experimental/ for things worth keeping but not worth advertising
• keep macOS Finder metadata out of the repo; .DS_Store, AppleDouble files, and __MACOSX/ bundles are ignored

Applied to the current tree, the remaining rough edges are:

• a few tracked binaries still sit beside source and may eventually deserve a dedicated bin/ or dist/ convention
• some material under archive/experimental/ may still be worth deleting rather than merely archiving

Top-Level Files

• what: repository search helper. It can list known tools, search by query, and progressively falls back from Ollama-based natural-language search to fzf or plain grep.
• .what_db.json: the metadata database used by what. It stores short descriptions for known tools.
• README.md: this guide.
• .gitignore: standard repository ignore rules.

tools/: Standalone Utilities

Security

• tools/security/scan_vt.py: VirusTotal file lookup helper.
• tools/security/imphash.py: PE import-hash calculator for malware triage.
• tools/security/scapy_arp.py: ARP-based host discovery on local networks.
• tools/security/simple_portscan.py: lightweight port scanner.
• tools/security/smtpbanner.py: banner grabber for SMTP targets.
• tools/security/testpw.py: password-testing helper.
• tools/security/vt_download.py, vt_ip.py, vt_pdns.py: VirusTotal-related download and enrichment helpers.
• tools/security/certwipe: disk wiping helper with secure-erase focused intent.

Forensics

• tools/forensics/chechsqlite.py: inspects SQLite databases for password/hash-like fields and consistency issues.
• tools/forensics/extractfolder.py: folder extraction/helper script for bulk processing.
• tools/forensics/mailunpack: containerized munpack wrapper for extracting mail attachments safely.
• tools/forensics/process_leak.py: process-memory or artifact triage helper.
• tools/forensics/showgm.sh, showosm.sh: extract GPS EXIF data from images and open the location in Google Maps or OpenStreetMap.

Data / Text Processing

• tools/data/domgrep.py: extracts domains from URL-ish input.
• tools/data/geturls.py: URL extraction helper.
• tools/data/unum.py: Unicode inspection tool that prints code points, categories, and names.
• tools/data/quickchardet.py: quick character-encoding detection.
• tools/data/json_save.py, kv_parse.py, vba_chr_decode.py: small transformation helpers for structured or obfuscated data.
• tools/data/concat.py, split_linewise.py, uniq.py, between, csv_get: general command-line text and record wrangling tools.
• tools/data/urldecode.py: URL-decoding helper.

Hashing / Archives

• tools/hashing/libarchivesum.py: hashes files inside archives without extracting them first.
• tools/hashing/scatterhash.py: sparse hashing for large files.
• tools/hashing/hashzip.py: ZIP-oriented hashing helper.
• tools/hashing/sparsecmp.sh: spot-check large files or block devices by comparing chunks at intervals.
• tools/hashing/tarsum.py: tar hashing utility in Python.
• tools/hashing/trunc_by_hash.py: finds the byte offset where a stream first matches a target hash so a file can be truncated deterministically.

Networking / Cloud

• tools/network/ipgrep: network indicator extractor for IPs and MACs; there is a fuller Go project version under projects/go-tools/go/goipgrep/.
• tools/network/fritzshark.sh, fritzshark2.sh: FritzBox traffic / packet capture helpers.
• tools/network/get_ntp.py, get_stp.sh: NTP and STP inspection scripts.
• tools/cloud/cloudsend.py, cloudsend.sh: Nextcloud/OwnCloud share upload helpers.
• tools/cloud/docker_pull.py: registry client that downloads container images without requiring docker pull.
• tools/cloud/speech.py: cloud-backed speech or transcription helper.
• tools/cloud/vqa3.py: CLIP-based image classification experiment.
• tools/cloud/youtube_resolve.sh: YouTube/media URL resolution helper.

Formats / System / Misc

• tools/formats/convert2pdf.sh, flatpdf.sh, rename.mime.py: document conversion and MIME-based file organization helpers.
• tools/formats/openflattenpdf.sh: flatten a PDF by round-tripping it through PostScript, then open the result for viewing.
• tools/system/backup_docker.sh, restore_docker.sh: Docker/Compose backup and restore workflows.
• tools/system/watchgrowth.sh: live file growth watcher.
• tools/system/ltop.py: top-style frequency counter for streamed lines.
• tools/system/noerr: tiny wrapper that runs a command with stderr suppressed.
• tools/system/wipe.sh: destructive cleanup / wiping helper.
• tools/ctf/: small CTF-solving utilities such as text filtering, JS extraction, guessing, search, transpose, and flag submission helpers.
• tools/text/probability.py, tools/text/depth: text-analysis experiments.

Go Tools Under tools/go/

• tools/go/bincmp/: fuzzy binary and directory comparison using ssdeep.
• tools/go/gopname/: process-title renaming demo using gspt.
• tools/go/tarsum/: Go tar checksum utility; currently behaves as a raw SHA-256 of the tar file contents.

projects/: Larger Multi-File Work

• projects/go-tools/go/goipgrep/: the most complete standalone project in the repo. It extracts IPs and MACs from text and can optionally do ping checks, reverse DNS, and IP info lookups. Read its local README first.
• projects/go-tools/go/csv2json/, gobetween/, goinfo/, gosoft/, gouniq/: smaller Go experiments and utilities.
• projects/puzzlebox/: puzzle-solving and voxel-based search experiments with several solver variants.
• projects/rust-tools/: small Rust utilities and built binaries.
• projects/timesketch/deploy_timesketch.sh: deployment helper for Timesketch.

config/: Shell, Desktop, and VisiData

• config/shell/: shell prompt, aliases, input settings, dircolors, and local completions. config/shell/completions/eslogger.zsh is the local Zsh completion for Apple’s eslogger.
• config/applications/: desktop launchers.
• config/install.sh: environment/bootstrap installer.
• config/z.sh, config/zupdate.sh: directory-jump helper and update script.
• config/visidata/: a real subsystem, not just dotfiles. It contains installation logic, a visidatarc, local plugins, validation scripts, sample datasets, and a dedicated README. If you use VisiData in this repo, start there.

dockerfiles/: Container Recipes

• dockerfiles/build_firefox.sh, build_kali.sh: convenience builders.
• dockerfiles/*.dockerfile and subdirectory Dockerfiles: one-off environments for Firefox, Kali, Plaso, Rekall, RegRipper, Volatility, FlatPDF, and several security tools.
• dockerfiles/elk/: ELK stack setup with a short README about required kernel tuning.
• dockerfiles/pdf-analysis/: PDF analysis container with Didier Stevens tools, peepdf, and related utilities; see its local README and command help.
• dockerfiles/droppy/, openhab/: small Compose-based service setups.
• dockerfiles/logstash/: Logstash configs for different ingestion cases.

scripts/

• scripts/proxy/: proxy environment propagation for apt, bash, and services.
• scripts/display/: display and touchpad toggles, including named xrandr screen-layout presets.
• scripts/setup/: host setup helpers such as automounting, sharing, terminal logging, encrypted container mounting, and Ubuntu telemetry disabling.
• scripts/windows/: PowerShell helpers, including Zimmerman tools bootstrap and SBOM-related work.

archive/: Reference Material

• archive/collected/: old collected command and tool lists plus a small README.
• archive/experimental/: scripts kept for reference or later salvage, including older PhantomJS screenshot helpers, one-off CTF code, USB reset code, and narrow test scripts.
• archive/awk/ and archive/binaries/: older helpers that were kept but de-emphasized.

Where To Start

• If you want a single-file utility, start in tools/.
• If you want a more complete tool with tests or a build flow, check projects/.
• If you want local environment setup or terminal customizations, check config/.
• If you want containers, check dockerfiles/.
• If you are unsure, run ./what -l or search through .what_db.json.