619b0bc432
- Move single-file tools to tools/ organized by category (security, forensics, data, etc.) - Move multi-file projects to projects/ (go-tools, puzzlebox, timesketch, rust-tools) - Move system scripts to scripts/ (proxy, display, setup, windows) - Organize config files in config/ (shell, visidata, applications) - Move experimental tools to archive/experimental - Create 'what' fuzzy search tool with progressive enhancement (ollama->fzf->grep) - Add initial metadata database for intelligent tool discovery - Preserve git history using 'git mv' commands
142 lines
4.3 KiB
Markdown
142 lines
4.3 KiB
Markdown
# WARP.md
|
|
|
|
This file provides guidance to WARP (warp.dev) when working with code in this repository.
|
|
|
|
## Repository Overview
|
|
|
|
This is a collection of utility scripts, tools, and gists organized for cybersecurity, forensics, data analysis, and system administration tasks. The repository contains standalone utilities rather than a cohesive application, with scripts written in Python, Bash, Go, JavaScript, PowerShell, and C.
|
|
|
|
## Key Directory Structure
|
|
|
|
- **`codegrab/`** - Main collection of security and analysis tools
|
|
- `ctf/` - CTF challenge solving scripts
|
|
- `puzzlebox/` - 3D puzzle solving algorithms with visualization
|
|
- **`tools/`** - System utilities and data processing tools
|
|
- **`config/`** - System configuration and installation scripts
|
|
- **`systemscripts/`** - System administration and environment setup
|
|
- `proxy/` - Network proxy configuration utilities
|
|
- **`dockerfiles/`** - Docker container build scripts
|
|
- **`collected/`** - Archive of older utilities with documentation
|
|
|
|
## Common Development Tasks
|
|
|
|
### Running Security Analysis Tools
|
|
|
|
Most security tools are standalone and follow this pattern:
|
|
```bash
|
|
# VirusTotal scanning
|
|
./codegrab/scan_vt.py <filename>
|
|
|
|
# Import hash calculation
|
|
python3 codegrab/imphash.py <pe_file>
|
|
|
|
# Network analysis
|
|
./codegrab/scapy_arp.py
|
|
./codegrab/simple_portscan.py
|
|
```
|
|
|
|
### Data Processing Utilities
|
|
|
|
```bash
|
|
# Hash utilities for archives
|
|
python3 tools/libarchivesum.py archive.zip
|
|
|
|
# Unicode character analysis
|
|
echo "text" | python3 tools/unum.py
|
|
|
|
# Domain extraction from URLs
|
|
cat urls.txt | python3 tools/domgrep.py
|
|
|
|
# File organization by MIME type
|
|
python3 tools/rename.mime.py
|
|
```
|
|
|
|
### Docker Environment Management
|
|
|
|
```bash
|
|
# Backup Docker Compose stacks
|
|
./tools/backup_docker.sh docker-compose.yml
|
|
|
|
# Restore Docker environments
|
|
./tools/restore_docker.sh
|
|
|
|
# Build forensics containers
|
|
./dockerfiles/build_kali.sh
|
|
```
|
|
|
|
### System Configuration
|
|
|
|
```bash
|
|
# Install dependencies and configure environment
|
|
./config/install.sh
|
|
|
|
# Proxy configuration
|
|
./systemscripts/proxy/get_proxy.sh
|
|
./systemscripts/proxy/update_apt_proxy.sh
|
|
```
|
|
|
|
## Architecture and Patterns
|
|
|
|
### Security Tools Pattern
|
|
Most security utilities in `codegrab/` follow this pattern:
|
|
- Standalone executables with shebang
|
|
- Take file paths or stdin as input
|
|
- Output results in structured format (often CSV-like with custom separators)
|
|
- Use external APIs (VirusTotal, etc.) with API keys from `~/.virustotal_api_key`
|
|
|
|
### Data Processing Pattern
|
|
Tools in `tools/` directory typically:
|
|
- Accept multiple file inputs via command line arguments
|
|
- Use argparse for option handling
|
|
- Support multiple hash algorithms or processing modes
|
|
- Include error handling for malformed inputs
|
|
|
|
### System Scripts Pattern
|
|
Scripts in `systemscripts/` are designed for:
|
|
- Environment detection and configuration
|
|
- Proxy and network setup automation
|
|
- Service management and monitoring
|
|
- Display and hardware management
|
|
|
|
### Specialized Solvers
|
|
The `puzzlebox/` directory contains algorithmic solvers featuring:
|
|
- 3D spatial problem solving with numpy
|
|
- Visualization using matplotlib
|
|
- Recursive backtracking algorithms
|
|
- Multi-processing optimization variants
|
|
|
|
## Key Dependencies
|
|
|
|
The repository relies on various Python packages that should be available:
|
|
- **Security**: `pefile`, `requests`, `scapy`
|
|
- **Data Processing**: `libarchive-c`, `openpyxl`, `visidata`
|
|
- **Scientific**: `numpy`, `matplotlib`, `scipy`
|
|
- **Forensics**: `AnalyzeMFT`, `pymisp`
|
|
- **System**: `ntplib`, `mac-vendor-lookup`, `dateparser`
|
|
|
|
## API Keys and Configuration
|
|
|
|
Several tools expect API keys in home directory files:
|
|
- `~/.virustotal_api_key` - VirusTotal API access
|
|
- Tools may also use environment variables for proxy configuration (`http_proxy`, etc.)
|
|
|
|
## Testing and Validation
|
|
|
|
Tools are typically tested individually:
|
|
```bash
|
|
# Test with sample data
|
|
python3 codegrab/chechsqlite.py sample.db
|
|
python3 tools/quickchardet.py sample.txt
|
|
|
|
# Validate with CTF challenges
|
|
python3 codegrab/ctf/solve.py
|
|
```
|
|
|
|
## Development Notes
|
|
|
|
- Most utilities are designed as single-file executables for easy deployment
|
|
- Scripts include minimal error handling suitable for command-line usage
|
|
- Many tools output to stdout in formats suitable for piping to other commands
|
|
- Docker-based tools assume availability of container runtime
|
|
- Forensics tools may require elevated privileges for certain operations
|