diff --git a/.cleanup-summary.md b/.cleanup-summary.md new file mode 100644 index 0000000..940d7bd --- /dev/null +++ b/.cleanup-summary.md @@ -0,0 +1,226 @@ +# Repository Cleanup Summary + +## Changes Made + +### ๐Ÿ“ Repository Organization + +1. **Created `docs/` directory** - Moved old documentation files: + - `HELP_SYSTEM.md` + - `HELP_SYSTEM_ENHANCED.md` + - `STREAMLINED_HELP_SYSTEM.md` + - `tool-comparison.md` + - `remnux-dockerfile-focal.txt` + +2. **Updated root directory** - Now contains only active files: + - `README.md` - Comprehensive project documentation + - `CONTRIBUTING.md` - Detailed contribution guide + - `Makefile` - Build automation + - Three Dockerfile variants + - `WARP.md` - AI assistant context + +### ๐Ÿ“š New Documentation + +#### README.md +- Quick start guide +- Three build options (upstream, scratch, Kali) +- Comprehensive usage examples +- Help system documentation +- Tool coverage overview +- Repository structure diagram +- Testing instructions +- Docker Hub information + +#### CONTRIBUTING.md +- Getting started guide +- Adding cheat sheets (with examples) +- Adding TLDR pages (with format guide) +- Adding Fish shell completions +- Importing bulk cheatsheets +- Checking help coverage +- Adding new tools +- Testing procedures +- Quick reference tables + +### ๐Ÿณ Build System + +#### Makefile Targets +```bash +make help # Show available commands +make build-upstream # Build from REMnux upstream (recommended) +make build-scratch # Build from Ubuntu 20.04 base +make build-kali # Build from Kali Linux base +make build-all # Build all variants +make test # Test all builds +make test-upstream # Test upstream build +make test-scratch # Test scratch build +make test-kali # Test Kali build +make push # Push to Docker Hub +make clean # Remove built images +make shell-upstream # Interactive shell (upstream) +make shell-scratch # Interactive shell (scratch) +make shell-kali # Interactive shell (Kali) +make coverage # Check help coverage +``` + +### ๐Ÿ—๏ธ Three Build Options + +#### 1. Dockerfile.remnux (Upstream - Recommended) +- Uses `remnux/remnux-distro:latest` +- Fast build time +- All REMnux tools included +- Production-ready +- **Fixed:** Added missing `add-tool-cheats.sh` script reference + +#### 2. Dockerfile.scratch (New - Full Control) +- Built from `ubuntu:20.04` +- Complete package control +- Mimics REMnux installation +- Customizable base system +- Includes: + - PowerShell + PSScriptAnalyzer + - Modern shells (zsh, fish) + - Modern tools (fd-find, ripgrep) + - PDF analysis suite + - Malware analysis tools + - Help system integration + +#### 3. Dockerfile (Kali - Legacy) +- Uses `kalilinux/kali-rolling` +- Original implementation +- Kali security tools +- Alternative ecosystem + +### ๐Ÿ”ง Enhanced Features + +All build variants now include: + +1. **PowerShell Integration** + - Cross-platform PowerShell (`pwsh`) + - PSScriptAnalyzer module pre-installed + +2. **Modern Shells** + - zsh with plugins (autosuggestions, syntax-highlighting) + - fish with custom configuration + - Configured prompts and aliases + +3. **Modern CLI Tools** + - fd-find (fast file finder) + - ripgrep (ultra-fast grep) + - fzf (fuzzy finder) + +4. **Help System** + - Cheat sheets for all major tools + - TLDR pages for quick reference + - Fish completions for smart autocomplete + - Coverage checking tool + +### ๐Ÿ“‹ Help Content Management + +Users can now: + +1. **Add cheat sheets** - Simple text files with examples +2. **Add TLDR pages** - Standardized help format +3. **Add fish completions** - Smart command completion +4. **Import bulk** - Convert markdown to help content +5. **Check coverage** - Find tools missing documentation + +See `CONTRIBUTING.md` for detailed instructions. + +## Usage Examples + +### Build and Run + +```bash +# Clone and build +git clone https://github.com/tabledevil/docker_file_analysis.git +cd docker_file_analysis +make build-upstream + +# Run container +docker run -it --rm -v "$(pwd):/data" tabledevil/file-analysis:remnux + +# Inside container +fhelp # Main help +fhelp cheat pdfid.py # Quick examples +fhelp tools pdf # Find tools +zsh # Try zsh shell +fish # Try fish shell +``` + +### Switching Shells + +```bash +# Default bash +bash + +# Try zsh (advanced features) +zsh + +# Try fish (friendly syntax) +fish +``` + +### Help System + +```bash +fhelp # Main menu +fhelp cheat # Quick examples +fhelp tldr # Simplified man pages +fhelp tools pdf # Find PDF tools +fhelp examples # Browse all examples +check-help-coverage.sh # Check documentation coverage +``` + +## Next Steps + +### For Users +1. Read `README.md` for overview +2. Choose a build option +3. Run `make build-upstream` (recommended) +4. Try the help system with `fhelp` + +### For Contributors +1. Read `CONTRIBUTING.md` +2. Add cheat sheets for your favorite tools +3. Test with `make test` +4. Submit pull request + +## File Structure + +``` +docker_file_analysis/ +โ”œโ”€โ”€ README.md # Main documentation โœจ NEW +โ”œโ”€โ”€ CONTRIBUTING.md # Contribution guide โœจ NEW +โ”œโ”€โ”€ Makefile # Build automation โœจ NEW +โ”œโ”€โ”€ Dockerfile # Kali variant +โ”œโ”€โ”€ Dockerfile.remnux # REMnux variant (fixed) +โ”œโ”€โ”€ Dockerfile.scratch # Ubuntu variant โœจ NEW +โ”œโ”€โ”€ WARP.md # AI context +โ”œโ”€โ”€ files/ # Config files +โ”‚ โ”œโ”€โ”€ README +โ”‚ โ”œโ”€โ”€ command_help +โ”‚ โ”œโ”€โ”€ zshrc +โ”‚ โ””โ”€โ”€ fish_config.fish +โ”œโ”€โ”€ scripts/ # Helper scripts +โ”‚ โ”œโ”€โ”€ fhelp +โ”‚ โ”œโ”€โ”€ add-tool-cheats.sh +โ”‚ โ”œโ”€โ”€ check-help-coverage.sh +โ”‚ โ”œโ”€โ”€ import-remnux-cheatsheets.sh +โ”‚ โ””โ”€โ”€ ... +โ”œโ”€โ”€ cheatsheets/ # Custom cheat sheets +โ”œโ”€โ”€ docs/ # Archived documentation โœจ NEW +โ””โ”€โ”€ tests/ # Test scripts +``` + +## Benefits + +1. **Cleaner Repository** - Organized structure, clear purpose +2. **Better Documentation** - Comprehensive guides for users and contributors +3. **Flexible Building** - Three build options for different needs +4. **Automated Testing** - Make targets for CI/CD +5. **Enhanced Features** - Modern shells, PowerShell, better help +6. **Easy Contributing** - Clear guidelines and examples + +--- + +Repository cleanup completed on 2025-10-01 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..099685c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,614 @@ +# Contributing to File Analysis Container + +Thank you for your interest in improving the File Analysis Container! This guide will help you add and modify help content, cheat sheets, TLDR pages, and shell configurations. + +## ๐Ÿ“‹ Table of Contents + +- [Getting Started](#getting-started) +- [Adding Cheat Sheets](#adding-cheat-sheets) +- [Adding TLDR Pages](#adding-tldr-pages) +- [Adding Fish Shell Completions](#adding-fish-shell-completions) +- [Importing Bulk Cheatsheets](#importing-bulk-cheatsheets) +- [Checking Help Coverage](#checking-help-coverage) +- [Adding New Tools](#adding-new-tools) +- [Testing Your Changes](#testing-your-changes) + +## ๐Ÿš€ Getting Started + +### Prerequisites + +```bash +# Clone the repository +git clone https://github.com/tabledevil/docker_file_analysis.git +cd docker_file_analysis + +# Build the container +make build-upstream +``` + +### Directory Structure + +``` +docker_file_analysis/ +โ”œโ”€โ”€ scripts/ +โ”‚ โ”œโ”€โ”€ fhelp # Main help system +โ”‚ โ”œโ”€โ”€ add-tool-cheats.sh # Add default cheat sheets +โ”‚ โ”œโ”€โ”€ create-offline-help-system.sh # Initialize help database +โ”‚ โ”œโ”€โ”€ import-remnux-cheatsheets.sh # Import from markdown +โ”‚ โ””โ”€โ”€ convert-remnux-cheats.py # Markdown converter +โ”œโ”€โ”€ cheatsheets/ +โ”‚ โ””โ”€โ”€ *.cheat # Custom cheat sheets +โ””โ”€โ”€ files/ + โ”œโ”€โ”€ command_help # Detailed examples + โ””โ”€โ”€ README # Welcome message +``` + +## ๐Ÿ“ Adding Cheat Sheets + +Cheat sheets provide quick command examples and common usage patterns. + +### Format + +Cheat sheets use a simple markdown-like syntax: + +```bash +# tool-name - Short description + +# Example description +tool-name basic_command + +# Another example with options +tool-name --option argument + +# More complex example +tool-name --verbose --output result.txt input.txt +``` + +### Location + +- **Personal cheat sheets:** `/opt/cheatsheets/personal/` +- **Community cheat sheets:** `/opt/cheatsheets/community/` + +### Creating a New Cheat Sheet + +**Inside the container:** + +```bash +# Create the cheat sheet +cat > /opt/cheatsheets/personal/mytool << 'EOF' +# mytool - Analysis tool for files + +# Basic file analysis +mytool sample.bin + +# Verbose output +mytool -v sample.bin + +# Export results to JSON +mytool --format json --output results.json sample.bin + +# Scan directory recursively +mytool -r /data/samples/ + +# Common analysis workflow +mytool --extract-strings sample.bin | grep -i "http" +EOF + +# Test it +fhelp cheat mytool +``` + +**Outside the container (for permanent addition):** + +```bash +# Add to repository +cat > cheatsheets/mytool.cheat << 'EOF' +# mytool - Analysis tool for files +... +EOF + +# Rebuild container +make build-upstream +``` + +### Cheat Sheet Best Practices + +1. **Start with tool name and brief description** +2. **Order examples from simple to complex** +3. **Include common workflow patterns** +4. **Add security-relevant examples** +5. **Keep examples practical and concise** + +### Example: PDF Analysis Cheat Sheet + +```bash +# pdfid.py - Quick PDF structure analysis + +# Basic PDF analysis +pdfid.py document.pdf + +# Scan all PDFs in directory +pdfid.py *.pdf + +# Look for suspicious elements +pdfid.py -a document.pdf + +# Export to JSON +pdfid.py -o json document.pdf + +# Detect common malicious patterns +pdfid.py document.pdf | grep -i "javascript\|openaction\|aa\|launch" +``` + +## ๐Ÿ“– Adding TLDR Pages + +TLDR pages are simplified man pages focusing on practical examples. + +### Format + +TLDR pages follow a specific structure: + +```markdown +# tool-name + +> Brief description of what the tool does. +> More information: . + +- Brief description of example: + +`tool-name argument` + +- Another example with options: + +`tool-name --option {{value}}` + +- Complex example: + +`tool-name --input {{input.txt}} --output {{output.txt}}` +``` + +### Location + +- **TLDR cache:** `~/.local/share/tldr/` +- **Pages:** `~/.local/share/tldr/pages/common/` + +### Creating a TLDR Page + +```bash +# Create directory if needed +mkdir -p ~/.local/share/tldr/pages/common + +# Create TLDR page +cat > ~/.local/share/tldr/pages/common/mytool.md << 'EOF' +# mytool + +> Analysis tool for examining potentially malicious files. +> Part of the file analysis toolkit. +> More information: . + +- Analyze a single file: + +`mytool {{path/to/file}}` + +- Analyze with verbose output: + +`mytool --verbose {{path/to/file}}` + +- Export analysis results: + +`mytool --output {{results.json}} {{path/to/file}}` + +- Scan directory recursively: + +`mytool --recursive {{path/to/directory}}` +EOF + +# Test it +tldr mytool +# OR +fhelp tldr mytool +``` + +### TLDR Best Practices + +1. **Use placeholders** like `{{filename}}` for user input +2. **Keep descriptions brief** (one line) +3. **Include URL** to official documentation +4. **Start with simplest examples** +5. **Cover most common use cases** (4-8 examples) + +## ๐Ÿš Adding Fish Shell Completions + +Fish completions provide smart command-line autocompletion. + +### Format + +Fish completions use a declarative syntax: + +```fish +# mytool completions +complete -c mytool -s v -l verbose -d "Enable verbose output" +complete -c mytool -s o -l output -d "Output file" -r +complete -c mytool -s r -l recursive -d "Scan recursively" +complete -c mytool -s h -l help -d "Show help message" +``` + +### Location + +Fish completions go in: `~/.config/fish/completions/` + +### Creating Fish Completions + +```bash +# Create directory +mkdir -p ~/.config/fish/completions + +# Create completions file +cat > ~/.config/fish/completions/mytool.fish << 'EOF' +# mytool - File analysis tool completions + +# Basic options +complete -c mytool -s v -l verbose -d "Verbose output" +complete -c mytool -s h -l help -d "Show help" +complete -c mytool -s V -l version -d "Show version" + +# File/directory arguments +complete -c mytool -s o -l output -d "Output file" -r -F +complete -c mytool -s i -l input -d "Input file" -r -F + +# Analysis options +complete -c mytool -l format -d "Output format" -r -a "json xml text" +complete -c mytool -l extract -d "Extract embedded files" +complete -c mytool -s r -l recursive -d "Recursive scan" +EOF + +# Test it (restart fish or source completions) +fish +mytool -- +``` + +### Fish Completion Best Practices + +1. **Complete both short (-v) and long (--verbose) options** +2. **Add descriptions** with `-d "description"` +3. **Use `-r`** for options requiring arguments +4. **Use `-F`** for file/path completion +5. **Use `-a`** for fixed value choices + +## ๐Ÿ“ฆ Importing Bulk Cheatsheets + +For adding many cheat sheets at once from a consolidated markdown file. + +### Prepare Your Markdown + +Create a consolidated cheatsheet file with this structure: + +```markdown +## tool1 + +```bash +# Example 1 +tool1 command +``` + +## tool2 + +```bash +# Example 2 +tool2 --option value +``` +``` + +### Import Process + +```bash +# Inside the container: + +# Place your markdown file in /data +# (mounted from host) + +# Import cheatsheets +import-remnux-cheatsheets.sh /data/my-cheatsheets.md + +# Verify import +fhelp cheat tool1 +fhelp cheat tool2 +``` + +### Converter Script + +The `convert-remnux-cheats.py` script handles the conversion: + +```bash +# Convert markdown to cheat/tldr format +convert-remnux-cheats.py input.md output_dir/ + +# This creates: +# output_dir/cheat/tool1 +# output_dir/cheat/tool2 +# output_dir/tldr/tool1.md +# output_dir/tldr/tool2.md +``` + +### Bulk Import Example + +```bash +# Example workflow +cat > my-tools.md << 'EOF' +## exiftool + +```bash +# Extract metadata +exiftool file.jpg + +# Remove all metadata +exiftool -all= file.jpg +``` + +## binwalk + +```bash +# Scan for embedded files +binwalk firmware.bin + +# Extract files +binwalk -e firmware.bin +``` +EOF + +# Import +import-remnux-cheatsheets.sh my-tools.md + +# Verify +fhelp cheat exiftool +fhelp cheat binwalk +``` + +## ๐Ÿ” Checking Help Coverage + +Use the coverage checker to find tools missing documentation. + +### Run Coverage Check + +```bash +# Inside container: +check-help-coverage.sh + +# Output shows: +# โœ“ Tools with both cheat and TLDR +# โš  Tools with only cheat +# โš  Tools with only TLDR +# โœ— Tools with no help +``` + +### Example Output + +``` +Checking help coverage for analysis tools... + +โœ“ pdfid [cheat] [tldr] +โœ“ pdf-parser [cheat] [tldr] +โš  oledump [cheat] [ ] +โš  rtfdump [ ] [tldr] +โœ— newtool [ ] [ ] + +Summary: + Total tools: 5 + Complete: 2 (40%) + Partial: 2 (40%) + Missing: 1 (20%) +``` + +### Adding Missing Help + +Based on coverage report, add missing help content: + +```bash +# Tool with no help +cat > /opt/cheatsheets/personal/newtool << 'EOF' +# newtool - Description +# Basic usage +newtool file +EOF + +# Tool missing TLDR +mkdir -p ~/.local/share/tldr/pages/common +cat > ~/.local/share/tldr/pages/common/oledump.md << 'EOF' +# oledump +> Analyze OLE files +... +EOF + +# Verify +check-help-coverage.sh +``` + +## ๐Ÿ”ง Adding New Tools + +### 1. Add to Dockerfile + +```dockerfile +# Install via apt +RUN apt-get update && apt-get install -y newtool + +# OR install via pip +RUN pip install newtool + +# OR install from source +RUN git clone https://github.com/author/newtool /opt/newtool \ + && cd /opt/newtool \ + && make install +``` + +### 2. Add Help Content + +```bash +# Cheat sheet +cat > cheatsheets/newtool.cheat << 'EOF' +# newtool - Tool description +# Basic usage +newtool file.bin +EOF + +# TLDR page +cat > tldr/newtool.md << 'EOF' +# newtool +> Tool description +- Basic usage: +`newtool {{file}}` +EOF +``` + +### 3. Add to Help System Script + +Edit `scripts/add-tool-cheats.sh`: + +```bash +# Add newtool cheat sheet +cat > "$CHEAT_DIR/newtool" << 'EOF' +# newtool - Tool description +# Basic usage +newtool file.bin +EOF +``` + +### 4. Rebuild and Test + +```bash +# Rebuild container +make build-upstream + +# Test +docker run --rm tabledevil/file-analysis:remnux newtool --version +docker run --rm tabledevil/file-analysis:remnux fhelp cheat newtool +``` + +## ๐Ÿงช Testing Your Changes + +### Test Locally + +```bash +# Build container +make build-upstream + +# Run interactive session +docker run -it --rm tabledevil/file-analysis:remnux + +# Inside container, test help system +fhelp cheat yourtool +fhelp tldr yourtool +``` + +### Test Help System + +```bash +# Test main help +docker run --rm tabledevil/file-analysis:remnux fhelp + +# Test tool search +docker run --rm tabledevil/file-analysis:remnux fhelp tools pdf + +# Test cheat access +docker run --rm tabledevil/file-analysis:remnux fhelp cheat pdfid + +# Run coverage check +docker run --rm tabledevil/file-analysis:remnux check-help-coverage.sh +``` + +### Test Shells + +```bash +# Test zsh +docker run --rm -it tabledevil/file-analysis:remnux zsh + +# Test fish +docker run --rm -it tabledevil/file-analysis:remnux fish + +# Test completions +docker run --rm -it tabledevil/file-analysis:remnux fish -c "complete -C'pdfid.py -'" +``` + +## ๐Ÿ“ค Submitting Changes + +### Create a Pull Request + +1. **Fork the repository** +2. **Create a feature branch** + ```bash + git checkout -b add-mytool-help + ``` +3. **Make your changes** + - Add cheat sheets + - Add TLDR pages + - Update scripts if needed +4. **Test your changes** + ```bash + make build-upstream + make test + ``` +5. **Commit and push** + ```bash + git add cheatsheets/mytool.cheat + git commit -m "Add cheat sheet for mytool" + git push origin add-mytool-help + ``` +6. **Open pull request** on GitHub + +### Commit Message Guidelines + +``` +Add cheat sheet for mytool + +- Include basic usage examples +- Add common workflow patterns +- Covers malware analysis use cases +``` + +## ๐ŸŽฏ Quick Reference + +### Common Tasks + +| Task | Command | +|------|---------| +| Add cheat sheet | `vim /opt/cheatsheets/personal/tool` | +| Add TLDR page | `vim ~/.local/share/tldr/pages/common/tool.md` | +| Test cheat | `fhelp cheat tool` | +| Test TLDR | `tldr tool` or `fhelp tldr tool` | +| Check coverage | `check-help-coverage.sh` | +| Import bulk | `import-remnux-cheatsheets.sh file.md` | + +### File Locations + +| Content Type | Location | +|--------------|----------| +| Cheat sheets | `/opt/cheatsheets/personal/` | +| TLDR pages | `~/.local/share/tldr/pages/common/` | +| Fish completions | `~/.config/fish/completions/` | +| Help script | `/usr/local/bin/fhelp` | + +## ๐Ÿ’ก Tips and Best Practices + +1. **Test in container** - Always test help content inside the container +2. **Keep examples practical** - Focus on real analysis workflows +3. **Use consistent formatting** - Follow existing cheat sheet styles +4. **Add context** - Include brief descriptions for each example +5. **Cover common cases** - 80% of use cases with 20% of examples +6. **Security focus** - Emphasize malware analysis patterns +7. **Cross-reference** - Link related tools in descriptions + +## ๐Ÿ†˜ Getting Help + +- **Issues**: [GitHub Issues](https://github.com/tabledevil/docker_file_analysis/issues) +- **Discussions**: [GitHub Discussions](https://github.com/tabledevil/docker_file_analysis/discussions) +- **Documentation**: See `docs/` directory for archived documentation + +## ๐Ÿ“š Additional Resources + +- [cheat syntax guide](https://github.com/cheat/cheat) +- [TLDR pages format](https://github.com/tldr-pages/tldr/blob/main/CONTRIBUTING.md) +- [Fish shell completions](https://fishshell.com/docs/current/completions.html) +- [REMnux documentation](https://docs.remnux.org/) + +--- + +Thank you for contributing to the File Analysis Container project! ๐ŸŽ‰ diff --git a/Dockerfile b/Dockerfile index 93a055a..4286f9a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,6 +11,8 @@ RUN apt-get update && apt-get install -y \ catdoc \ docx2txt \ exiftool \ + fd-find \ + fish \ git \ imagemagick \ libboost-python-dev \ @@ -28,13 +30,18 @@ RUN apt-get update && apt-get install -y \ pev \ pkg-config \ pipx \ + powershell \ python3 \ python3-lxml \ python3-pip \ + ripgrep \ ruby \ unrtf \ unzip \ wget \ + zsh \ + zsh-autosuggestions \ + zsh-syntax-highlighting \ ; \ rm -rf /var/lib/apt/lists/* @@ -53,11 +60,22 @@ RUN gem install origami RUN sed -i '/PDF/s/"none"/"read|write"/' /etc/ImageMagick-6/policy.xml +# PowerShell: Install PSScriptAnalyzer module +RUN pwsh -NoProfile -Command "Install-Module -Name PSScriptAnalyzer -Force" + ENV PATH="${PATH}:/opt/didierstevenssuite/:/opt/pypy2.7-v7.3.5-linux64/bin" ADD files/README /opt/README ADD files/command_help /opt/command_help RUN echo 'cat /opt/README' >> /etc/bash.bashrc +# Add shell configurations +ADD files/zshrc /etc/zsh/zshrc +ADD files/fish_config.fish /etc/fish/conf.d/remnux.fish + +# Add helper scripts for cheat sheet management +ADD scripts/add-tool-cheats.sh /tmp/add-tool-cheats.sh +RUN chmod +x /tmp/add-tool-cheats.sh && /tmp/add-tool-cheats.sh && rm /tmp/add-tool-cheats.sh + ### VIPERMONKEY ## install pypy diff --git a/Dockerfile.remnux b/Dockerfile.remnux index 239cd5b..2ad9640 100644 --- a/Dockerfile.remnux +++ b/Dockerfile.remnux @@ -10,10 +10,16 @@ RUN apt-get update && apt-get install -y \ busybox \ catdoc \ docx2txt \ + fd-find \ + fish \ fzf \ mc \ pipx \ + ripgrep \ unrtf \ + zsh \ + zsh-autosuggestions \ + zsh-syntax-highlighting \ && rm -rf /var/lib/apt/lists/* # Configure pip @@ -38,38 +44,42 @@ RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps cheat \ && PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps tldr -# Install navi for interactive cheat sheets -RUN wget https://github.com/denisidoro/navi/releases/download/v2.23.0/navi-v2.23.0-x86_64-unknown-linux-musl.tar.gz -O /tmp/navi.tar.gz \ - && tar -xzf /tmp/navi.tar.gz -C /usr/local/bin \ - && chmod +x /usr/local/bin/navi \ - && rm /tmp/navi.tar.gz +# Removed navi - focus on tldr and cheat for reliable help system # Create data directory and set permissions RUN mkdir -p /data \ && chown remnux:remnux /data -# Add documentation and search tools +# Add documentation and streamlined help system ADD files/README /opt/README ADD files/command_help /opt/command_help -ADD scripts/download-docs.sh /usr/local/bin/download-docs.sh +ADD files/zshrc /etc/zsh/zshrc +ADD files/fish_config.fish /etc/fish/conf.d/remnux.fish +ADD scripts/create-offline-help-system.sh /usr/local/bin/create-offline-help-system.sh ADD scripts/find-tool /usr/local/bin/find-tool -ADD scripts/help /usr/local/bin/help -ADD cheatsheets/ /opt/cheatsheets/ +ADD scripts/fhelp /usr/local/bin/fhelp +ADD scripts/import-remnux-cheatsheets.sh /usr/local/bin/import-remnux-cheatsheets.sh +ADD scripts/convert-remnux-cheats.py /usr/local/bin/convert-remnux-cheats.py +ADD scripts/add-tool-cheats.sh /usr/local/bin/add-tool-cheats.sh -# Download REMnux documentation and create searchable database -RUN chmod +x /usr/local/bin/download-docs.sh /usr/local/bin/find-tool /usr/local/bin/help \ - && /usr/local/bin/download-docs.sh +# Create streamlined offline help system (tldr + cheat) +RUN chmod +x /usr/local/bin/create-offline-help-system.sh /usr/local/bin/find-tool /usr/local/bin/fhelp /usr/local/bin/import-remnux-cheatsheets.sh \ + && chmod +x /usr/local/bin/convert-remnux-cheats.py /usr/local/bin/add-tool-cheats.sh \ + && /usr/local/bin/create-offline-help-system.sh \ + && /usr/local/bin/add-tool-cheats.sh # Update bashrc with welcome message and comprehensive help info RUN echo 'cat /opt/README' >> /etc/bash.bashrc \ && echo 'echo ""' >> /etc/bash.bashrc \ && echo 'echo "๐Ÿ“š Comprehensive Help System:"' >> /etc/bash.bashrc \ - && echo 'echo " help - Complete help system"' >> /etc/bash.bashrc \ - && echo 'echo " help tools pdf - Find PDF analysis tools"' >> /etc/bash.bashrc \ - && echo 'echo " help cheat pdfid.py - Show command examples"' >> /etc/bash.bashrc \ - && echo 'echo " help examples - Browse all examples (navi)"' >> /etc/bash.bashrc \ - && echo 'echo " help pdf - PDF analysis workflow"' >> /etc/bash.bashrc \ - && echo 'echo ""' >> /etc/bash.bashrc + && echo 'echo " fhelp - Complete file analysis help"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp tools pdf - Find PDF analysis tools"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp cheat pdfid.py - Show command examples"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp examples - Browse all examples"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp pdf - PDF analysis workflow"' >> /etc/bash.bashrc \ + && echo 'echo ""' >> /etc/bash.bashrc \ + && echo 'alias analyse="fhelp"' >> /etc/bash.bashrc \ + && echo 'alias ?="fhelp"' >> /etc/bash.bashrc # Switch to remnux user (REMnux default user) USER remnux diff --git a/Dockerfile.scratch b/Dockerfile.scratch new file mode 100644 index 0000000..ea3b620 --- /dev/null +++ b/Dockerfile.scratch @@ -0,0 +1,212 @@ +# Build File Analysis Container from scratch using Ubuntu 20.04 +# This provides full control over package selection and configuration + +FROM ubuntu:20.04 +LABEL maintainer="tabledevil" +LABEL description="File Analysis toolkit built from Ubuntu 20.04 base" + +USER root +ARG DEBIAN_FRONTEND=noninteractive +ENV TZ=Europe/Berlin + +# Install essential system utilities +RUN apt-get update && apt-get install -y \ + apt-transport-https \ + build-essential \ + ca-certificates \ + curl \ + git \ + gnupg \ + software-properties-common \ + sudo \ + tzdata \ + wget \ + && rm -rf /var/lib/apt/lists/* + +# Install Python and Node.js repositories +RUN apt-get update && apt-get install -y \ + python3 \ + python3-pip \ + python3-dev \ + python3-setuptools \ + python3-wheel \ + python3-lxml \ + npm \ + && rm -rf /var/lib/apt/lists/* + +# Install Ruby +RUN apt-get update && apt-get install -y \ + ruby \ + ruby-dev \ + && rm -rf /var/lib/apt/lists/* + +# Install PowerShell +RUN wget -q https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb \ + && dpkg -i packages-microsoft-prod.deb \ + && rm packages-microsoft-prod.deb \ + && apt-get update \ + && apt-get install -y powershell \ + && rm -rf /var/lib/apt/lists/* + +# Install core analysis tools +RUN apt-get update && apt-get install -y \ + autoconf \ + binutils \ + binwalk \ + busybox \ + catdoc \ + docx2txt \ + exiftool \ + fd-find \ + file \ + fish \ + foremost \ + fzf \ + hexedit \ + imagemagick \ + jq \ + libboost-python-dev \ + libboost-thread-dev \ + libjpeg-dev \ + libreoffice \ + libssl-dev \ + libtool \ + mc \ + mpack \ + osslsigncode \ + p7zip-full \ + pdftk \ + pev \ + pipx \ + pkg-config \ + ripgrep \ + sqlite3 \ + ssdeep \ + tesseract-ocr \ + unrtf \ + unzip \ + upx-ucl \ + xxd \ + yara \ + zsh \ + zsh-autosuggestions \ + zsh-syntax-highlighting \ + && rm -rf /var/lib/apt/lists/* + +# Configure pip +ENV PYTHONDONTWRITEBYTECODE=1 +ADD pip.conf /etc/pip.conf + +# Upgrade pip +RUN python3 -m pip install --upgrade pip + +# Install PowerShell PSScriptAnalyzer module +RUN pwsh -NoProfile -Command "Install-Module -Name PSScriptAnalyzer -Force" + +# Install Didier Stevens Suite (PDF analysis tools) +RUN git clone https://github.com/DidierStevens/DidierStevensSuite /opt/didierstevenssuite \ + && chmod +x /opt/didierstevenssuite/*py \ + && ln -s /opt/didierstevenssuite/pdfid.py /usr/local/bin/pdfid.py \ + && ln -s /opt/didierstevenssuite/pdf-parser.py /usr/local/bin/pdf-parser.py \ + && ln -s /opt/didierstevenssuite/oledump.py /usr/local/bin/oledump.py \ + && ln -s /opt/didierstevenssuite/rtfdump.py /usr/local/bin/rtfdump.py + +# Install Python analysis tools +RUN python3 -m pip install --break-system-packages \ + psutil \ + oletools \ + pefile \ + python-magic \ + pytz \ + requests \ + six + +# Install PDF tools via Ruby +RUN gem install origami + +# Configure ImageMagick for PDF processing +RUN sed -i '/PDF/s/"none"/"read|write"/' /etc/ImageMagick-6/policy.xml + +# Install PyPy for performance-critical tasks +RUN wget -O- https://downloads.python.org/pypy/pypy2.7-v7.3.5-linux64.tar.bz2 | tar -C /opt/ -xvj \ + && ln -s /opt/pypy2.7-v7.3.5-linux64/bin/pypy /usr/local/bin/pypy + +# Install pypy pip +RUN pypy -m ensurepip && pypy -m pip install -U pip + +# Install Mandiant CAPA for malware analysis +RUN wget -O- https://github.com/mandiant/capa/releases/download/v7.4.0/capa-v7.4.0-linux.zip | busybox unzip -d /usr/bin - \ + && chmod +x /usr/bin/capa + +# Install JavaScript sandbox +RUN npm install box-js --global --production + +# Setup pipx environment +ENV PIPX_HOME=/opt/pipx +ENV PIPX_BIN_DIR=/usr/local/bin + +# Install tools via pipx for isolation +RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps dfir-unfurl \ + && PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx inject dfir-unfurl requests six maclookup + +RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps visidata + +RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps oletools + +RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps cheat + +RUN PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install --include-deps tldr + +# Create remnux user (matching REMnux convention) +RUN groupadd -g 1000 -r remnux && \ + useradd -u 1000 -r -g remnux -d /home/remnux -s /bin/bash -c "REMnux User" remnux && \ + mkdir -p /home/remnux && \ + chown -R remnux:remnux /home/remnux + +# Create data directory +RUN mkdir -p /data && chown remnux:remnux /data + +# Add documentation and help system +ADD files/README /opt/README +ADD files/command_help /opt/command_help +ADD files/zshrc /etc/zsh/zshrc +ADD files/fish_config.fish /etc/fish/conf.d/remnux.fish + +# Add helper scripts +ADD scripts/create-offline-help-system.sh /usr/local/bin/create-offline-help-system.sh +ADD scripts/find-tool /usr/local/bin/find-tool +ADD scripts/fhelp /usr/local/bin/fhelp +ADD scripts/import-remnux-cheatsheets.sh /usr/local/bin/import-remnux-cheatsheets.sh +ADD scripts/convert-remnux-cheats.py /usr/local/bin/convert-remnux-cheats.py +ADD scripts/add-tool-cheats.sh /usr/local/bin/add-tool-cheats.sh +ADD scripts/check-help-coverage.sh /usr/local/bin/check-help-coverage.sh + +# Create offline help system +RUN chmod +x /usr/local/bin/create-offline-help-system.sh \ + /usr/local/bin/find-tool \ + /usr/local/bin/fhelp \ + /usr/local/bin/import-remnux-cheatsheets.sh \ + /usr/local/bin/convert-remnux-cheats.py \ + /usr/local/bin/add-tool-cheats.sh \ + /usr/local/bin/check-help-coverage.sh \ + && /usr/local/bin/create-offline-help-system.sh \ + && /usr/local/bin/add-tool-cheats.sh + +# Update bashrc with welcome message +RUN echo 'cat /opt/README' >> /etc/bash.bashrc \ + && echo 'echo ""' >> /etc/bash.bashrc \ + && echo 'echo "๐Ÿ“š Help System:"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp - File analysis help"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp cheat - Command examples"' >> /etc/bash.bashrc \ + && echo 'echo " fhelp tools pdf - Find PDF tools"' >> /etc/bash.bashrc \ + && echo 'echo ""' >> /etc/bash.bashrc \ + && echo 'alias analyse="fhelp"' >> /etc/bash.bashrc \ + && echo 'alias ?="fhelp"' >> /etc/bash.bashrc + +# Set environment +USER remnux +ENV LANG=en_US.UTF-8 +ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/didierstevenssuite:/opt/pypy2.7-v7.3.5-linux64/bin:/home/remnux/.local/bin +WORKDIR /data + +CMD ["/bin/bash"] diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..434e1e8 --- /dev/null +++ b/Makefile @@ -0,0 +1,93 @@ +.PHONY: help build-upstream build-scratch build-kali build-all test clean push + +# Default target +help: + @echo "File Analysis Container - Build System" + @echo "" + @echo "Available targets:" + @echo " build-upstream Build from REMnux upstream image (recommended)" + @echo " build-scratch Build from Ubuntu 20.04 base (full control)" + @echo " build-kali Build from Kali Linux base (legacy)" + @echo " build-all Build all variants" + @echo " test Run tests on all builds" + @echo " test-upstream Test upstream build" + @echo " test-scratch Test scratch build" + @echo " test-kali Test Kali build" + @echo " push Push images to Docker Hub" + @echo " clean Remove all built images" + @echo "" + +# Build targets +build-upstream: + @echo "Building REMnux upstream variant..." + docker build -f Dockerfile.remnux -t tabledevil/file-analysis:remnux . + @echo "โœ“ Build complete: tabledevil/file-analysis:remnux" + +build-scratch: + @echo "Building from scratch (Ubuntu 20.04 base)..." + docker build -f Dockerfile.scratch -t tabledevil/file-analysis:scratch . + @echo "โœ“ Build complete: tabledevil/file-analysis:scratch" + +build-kali: + @echo "Building Kali Linux variant..." + docker build -f Dockerfile -t tabledevil/file-analysis:kali . + @echo "โœ“ Build complete: tabledevil/file-analysis:kali" + +build-all: build-upstream build-scratch build-kali + @echo "โœ“ All variants built successfully" + +# Test targets +test: test-upstream test-scratch test-kali + @echo "โœ“ All tests passed" + +test-upstream: + @echo "Testing REMnux upstream build..." + @docker run --rm tabledevil/file-analysis:remnux bash -c "which fhelp && fhelp cheat pdfid" + @docker run --rm tabledevil/file-analysis:remnux bash -c "which pdfid.py && which capa && which visidata" + @echo "โœ“ Upstream build tests passed" + +test-scratch: + @echo "Testing scratch build..." + @docker run --rm tabledevil/file-analysis:scratch bash -c "which fhelp && fhelp cheat pdfid" + @docker run --rm tabledevil/file-analysis:scratch bash -c "which pdfid.py && which capa" + @echo "โœ“ Scratch build tests passed" + +test-kali: + @echo "Testing Kali build..." + @docker run --rm tabledevil/file-analysis:kali bash -c "which pdfid.py && which capa" + @docker run --rm tabledevil/file-analysis:kali bash -c "which pwsh" + @echo "โœ“ Kali build tests passed" + +# Docker Hub push +push: + @echo "Pushing images to Docker Hub..." + docker push tabledevil/file-analysis:remnux + docker push tabledevil/file-analysis:scratch + docker push tabledevil/file-analysis:kali + docker tag tabledevil/file-analysis:remnux tabledevil/file-analysis:latest + docker push tabledevil/file-analysis:latest + @echo "โœ“ Images pushed successfully" + +# Clean up +clean: + @echo "Removing built images..." + -docker rmi tabledevil/file-analysis:remnux + -docker rmi tabledevil/file-analysis:scratch + -docker rmi tabledevil/file-analysis:kali + -docker rmi tabledevil/file-analysis:latest + @echo "โœ“ Cleanup complete" + +# Interactive shells for testing +shell-upstream: + docker run -it --rm -v "$$(pwd):/data" tabledevil/file-analysis:remnux + +shell-scratch: + docker run -it --rm -v "$$(pwd):/data" tabledevil/file-analysis:scratch + +shell-kali: + docker run -it --rm -v "$$(pwd):/data" tabledevil/file-analysis:kali + +# Run help coverage check +coverage: + @echo "Checking help coverage..." + docker run --rm tabledevil/file-analysis:remnux check-help-coverage.sh diff --git a/README.md b/README.md index 706c841..cc5ed05 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,304 @@ -# docker_file_analysis +# File Analysis Container -## REMnux-Based File Analysis Container +A comprehensive Docker-based toolkit for malware and file forensics analysis, featuring an extensive offline help system and modern shell environments. -This container is now based on the REMnux malware analysis toolkit, providing a comprehensive set of tools for file analysis, especially PDFs and malware samples. +## ๐ŸŽฏ Quick Start -## Usage ```bash -# REMnux-based version -docker build -f Dockerfile.remnux -t tabledevil/file-analysis:remnux . +# Clone the repository +git clone https://github.com/tabledevil/docker_file_analysis.git +cd docker_file_analysis + +# Build using REMnux upstream (recommended) +make build-upstream + +# Run the container docker run -it --rm -v "$(pwd):/data" tabledevil/file-analysis:remnux -# Original Kali-based version (legacy) -docker run -it --rm -v "$(pwd):/data" tabledevil/file-analysis +# Inside the container, get help +fhelp ``` -## Included Tools (REMnux Base + Additional) +## ๐Ÿ“ฆ Build Options -### PDF Analysis Suite (from REMnux) -* **peepdf** - PDF analysis framework with JavaScript detection -* **pdf-parser.py** - Extract and analyze PDF elements (Didier Stevens) -* **pdfid.py** - Quick PDF structure overview (Didier Stevens) -* **origami** - Ruby gem suite (pdfcop, pdfextract, pdfmetadata) -* **pdftk-java** - PDF manipulation and flattening -* **qpdf** - PDF manipulation (merge, convert, transform) -* **pdfresurrect** - Extract previous versions from PDFs -* **pdftool** - Analyze PDF incremental updates +This project offers **three different build strategies** to suit your needs: -### Malware Analysis (Additional) -* **capa** - Malware capability detection (Mandiant) -* **box-js** - JavaScript sandbox analysis -* **oletools** - Office document analysis suite - * oledump.py - * rtfdump.py - * emldump.py - * and more +### 1. **REMnux Upstream** (Recommended) +Uses the official `remnux/remnux-distro` image as a base and adds enhancements. -### Data Analysis & Utilities (Additional) -* **visidata** - Data exploration and analysis -* **unfurl** - URL and data analysis (DFIR) -* **base64dump** - Base64 decoder (Didier Stevens) -* **tesseract** - OCR text extraction -* **exiftool** - Metadata extraction +```bash +make build-upstream +# OR +docker build -f Dockerfile.remnux -t tabledevil/file-analysis:remnux . +``` -### System Tools -* **mc** - Midnight Commander file manager -* **p7zip-full** - Archive utilities -* All standard REMnux tools and utilities - \ No newline at end of file +**Advantages:** +- Fast build (uses pre-built REMnux image) +- Includes all REMnux tools and configurations +- Regular updates from upstream +- Production-ready + +### 2. **From Scratch** (Full Control) +Builds a REMnux-like environment from Ubuntu 20.04 base, mimicking the official build. + +```bash +make build-scratch +# OR +docker build -f Dockerfile.scratch -t tabledevil/file-analysis:scratch . +``` + +**Advantages:** +- Full control over every package and configuration +- Understand exactly what's installed +- Customize the base system +- Smaller final image (optional) + +### 3. **Kali Base** (Legacy) +Original implementation using Kali Linux rolling as the base. + +```bash +make build-kali +# OR +docker build -f Dockerfile -t tabledevil/file-analysis:kali . +``` + +**Advantages:** +- Access to Kali Linux security tools +- Different package ecosystem +- Alternative to REMnux + +## ๐Ÿš€ Usage + +### Basic Analysis Workflow + +```bash +# Start the container with a directory containing files to analyze +docker run -it --rm -v "/path/to/suspicious/files:/data" tabledevil/file-analysis:remnux + +# Inside the container: + +# Get help on available tools +fhelp + +# Find PDF analysis tools +fhelp tools pdf + +# Quick command examples for a specific tool +fhelp cheat pdfid.py + +# Analyze a PDF +pdfid.py suspicious.pdf +pdf-parser.py suspicious.pdf + +# Interactive cheat sheet browser +fhelp examples + +# Switch to zsh or fish for better interactivity +zsh +# or +fish +``` + +### Alternative Shells + +The container includes three shells with different features: + +- **bash** (default) - Traditional, reliable +- **zsh** - Advanced completion, history search, plugins +- **fish** - Friendly syntax, autosuggestions + +```bash +# Try zsh +zsh + +# Try fish +fish +``` + +## ๐Ÿ“š Comprehensive Help System + +The container features an **offline-first help system** with multiple layers: + +### 1. Command-Line Help (`fhelp`) + +```bash +fhelp # Main help menu +fhelp tools pdf # Find PDF analysis tools +fhelp cheat # Quick examples for a tool +fhelp tldr # Simplified man pages +fhelp examples # Browse all cheat sheets interactively +fhelp pdf # PDF analysis workflow guide +``` + +### 2. Tool Coverage + +The help system includes documentation for **100+ analysis tools**: + +- **PDF Analysis:** pdfid, pdf-parser, peepdf, pdftk, qpdf, pdfresurrect, origami suite +- **Office Documents:** oledump, rtfdump, oletools, emldump +- **Malware Analysis:** capa, box-js, strings, vivisect +- **File Inspection:** exiftool, file, binwalk, hexdump +- **Scripting:** python, ruby, perl, powershell (with PSScriptAnalyzer) +- **Data Analysis:** visidata, jq, yq, sqlite3 +- **System Tools:** fd-find, ripgrep, zsh, fish + +### 3. Help Content Types + +- **Cheat Sheets** - Quick command examples and common patterns +- **TLDR Pages** - Simplified, example-focused documentation +- **Fish Completions** - Smart command-line autocompletion +- **Workflow Guides** - Multi-tool analysis procedures + +## ๐Ÿ› ๏ธ Adding & Modifying Help Content + +See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed instructions on: + +- Adding cheat sheets for new tools +- Creating TLDR pages +- Writing fish shell completions +- Importing bulk cheatsheets from markdown +- Checking help coverage for installed tools + +### Quick Example: Add a Cheat Sheet + +```bash +# Inside the container: +cat > /opt/cheatsheets/personal/mytool << 'EOF' +# mytool - Description + +# Basic usage +mytool file.txt + +# Advanced options +mytool -v --output result.txt input.txt +EOF + +# Test it +fhelp cheat mytool +``` + +## ๐Ÿ”ง Included Tools + +### PDF Analysis Suite +- **peepdf** - Interactive PDF analysis with JavaScript detection +- **pdf-parser.py** - Extract and analyze PDF elements (Didier Stevens) +- **pdfid.py** - Quick PDF structure overview +- **pdftk** - PDF manipulation and transformation +- **qpdf** - PDF inspection and transformation +- **origami** - Ruby suite (pdfcop, pdfextract, pdfmetadata) +- **pdfresurrect** - Extract previous versions from PDFs + +### Malware Analysis +- **capa** - Detect malware capabilities (Mandiant) +- **box-js** - JavaScript sandbox +- **oletools** - Office document analysis (oledump, rtfdump, emldump) +- **vivisect** - Malware analysis framework +- **strings** - Extract printable strings +- **upx** - Executable packer/unpacker + +### Modern Shells & Tools +- **PowerShell** - Cross-platform PowerShell with PSScriptAnalyzer +- **zsh** - With autosuggestions and syntax highlighting +- **fish** - Friendly interactive shell +- **fd-find** - Modern, fast file finder +- **ripgrep** - Ultra-fast recursive grep + +### Data Analysis +- **visidata** - Terminal spreadsheet and data explorer +- **unfurl** - URL and forensics data analyzer +- **jq** - JSON processor +- **sqlite3** - Database analysis + +### File Inspection +- **exiftool** - Metadata extraction +- **binwalk** - Firmware analysis +- **hexdump** / **xxd** - Binary viewers +- **file** - File type identification + +## ๐Ÿ“ Repository Structure + +``` +docker_file_analysis/ +โ”œโ”€โ”€ Dockerfile # Kali-based build (legacy) +โ”œโ”€โ”€ Dockerfile.remnux # REMnux upstream build (recommended) +โ”œโ”€โ”€ Dockerfile.scratch # Build from Ubuntu base (full control) +โ”œโ”€โ”€ Makefile # Build automation +โ”œโ”€โ”€ README.md # This file +โ”œโ”€โ”€ CONTRIBUTING.md # How to add/modify help content +โ”œโ”€โ”€ WARP.md # WARP AI assistant context +โ”œโ”€โ”€ files/ # Container configuration files +โ”‚ โ”œโ”€โ”€ README # Welcome message shown on login +โ”‚ โ”œโ”€โ”€ command_help # Detailed command examples +โ”‚ โ”œโ”€โ”€ zshrc # Zsh shell configuration +โ”‚ โ””โ”€โ”€ fish_config.fish # Fish shell configuration +โ”œโ”€โ”€ scripts/ # Helper scripts +โ”‚ โ”œโ”€โ”€ fhelp # Main help system +โ”‚ โ”œโ”€โ”€ create-offline-help-system.sh # Build help database +โ”‚ โ”œโ”€โ”€ add-tool-cheats.sh # Add default cheat sheets +โ”‚ โ”œโ”€โ”€ import-remnux-cheatsheets.sh # Import bulk cheatsheets +โ”‚ โ”œโ”€โ”€ convert-remnux-cheats.py # Convert markdown to cheat format +โ”‚ โ”œโ”€โ”€ check-help-coverage.sh # Verify help coverage +โ”‚ โ””โ”€โ”€ find-tool # Search for tools +โ”œโ”€โ”€ cheatsheets/ # Custom cheat sheets +โ”‚ โ”œโ”€โ”€ pdf-analysis.cheat +โ”‚ โ”œโ”€โ”€ malware-analysis.cheat +โ”‚ โ””โ”€โ”€ system-utilities.cheat +โ”œโ”€โ”€ docs/ # Additional documentation +โ””โ”€โ”€ tests/ # Test scripts +``` + +## ๐Ÿงช Testing + +```bash +# Test all builds +make test + +# Test specific build +docker run --rm tabledevil/file-analysis:remnux fhelp cheat pdfid + +# Run help coverage check +docker run --rm tabledevil/file-analysis:remnux check-help-coverage.sh +``` + +## ๐Ÿณ Docker Hub + +Pre-built images are available: + +```bash +# Pull the latest REMnux-based image +docker pull tabledevil/file-analysis:remnux + +# Pull the Kali-based image (legacy) +docker pull tabledevil/file-analysis:latest +``` + +## ๐Ÿค Contributing + +Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for: + +- Adding new tools +- Improving help content +- Adding cheat sheets and TLDR pages +- Enhancing shell configurations +- Reporting bugs + +## ๐Ÿ“ License + +This project packages various open-source tools. Please respect individual tool licenses. + +## ๐Ÿ™ Acknowledgments + +- [REMnux](https://remnux.org/) - Malware analysis toolkit +- [Didier Stevens](https://blog.didierstevens.com/) - PDF analysis tools +- [Mandiant](https://www.mandiant.com/) - CAPA malware analysis +- [cheat](https://github.com/cheat/cheat) - Cheat sheet system +- [tldr](https://tldr.sh/) - Simplified man pages + +## ๐Ÿ“ฎ Support + +- **Issues**: [GitHub Issues](https://github.com/tabledevil/docker_file_analysis/issues) +- **Discussions**: [GitHub Discussions](https://github.com/tabledevil/docker_file_analysis/discussions) + +--- + +**Security Note**: This container is designed for analyzing potentially malicious files. Always run it with appropriate isolation and never execute untrusted code outside the container. diff --git a/docs/HELP_SYSTEM.md b/docs/HELP_SYSTEM.md new file mode 100644 index 0000000..0e10b40 --- /dev/null +++ b/docs/HELP_SYSTEM.md @@ -0,0 +1,146 @@ +# File Analysis Container - Help System Guide + +The enhanced container includes a comprehensive offline help system for malware and file analysis. + +## ๐ŸŽฏ Main Help Commands + +### Core Help System +```bash +fhelp # Complete help system overview +fhelp --offline # Verify all offline capabilities work +fhelp --all # Show everything available +``` + +### Tool Discovery (Fuzzy Search) +```bash +fhelp tools pdf # Find all PDF-related tools +fhelp tools malware # Find malware analysis tools +fhelp tools javascript # Find JavaScript analysis tools +fhelp tools --interactive # Browse all tools with fzf +fhelp tools --list # List all available tools +fhelp tools --categories # Show tool categories +``` + +### Command Examples & Cheat Sheets +```bash +fhelp cheat pdf # PDF analysis commands +fhelp cheat malware # Malware analysis commands +fhelp cheat system # System utilities commands +fhelp cheat pdfid.py # Specific tool examples +fhelp examples # Browse all examples interactively (navi) +``` + +### Analysis Workflows +```bash +fhelp pdf # Complete PDF analysis workflow +fhelp malware # Malware analysis workflow +fhelp forensics # System forensics workflow +``` + +### Quick Examples (tldr style) +```bash +fhelp quick tar # Quick tar examples +fhelp quick grep # Quick grep examples +fhelp quick find # Quick find examples +``` + +## ๐Ÿ› ๏ธ Direct Tool Access + +### Interactive Cheat Sheet Browser +```bash +navi --fzf # Browse all cheat sheets interactively +navi --print # Print cheat sheet for current directory +``` + +### Standard Help Tools +```bash +cheat tar # Show tar cheat sheet +tldr grep # Quick grep examples +find-tool pdf # Direct tool search (same as fhelp tools) +``` + +## ๐Ÿ“š Available Cheat Sheet Categories + +- **PDF Analysis** - pdfid.py, pdf-parser.py, peepdf, pdftk, origami tools +- **Malware Analysis** - capa, box-js, oledump.py, unfurl, visidata +- **System Utilities** - 7z, mc, grep, find, file operations + +## ๐ŸŽจ Example Usage Sessions + +### Find PDF Tools +```bash +remnux@container:/data$ fhelp tools pdf +๐Ÿ” Searching for: pdf + +๐Ÿ”ง peepdf + Description: Examine elements of the PDF file. + Category: analyze documents pdf + โœ“ Available + +๐Ÿ”ง pdfid.py + Description: Identify suspicious elements of the PDF file. + โœ“ Available +``` + +### Get PDF Analysis Commands +```bash +remnux@container:/data$ fhelp cheat pdf +๐Ÿ“„ PDF Analysis Commands: +pdfid.py +pdf-parser.py +peepdf -i +pdftk cat output flattened_ +``` + +### View Analysis Workflow +```bash +remnux@container:/data$ fhelp pdf +๐Ÿ“„ PDF Analysis Workflow: + +1. Initial Analysis: + pdfid.py sample.pdf # Quick overview + file sample.pdf # Verify file type + +2. Deep Analysis: + pdf-parser.py sample.pdf # Structure analysis + peepdf -i sample.pdf # Interactive analysis +``` + +## ๐Ÿ”ง Troubleshooting + +### If `fhelp` doesn't work: +```bash +# Use the full path +/usr/local/bin/fhelp + +# Check if it's installed +which fhelp +ls -la /usr/local/bin/fhelp +``` + +### If tools are missing: +```bash +# Check offline capabilities +fhelp --offline + +# Verify tool database +ls -la /opt/remnux-docs/tools.db +``` + +### Alternative Commands +- Use `find-tool` instead of `fhelp tools` +- Use `navi --fzf` for interactive browsing +- Use `cheat` or `tldr` for standard command help + +## โšก Quick Reference + +| Command | Purpose | +|---------|---------| +| `fhelp` | Main help system | +| `fhelp tools [term]` | Search for tools | +| `fhelp cheat [tool]` | Show examples | +| `fhelp [workflow]` | Show analysis workflow | +| `navi --fzf` | Interactive cheat sheets | +| `find-tool [term]` | Alternative tool search | + +The entire help system works **completely offline** with local documentation! \ No newline at end of file diff --git a/docs/HELP_SYSTEM_ENHANCED.md b/docs/HELP_SYSTEM_ENHANCED.md new file mode 100644 index 0000000..605f88f --- /dev/null +++ b/docs/HELP_SYSTEM_ENHANCED.md @@ -0,0 +1,154 @@ +# Enhanced File Analysis Help System + +## ๐ŸŽฏ Overview + +This document describes the comprehensive improvements made to the container's help system, addressing all the issues you encountered and providing a robust, offline-capable help experience. + +## โœ… Problems Fixed + +### 1. **Navi Configuration Issues** +- **Problem**: `invalid preview window layout: up:2:nohidden` errors +- **Solution**: Created proper navi configuration file at `/root/.config/navi/config.yaml` with correct fzf settings +- **Result**: Navi now works without preview window errors + +### 2. **Inaccurate Tool Information** +- **Problem**: Tools listed that don't exist (evince, ioc_parser), unclear command names +- **Solution**: Created curated tools database with only verified, available tools and their exact command names +- **Result**: `fhelp tools` now shows accurate, actionable information + +### 3. **Missing Cheat Sheets** +- **Problem**: Limited or missing practical examples for PDF/malware analysis +- **Solution**: Downloaded existing cheat sheets from popular repos and enhanced them with analysis-specific examples +- **Result**: Comprehensive cheat sheets for analysis workflows with exact commands + +### 4. **TLDR Cache Issues** +- **Problem**: TLDR not finding analysis-specific tools +- **Solution**: Created custom TLDR pages for pdfid.py, pdf-parser.py, peepdf, capa +- **Result**: `fhelp quick ` now works for specialized analysis tools + +### 5. **Command Clarity** +- **Problem**: Users couldn't find the exact command to run (e.g., confusion about ioc_parser) +- **Solution**: All help now shows exact command syntax with practical examples +- **Result**: Clear, copy-pasteable commands for all analysis scenarios + +## ๐Ÿ› ๏ธ Enhanced Components + +### 1. **Accurate Tools Database** (`/opt/remnux-docs/tools.db`) +- Only verified, available tools +- Exact command names (not just descriptions) +- Usage examples for each tool +- Availability status indicators + +### 2. **Enhanced Cheat Sheets** (`/opt/cheatsheets/`) +- Downloads existing cheat sheets from `cheat/cheatsheets` repo +- Adds analysis-specific examples on top +- Covers PDF analysis workflow, malware analysis workflow +- Security-focused examples (PDF flattening, password removal) + +### 3. **Interactive Navi Sheets** (`/opt/navi-cheats/`) +- Dynamic command completion with file discovery +- Interactive parameter selection +- Context-aware suggestions + +### 4. **Custom TLDR Pages** (`/root/.local/share/tldr/pages/common/`) +- Analysis-specific tools not in standard TLDR +- Follows standard TLDR format +- Quick reference examples + +### 5. **Unified fhelp Command** +- Single interface to all help systems +- Clear command structure with examples +- Workflow-based help (PDF, malware analysis) +- Offline capability verification + +## ๐Ÿ“‹ Usage Examples + +### Find Tools +```bash +fhelp tools pdf # Find PDF analysis tools +fhelp tools malware # Find malware analysis tools +fhelp tools --interactive # Interactive tool browser +``` + +### Get Command Examples +```bash +fhelp cheat pdf # PDF analysis workflow +fhelp cheat pdftk # pdftk-specific examples +fhelp quick tar # Quick tar examples +fhelp examples # Interactive example browser +``` + +### Analysis Workflows +```bash +fhelp pdf # Step-by-step PDF analysis +fhelp malware # Step-by-step malware analysis +``` + +### Verification +```bash +fhelp --offline # Check all help systems work offline +fhelp --all # Complete overview of capabilities +``` + +## ๐ŸŽฏ Key Improvements for File Analysis + +### PDF Security Analysis +- **Flattening**: `pdftk suspicious.pdf output safe.pdf flatten` +- **Password Removal**: `qpdf --password=PASSWORD --decrypt encrypted.pdf decrypted.pdf` +- **JavaScript Detection**: `strings document.pdf | grep -i "javascript"` +- **Safe Preview**: `convert document.pdf[0] preview.png` + +### Malware Analysis Workflow +- **Capability Detection**: `capa malware.exe` +- **JavaScript Sandbox**: `box-js --output-dir=/tmp suspicious.js` +- **Office Document Analysis**: `oledump.py document.doc` +- **Binary Analysis**: `binwalk malware.bin` + +### Data Analysis +- **Interactive Exploration**: `vd data.csv` +- **URL Analysis**: `unfurl http://suspicious.com/path` + +## ๐Ÿ”ง Testing + +Run comprehensive tests with: +```bash +./test-help-system.sh +``` + +This validates: +- All help commands work correctly +- Tool availability matches documentation +- File structure is correct +- Integration between help systems works + +## ๐Ÿ“ฆ Build Instructions + +The enhanced help system is automatically included when building: + +```bash +docker build -f Dockerfile.remnux -t tabledevil/file-analysis:enhanced . +``` + +## ๐ŸŽ‰ Benefits + +1. **No More Missing Tools**: Only lists tools that actually exist with correct names +2. **Practical Examples**: Real-world analysis commands, not just descriptions +3. **Workflow Guidance**: Step-by-step analysis procedures +4. **Offline Capable**: Works completely offline with comprehensive documentation +5. **Multiple Interfaces**: Choose between fhelp, navi, cheat, tldr based on preference +6. **Security Focus**: Emphasizes safe analysis practices (PDF flattening, sandboxing) + +## ๐Ÿ“ฑ Quick Reference Card + +| Command | Purpose | Example | +|---------|---------|---------| +| `fhelp` | Main help system | `fhelp tools pdf` | +| `fhelp cheat ` | Tool examples | `fhelp cheat pdftk` | +| `fhelp pdf` | PDF workflow | Shows step-by-step PDF analysis | +| `fhelp malware` | Malware workflow | Shows malware analysis steps | +| `navi` | Interactive browser | Browse all examples with fzf | +| `find-tool ` | Tool search | `find-tool javascript` | +| `tldr ` | Quick reference | `tldr tar` | +| `cheat ` | Detailed examples | `cheat 7z` | + +This enhanced system provides comprehensive, accurate, offline-capable help for file analysis workflows while leveraging existing community resources and adding specialized analysis examples. \ No newline at end of file diff --git a/docs/STREAMLINED_HELP_SYSTEM.md b/docs/STREAMLINED_HELP_SYSTEM.md new file mode 100644 index 0000000..a54e3e1 --- /dev/null +++ b/docs/STREAMLINED_HELP_SYSTEM.md @@ -0,0 +1,148 @@ +# Streamlined File Analysis Help System + +## ๐ŸŽฏ **Mission Accomplished!** + +Based on your test feedback and requirements, I've created a **streamlined, reliable help system** that focuses on what works: **TLDR + Cheat**, removing the problematic navi component. + +## โœ… **Issues Fixed** + +### **Problems Resolved:** +1. **โŒ Navi Removed** - No more broken encoding, preview errors, or crashes +2. **โœ… TLDR Working Offline** - Pre-populated cache with custom analysis tool pages +3. **โœ… Cheat Working Perfectly** - Proper configuration, fhelp cheat integration +4. **โœ… All Your Original Tools Covered** - pdfid.py, pdf-parser.py, peepdf, pdftk, capa, etc. + +## ๐Ÿ› ๏ธ **What Works Now** + +### **PDF Analysis Tools (Your Original Focus):** +```bash +# Quick reference +tldr pdfid.py # Standard tldr format +tldr pdf-parser.py # Command examples +tldr peepdf # Interactive analysis + +# Detailed cheat sheets +fhelp cheat pdfid # pdfid.py examples +fhelp cheat pdf-parser # PDF structure analysis +fhelp cheat pdftk # PDF manipulation (flatten, etc.) +fhelp cheat pdf-analysis # Complete workflow +``` + +### **Malware Analysis Tools:** +```bash +# Quick reference +tldr capa # Malware capabilities +tldr box-js # JavaScript sandbox + +# Detailed workflows +fhelp cheat capa # Capability detection +fhelp cheat malware-analysis # Complete workflow +``` + +### **Core Utilities:** +```bash +# Downloaded from popular repos +fhelp cheat tar # Archive examples +fhelp cheat 7z # Compression +fhelp cheat grep # Text search +fhelp cheat find # File search +``` + +## ๐ŸŽฏ **Key Features** + +### **1. Focused on Your Original Tools** +- **pdfid.py** - PDF suspicious element detection +- **pdf-parser.py** - PDF structure analysis +- **peepdf** - Interactive PDF analysis +- **pdftk** - PDF flattening for security +- **capa** - Malware capability detection +- Plus all standard utilities + +### **2. Security-Focused Examples** +```bash +# PDF Security (Your Priority) +pdftk suspicious.pdf output safe.pdf flatten # Remove JavaScript +qpdf --decrypt encrypted.pdf decrypted.pdf # Remove passwords +convert document.pdf[0] preview.png # Safe preview +``` + +### **3. Offline-Capable** +- **No Internet Required** - Everything works offline +- **Pre-populated TLDR Cache** - Standard + custom analysis tools +- **Local Cheat Sheets** - Downloaded from quality repos + custom ones + +### **4. Multiple Interfaces** +- **fhelp** - Main help system with workflows +- **tldr** - Quick command references +- **fhelp cheat** - Detailed examples +- **fhelp examples** - List all available help + +## ๐Ÿ“ฆ **Usage** + +### **Build & Run:** +```bash +# Build streamlined container +docker build -f Dockerfile.remnux -t tabledevil/file-analysis:streamlined . + +# Run with help system +docker run -it tabledevil/file-analysis:streamlined +``` + +### **Inside Container:** +```bash +# Main help and workflows +fhelp # Complete help menu +fhelp pdf # PDF analysis workflow +fhelp malware # Malware analysis workflow + +# Quick references +tldr pdfid.py # Quick pdfid.py examples +tldr capa # Quick capa examples + +# Detailed examples +fhelp cheat pdfid # Comprehensive pdfid.py usage +fhelp cheat pdf-analysis # Complete PDF workflow +fhelp examples # List all available help +``` + +## ๐Ÿ“Š **Resources Included** + +- **15+ Cheat Sheets** - Including your original analysis tools +- **5+ Custom TLDR Pages** - For analysis-specific tools +- **Complete Offline Cache** - No network dependencies +- **Security-Focused Examples** - PDF flattening, password removal +- **Workflow Guidance** - Step-by-step analysis procedures + +## ๐Ÿ† **Test Results** + +Run comprehensive tests: +```bash +./test-streamlined-help.sh +``` + +**Expected Results:** +- โœ… All basic help functions work +- โœ… All fhelp cheat sheets work +- โœ… TLDR functionality works (standard + custom) +- โœ… All analysis tools available +- โœ… Complete offline operation + +## ๐ŸŽ‰ **Benefits Achieved** + +1. **Reliability** - No more crashes or encoding issues +2. **Simplicity** - Focused on tools that work consistently +3. **Completeness** - All your original tools covered with examples +4. **Security Focus** - Practical commands for safe PDF analysis +5. **Offline Operation** - Works without internet connection +6. **Real Examples** - Copy-pasteable commands for actual analysis + +## ๐Ÿš€ **Ready for Production** + +The streamlined container is now **production-ready** with: +- **Removed problematic navi** - No more crashes +- **Working TLDR integration** - Both standard and custom pages +- **Reliable cheat system** - Via fhelp cheat integration +- **Complete coverage** - All your original analysis tools +- **Security examples** - PDF flattening, password removal, etc. + +**Perfect for your file analysis workflow!** ๐ŸŽฏ \ No newline at end of file diff --git a/remnux-dockerfile-focal.txt b/docs/remnux-dockerfile-focal.txt similarity index 100% rename from remnux-dockerfile-focal.txt rename to docs/remnux-dockerfile-focal.txt diff --git a/tool-comparison.md b/docs/tool-comparison.md similarity index 100% rename from tool-comparison.md rename to docs/tool-comparison.md diff --git a/files/fish_config.fish b/files/fish_config.fish new file mode 100644 index 0000000..8becc9c --- /dev/null +++ b/files/fish_config.fish @@ -0,0 +1,34 @@ +# REMnux File Analysis Container - Fish Configuration + +# Aliases +alias ls='ls --color=auto' +alias ll='ls -lah' +alias grep='grep --color=auto' +alias fd='fdfind' +alias rg='rg --color=auto' +alias analyse='fhelp' +alias ?='fhelp' + +# Fish prompt - simple and clean +function fish_prompt + set_color cyan + echo -n 'remnux' + set_color normal + echo -n '@' + set_color blue + echo -n (prompt_hostname) + set_color normal + echo -n ':' + set_color yellow + echo -n (prompt_pwd) + set_color normal + echo -n '> ' +end + +# Welcome message +if test -f /opt/README + cat /opt/README + echo "" + echo "๐Ÿš Shell: fish | Type 'fhelp' for help" + echo "" +end \ No newline at end of file diff --git a/files/zshrc b/files/zshrc new file mode 100644 index 0000000..ee9c684 --- /dev/null +++ b/files/zshrc @@ -0,0 +1,39 @@ +# REMnux File Analysis Container - ZSH Configuration + +# History +HISTFILE=~/.zsh_history +HISTSIZE=10000 +SAVEHIST=10000 +setopt SHARE_HISTORY HIST_IGNORE_ALL_DUPS HIST_FIND_NO_DUPS HIST_REDUCE_BLANKS + +# Navigation +setopt AUTO_CD AUTO_PUSHD PUSHD_IGNORE_DUPS PUSHD_SILENT + +# Completion +autoload -Uz compinit && compinit +zstyle ':completion:*' menu select +zstyle ':completion:*' matcher-list 'm:{a-zA-Z}={A-Za-z}' +setopt COMPLETE_IN_WORD AUTO_MENU + +# Load plugins +[[ -f /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh ]] && \ + source /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh +[[ -f /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh ]] && \ + source /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh + +# Prompt +autoload -U colors && colors +PROMPT='%F{cyan}remnux%f@%F{blue}%m%f:%F{yellow}%~%f%# ' +RPROMPT='%F{green}[%D{%H:%M:%S}]%f' + +# Aliases +alias ls='ls --color=auto' +alias ll='ls -lah' +alias grep='grep --color=auto' +alias fd='fdfind' +alias rg='rg --color=auto' +alias analyse='fhelp' +alias ?='fhelp' + +# Welcome +[[ -f /opt/README ]] && cat /opt/README && echo "" && echo "๐Ÿš Shell: zsh | Type 'fhelp' for help" && echo "" \ No newline at end of file diff --git a/remnux-tldr-cheatsheet.md b/remnux-tldr-cheatsheet.md new file mode 100644 index 0000000..251a873 --- /dev/null +++ b/remnux-tldr-cheatsheet.md @@ -0,0 +1,404 @@ +# REMnux Docker Container - TLDR Tools Cheat Sheet + +## Quick Start +```bash +# Run REMnux container +docker run --rm -it -u remnux -v ~/samples:/home/remnux/files remnux/remnux-distro:focal bash + +# Default credentials: remnux/malware +``` + +--- + +## ๐Ÿ“ GENERAL FILE ANALYSIS + +### ๐Ÿ”ง FILE +**Purpose**: Identify file type using magic numbers +**Syntax**: `file [filename]` +**Example**: `file malware.exe` +**Output**: Shows file type and basic properties + +### ๐Ÿ”ง EXIFTOOL +**Purpose**: Extract metadata from files +**Syntax**: `exiftool [options] [filename]` +**Example**: `exiftool -a malware.exe` +**Output**: Displays all metadata tags + +### ๐Ÿ”ง TRID +**Purpose**: Identify file types using signatures +**Syntax**: `trid [filename]` +**Example**: `trid unknown_file.bin` +**Output**: Shows percentage probability of file types + +### ๐Ÿ”ง DIE (Detect-It-Easy) +**Purpose**: Detect file types and examine properties +**Syntax**: `die [filename]` or `diec [filename]` +**Example**: `die malware.exe` +**Output**: Comprehensive file analysis + +### ๐Ÿ”ง YARA +**Purpose**: Pattern matching for malware identification +**Syntax**: `yara [rulefile] [target]` +**Example**: `yara rules.yar malware.exe` +**Output**: Shows matching rules + +### ๐Ÿ”ง SSDEEP +**Purpose**: Compute fuzzy hashes for similarity detection +**Syntax**: `ssdeep [options] [files]` +**Example**: `ssdeep -b malware1.exe malware2.exe` +**Output**: Fuzzy hash comparison results + +### ๐Ÿ”ง CLAMSCAN +**Purpose**: Scan files for malware signatures +**Syntax**: `clamscan [options] [file/directory]` +**Example**: `clamscan --verbose malware.exe` +**Output**: Virus detection results + +### ๐Ÿ”ง BINWALK +**Purpose**: Extract and analyze firmware images +**Syntax**: `binwalk [options] [filename]` +**Example**: `binwalk -e firmware.bin` +**Output**: Extracted files and entropy analysis + +### ๐Ÿ”ง BULK_EXTRACTOR +**Purpose**: Extract strings and artifacts from files +**Syntax**: `bulk_extractor [options] -o [output_dir] [input]` +**Example**: `bulk_extractor -o output_dir malware.exe` +**Output**: Directory with extracted artifacts + +### ๐Ÿ”ง SIGNSRCH +**Purpose**: Find crypto/compression algorithm signatures +**Syntax**: `signsrch [filename]` +**Example**: `signsrch malware.exe` +**Output**: List of detected algorithms + +--- + +## ๐Ÿ“ PE FILE ANALYSIS + +### ๐Ÿ”ง PEFRAME +**Purpose**: Comprehensive PE static analyzer +**Syntax**: `peframe [options] [pe_file]` +**Example**: `peframe -j malware.exe` +**Output**: JSON format analysis report + +### ๐Ÿ”ง PEFILE +**Purpose**: Python PE file analysis library +**Syntax**: `python3 -c "import pefile; pe=pefile.PE('file.exe'); print(pe.dump_info())"` +**Example**: `pefile-dump malware.exe` +**Output**: Detailed PE structure information + +### ๐Ÿ”ง PESCAN +**Purpose**: Scan PE files for suspicious indicators +**Syntax**: `pescan [pe_file]` +**Example**: `pescan malware.exe` +**Output**: Suspicious characteristics report + +### ๐Ÿ”ง MANALYZE +**Purpose**: Advanced PE analyzer with plugins +**Syntax**: `manalyze [options] [pe_file]` +**Example**: `manalyze --plugins=all malware.exe` +**Output**: Comprehensive analysis with threat score + +--- + +## ๐Ÿ“ STRING ANALYSIS & DEOBFUSCATION + +### ๐Ÿ”ง STRINGS +**Purpose**: Extract printable strings from files +**Syntax**: `strings [options] [filename]` +**Example**: `strings -n 6 malware.exe | head -20` +**Output**: List of printable strings + +### ๐Ÿ”ง FLOSS +**Purpose**: FLARE Obfuscated String Solver +**Syntax**: `floss [options] [filename]` +**Example**: `floss --no-static-strings malware.exe` +**Output**: Deobfuscated strings + +### ๐Ÿ”ง BASE64DUMP +**Purpose**: Extract and decode Base64 strings +**Syntax**: `base64dump.py [options] [filename]` +**Example**: `base64dump.py -s malware.txt` +**Output**: Decoded Base64 content + +### ๐Ÿ”ง XORSEARCH +**Purpose**: Search for XOR encoded strings +**Syntax**: `xorsearch [options] [filename] [string]` +**Example**: `xorsearch -W malware.exe kernel32` +**Output**: XOR key and decoded strings + +### ๐Ÿ”ง CYBERCHEF +**Purpose**: Web-based data transformation tool +**Setup**: Start with `cyberchef` command +**Access**: Open browser to http://localhost:8080 +**Usage**: Upload file and apply transformations +**Output**: Decoded/transformed data + +--- + +## ๐Ÿ“ DISASSEMBLERS & DECOMPILERS + +### ๐Ÿ”ง GHIDRA +**Purpose**: NSA reverse engineering framework +**Syntax**: `ghidra` or `ghidraRun` +**Usage**: Import malware.exe and analyze +**Output**: Decompiled C-like code + +### ๐Ÿ”ง RADARE2 +**Purpose**: Command-line reverse engineering framework +**Syntax**: `r2 [options] [filename]` +**Example**: `r2 -A malware.exe` +**Output**: Interactive analysis session + +### ๐Ÿ”ง CUTTER +**Purpose**: GUI for Radare2 +**Syntax**: `cutter [filename]` +**Example**: `cutter malware.exe` +**Output**: Graphical disassembly interface + +### ๐Ÿ”ง OBJDUMP +**Purpose**: GNU object file dumper +**Syntax**: `objdump [options] [filename]` +**Example**: `objdump -d -M intel malware.exe` +**Output**: Disassembled code + +### ๐Ÿ”ง RETDEC-DECOMPILER +**Purpose**: Multi-architecture decompiler +**Syntax**: `retdec-decompiler [options] [filename]` +**Example**: `retdec-decompiler malware.exe` +**Output**: Decompiled C code + +--- + +## ๐Ÿ“ DYNAMIC ANALYSIS & DEBUGGING + +### ๐Ÿ”ง GDB +**Purpose**: GNU debugger for Linux binaries +**Syntax**: `gdb [program]` or `gdb -p [pid]` +**Example**: `gdb ./malware` +**Output**: Interactive debugging session + +### ๐Ÿ”ง STRACE +**Purpose**: Trace system calls +**Syntax**: `strace [options] [command]` +**Example**: `strace -f -e trace=file ./malware` +**Output**: System call trace + +### ๐Ÿ”ง LTRACE +**Purpose**: Trace library calls +**Syntax**: `ltrace [options] [command]` +**Example**: `ltrace ./malware` +**Output**: Library function call trace + +### ๐Ÿ”ง FRIDA +**Purpose**: Dynamic instrumentation toolkit +**Syntax**: `frida [options] [target]` +**Example**: `frida -l script.js malware` +**Output**: Runtime manipulation results + +--- + +## ๐Ÿ“ MEMORY FORENSICS + +### ๐Ÿ”ง VOLATILITY (v2) +**Purpose**: Memory forensics framework +**Syntax**: `vol.py -f [memory_dump] --profile=[profile] [plugin]` +**Example**: `vol.py -f memory.dump --profile=Win7SP1x64 pslist` +**Output**: Process list from memory + +### ๐Ÿ”ง VOLATILITY3 (v3) +**Purpose**: Next-gen memory forensics +**Syntax**: `vol3 -f [memory_dump] [plugin]` +**Example**: `vol3 -f memory.dump windows.pslist` +**Output**: Enhanced process information + +### ๐Ÿ”ง REKALL +**Purpose**: Memory forensics and live analysis +**Syntax**: `rekall -f [memory_dump]` +**Example**: `rekall -f memory.dump` +**Output**: Interactive memory analysis + +--- + +## ๐Ÿ“ DOCUMENT ANALYSIS + +### ๐Ÿ”ง OLEVBA +**Purpose**: Extract VBA macros from Office documents +**Syntax**: `olevba [options] [document]` +**Example**: `olevba --decode malicious.docm` +**Output**: Extracted and decoded macros + +### ๐Ÿ”ง OLEDUMP +**Purpose**: Analyze OLE files and extract objects +**Syntax**: `oledump.py [options] [olefile]` +**Example**: `oledump.py -s 3 -d document.docm` +**Output**: Dumped OLE stream content + +### ๐Ÿ”ง PDFID +**Purpose**: Analyze PDF structure for suspicious elements +**Syntax**: `pdfid.py [options] [pdf_file]` +**Example**: `pdfid.py suspicious.pdf` +**Output**: PDF object statistics + +### ๐Ÿ”ง PEEPDF +**Purpose**: Interactive PDF analysis tool +**Syntax**: `peepdf [options] [pdf_file]` +**Example**: `peepdf -i malicious.pdf` +**Output**: Interactive PDF exploration + +### ๐Ÿ”ง RTFDUMP +**Purpose**: Analyze RTF documents +**Syntax**: `rtfdump.py [options] [rtf_file]` +**Example**: `rtfdump.py document.rtf` +**Output**: RTF structure analysis + +--- + +## ๐Ÿ“ NETWORK ANALYSIS + +### ๐Ÿ”ง WIRESHARK +**Purpose**: Network protocol analyzer GUI +**Syntax**: `wireshark [options] [capture_file]` +**Example**: `wireshark capture.pcap` +**Output**: Graphical packet analysis + +### ๐Ÿ”ง TSHARK +**Purpose**: Command-line Wireshark +**Syntax**: `tshark [options]` +**Example**: `tshark -r capture.pcap -Y http` +**Output**: Filtered packet information + +### ๐Ÿ”ง TCPDUMP +**Purpose**: Capture network packets +**Syntax**: `tcpdump [options] [expression]` +**Example**: `tcpdump -i eth0 -w capture.pcap` +**Output**: Captured network packets + +### ๐Ÿ”ง NGREP +**Purpose**: Network grep for packet payloads +**Syntax**: `ngrep [options] [pattern] [expression]` +**Example**: `ngrep -i 'GET|POST' port 80` +**Output**: Matching packet content + +--- + +## ๐Ÿ“ JAVASCRIPT ANALYSIS + +### ๐Ÿ”ง BOX-JS +**Purpose**: JavaScript sandbox for analysis +**Syntax**: `box-js [javascript_file]` +**Example**: `box-js malicious.js` +**Output**: Execution trace and IOCs + +### ๐Ÿ”ง JS-FILE +**Purpose**: JavaScript file analyzer +**Syntax**: `js-file [javascript_file]` +**Example**: `js-file obfuscated.js` +**Output**: JavaScript analysis report + +--- + +## ๐Ÿ“ ARCHIVE & PACKING + +### ๐Ÿ”ง UPX +**Purpose**: Ultimate Packer for eXecutables +**Syntax**: `upx [options] [files]` +**Example**: `upx -d packed.exe` (unpack) +**Output**: Unpacked executable + +### ๐Ÿ”ง 7Z +**Purpose**: 7-Zip archiver +**Syntax**: `7z [command] [archive] [files]` +**Example**: `7z x archive.7z` +**Output**: Extracted files + +### ๐Ÿ”ง UNZIP +**Purpose**: Extract ZIP archives +**Syntax**: `unzip [options] [zipfile]` +**Example**: `unzip -l archive.zip` +**Output**: Archive contents list + +--- + +## ๐Ÿ“ UTILITIES + +### ๐Ÿ”ง HEXDUMP +**Purpose**: Display file contents in hexadecimal +**Syntax**: `hexdump [options] [file]` +**Example**: `hexdump -C malware.exe | head` +**Output**: Hex and ASCII representation + +### ๐Ÿ”ง XXD +**Purpose**: Hex dump and reverse tool +**Syntax**: `xxd [options] [file]` +**Example**: `xxd malware.exe | head -10` +**Output**: Formatted hex dump + +### ๐Ÿ”ง CUT +**Purpose**: Extract columns from text +**Syntax**: `cut [options] [file]` +**Example**: `strings malware.exe | cut -c1-50` +**Output**: Truncated string output + +--- + +## ๐Ÿš€ Quick Analysis Workflow + +```bash +# 1. Basic file identification +file malware.exe +trid malware.exe +exiftool malware.exe + +# 2. Hash and scan +md5sum malware.exe +ssdeep malware.exe +clamscan malware.exe + +# 3. PE analysis (if Windows executable) +peframe malware.exe +manalyze malware.exe + +# 4. String analysis +strings malware.exe | head -50 +floss malware.exe + +# 5. Pattern matching +yara rules.yar malware.exe + +# 6. Static disassembly +r2 -A malware.exe +# or +ghidra malware.exe + +# 7. Dynamic analysis (Linux binaries only in container) +strace ./linux_malware +ltrace ./linux_malware +``` + +--- + +## ๐Ÿ“‹ Common Command Combinations + +```bash +# Extract and analyze strings +strings malware.exe | grep -i "http\|ftp\|\.exe\|\.dll" | head -20 + +# Search for Base64 encoded content +base64dump.py malware.exe | head -10 + +# Look for XOR patterns +xorsearch -W malware.exe http + +# Bulk artifact extraction +bulk_extractor -o artifacts malware.exe + +# Memory dump analysis +vol.py -f memory.dump --profile=Win7SP1x64 pslist +vol.py -f memory.dump --profile=Win7SP1x64 netscan +``` + +**Total Tools Covered**: 48 essential REMnux tools +**Container**: `remnux/remnux-distro:focal` +**Total Available Tools**: 300+ tools across all categories \ No newline at end of file diff --git a/scripts/add-tool-cheats.sh b/scripts/add-tool-cheats.sh new file mode 100755 index 0000000..905b0e8 --- /dev/null +++ b/scripts/add-tool-cheats.sh @@ -0,0 +1,173 @@ +#!/bin/bash +# Add cheat sheets for fd-find, ripgrep, zsh, fish + +CHEAT_DIR="/opt/cheatsheets/personal" +mkdir -p "$CHEAT_DIR" + +# fd-find cheat sheet +cat > "$CHEAT_DIR/fd" << 'EOF' +# fd (fdfind) - Fast and user-friendly alternative to find + +# Find files by name +fdfind pattern + +# Find in specific directory +fdfind pattern /path/to/search + +# Find only files (no directories) +fdfind -t f pattern + +# Find only directories +fdfind -t d pattern + +# Case-insensitive search +fdfind -i PDF + +# Include hidden and ignored files +fdfind -H pattern + +# Execute command on results +fdfind pattern -x rm + +# Find and show absolute paths +fdfind -a pattern + +# Find files modified in last 24 hours +fdfind -t f --changed-within 24h + +# Find PDF files in current directory +fdfind -e pdf + +# Exclude directories from search +fdfind pattern -E node_modules -E vendor +EOF + +# ripgrep cheat sheet +cat > "$CHEAT_DIR/rg" << 'EOF' +# ripgrep (rg) - Fast recursive grep alternative + +# Basic search +rg pattern + +# Case-insensitive search +rg -i pattern + +# Search specific file types +rg -t py pattern +rg -t js pattern + +# Search in specific files +rg pattern -g '*.pdf' + +# Show context (3 lines before/after) +rg -C 3 pattern + +# Show only filenames with matches +rg -l pattern + +# Count matches per file +rg -c pattern + +# Search for exact word +rg -w word + +# Search hidden files +rg --hidden pattern + +# Search compressed files +rg -z pattern file.gz + +# Find URLs in files +rg 'https?://[^\s]+' + +# Find suspicious JavaScript in PDFs +rg -i 'javascript|openaction' -g '*.pdf' + +# Search with file types excluded +rg pattern --type-not python + +# Multiline search +rg -U 'pattern1.*pattern2' +EOF + +# ripgrep alias +cp "$CHEAT_DIR/rg" "$CHEAT_DIR/ripgrep" + +# zsh cheat sheet +cat > "$CHEAT_DIR/zsh" << 'EOF' +# zsh - Z Shell with powerful features + +# Start zsh +zsh + +# Set zsh as default shell +chsh -s $(which zsh) + +# Autocompletion +# Type command and press TAB + +# History search +# Type start of command and press UP arrow + +# Directory stack +dirs -v # Show directory stack +pushd /path # Push directory and cd +popd # Pop and return to previous + +# Glob patterns +ls **/*.pdf # Recursive glob +ls *.pdf~*.bak # Exclude pattern + +# Aliases (already configured in container) +ll # ls -lah +fd # fdfind +analyse # fhelp +? # fhelp + +# History commands +history # Show all history +history 10 # Show last 10 +!$ # Last argument of previous command +!! # Repeat last command +EOF + +# fish cheat sheet +cat > "$CHEAT_DIR/fish" << 'EOF' +# fish - Friendly Interactive Shell + +# Start fish +fish + +# Set fish as default shell +chsh -s $(which fish) + +# Autocompletion +# Type command and press TAB (shows suggestions automatically) + +# History search +# Type start of command and see suggestions +# Press RIGHT arrow to accept + +# Command abbreviations +abbr --add gco git checkout +abbr --list +abbr --erase gco + +# Functions (instead of aliases) +function ll + ls -lah $argv +end +funcsave ll # Save function permanently + +# Environment variables +set -x PATH /new/path $PATH # Export +set -e VARIABLE # Erase + +# Configuration +fish_config # Open web-based configuration + +# Plugins (fisher) +# curl -sL https://git.io/fisher | source && fisher install jorgebucaran/fisher +EOF + +echo "โœ… Added cheat sheets for fd, ripgrep, zsh, fish" \ No newline at end of file diff --git a/scripts/check-help-coverage.sh b/scripts/check-help-coverage.sh new file mode 100644 index 0000000..849a288 --- /dev/null +++ b/scripts/check-help-coverage.sh @@ -0,0 +1,329 @@ +#!/usr/bin/env bash + +# check-help-coverage.sh +# Check cheat and tldr help coverage for a benchmark list of analysis tools. +# Designed to run inside the container (tabledevil/file-analysis:*). +# - Cheat sheets are expected under /opt/cheatsheets/personal or /opt/cheatsheets/*.cheat +# - TLDR pages are checked under /home/remnux/.local/share/tldr/pages/{common,linux} +# +# Exit codes: +# 0 - ran successfully +# 1 - minor warnings +# 2 - failed to run (e.g., missing directories) + +set -euo pipefail + +CHEAT_DIR_PERSONAL="/opt/cheatsheets/personal" +CHEAT_DIR_GLOBAL="/opt/cheatsheets" +TLDR_DIRS=("/home/remnux/.local/share/tldr/pages/common" "/home/remnux/.local/share/tldr/pages/linux") + +# Resolve a cheat file from a name +resolve_cheat_file() { + local name="$1" + local base + base="$(echo "$name" | sed 's/\.[Pp][Yy]$//')" + + # candidate names to try in order + local candidates=( + "$name" + "$base" + "${base}.py" + "${base// /-}" + "${base// /_}" + "${base//-/_}" + "${base//_/-}" + ) + + for cand in "${candidates[@]}"; do + # personal dir (preferred) + if [[ -f "$CHEAT_DIR_PERSONAL/$cand" ]]; then + echo "$CHEAT_DIR_PERSONAL/$cand"; return 0 + fi + # global .cheat files + if [[ -f "$CHEAT_DIR_GLOBAL/${cand}.cheat" ]]; then + echo "$CHEAT_DIR_GLOBAL/${cand}.cheat"; return 0 + fi + done + return 1 +} + +# Check if any TLDR page exists for a given tool name (try variants) +resolve_tldr_page() { + local name="$1" + local base lower + base="$(echo "$name" | sed 's/\.[Pp][Yy]$//; s/[() ]//g')" + lower="$(echo "$name" | tr '[:upper:]' '[:lower:]')" + + local candidates=( + "$name" + "$lower" + "$base" + "${base}.py" + "${lower// /-}" + "${lower// /_}" + "${lower//-/_}" + "${lower//_/-}" + ) + + for dir in "${TLDR_DIRS[@]}"; do + for cand in "${candidates[@]}"; do + if [[ -f "$dir/${cand}.md" ]]; then + echo "$dir/${cand}.md"; return 0 + fi + done + done + return 1 +} + +# Print a line with fixed columns +print_row() { + printf "%-18s | %-35s | %-6s | %-6s\n" "$1" "$2" "$3" "$4" +} + +# Benchmark list: Category | Display Name | Cheat candidates (comma) | TLDR candidates (comma) +# Keep names short for display; put multiple candidate names to test resolution logic. +TOOL_LIST=$(cat << 'EOF' +General | TrID | trid | trid +General | Magika | magika | magika +General | file | file | file +General | Detect-It-Easy | die,diec | die +General | ExifTool | exiftool | exiftool +General | YARA | yara,yarac | yara +General | signsrch | signsrch | signsrch +General | ssdeep | ssdeep | ssdeep +General | ClamAV | clamscan,freshclam | clamscan,freshclam +General | bulk_extractor | bulk_extractor | bulk_extractor +General | Hachoir | hachoir-metadata,hachoir-subfile | hachoir-metadata +General | Sleuth Kit | fls,icat,tsk_recover | fls,icat +General | binwalk | binwalk | binwalk +General | 7-Zip | 7z,7za,7zr,7zz | 7z +General | wxHexEditor | wxhexeditor | wxhexeditor +General | zipdump | zipdump,zipdump.py | zipdump +General | msitools | msiinfo,msiextract | msiinfo,msiextract +General | numbers-to-string | numbers-to-string,numbers-to-string.py | numbers-to-string +General | re-search | re-search,re-search.py | re-search +General | disitool | disitool,disitool.py | disitool +General | Name-That-Hash | nth | nth +General | Hash ID | hashid,hash-id.py | hashid + +PE | manalyze | manalyze | manalyze +PE | peframe | peframe | peframe +PE | pefile (lib) | pefile | pefile +PE | pescan | pescan,pescan.py | pescan +PE | portex | portex,portex.py | portex +PE | bearcommander | bearcommander | bearcommander +PE | pecheck | pecheck | pecheck + +ELF | pyew | pyew | pyew +ELF | readelf.py | readelf,readelf.py | readelf + +Strings | pestr | pestr | pestr +Strings | bbcrack | bbcrack | bbcrack +Strings | brxor | brxor,brxor.py | brxor +Strings | base64dump | base64dump,base64dump.py | base64dump +Strings | xorsearch | xorsearch | xorsearch +Strings | XORStrings | xorstrings | xorstrings +Strings | flarestrings | flarestrings | flarestrings +Strings | FLOSS | floss | floss +Strings | CyberChef | cyberchef | cyberchef + +Emulation | binee | binee | binee +Emulation | capa | capa | capa +Emulation | vivbin | vivbin | vivbin + +RE | Ghidra | ghidra,analyzeHeadless | ghidra +RE | Cutter | cutter | cutter +RE | Radare2 | r2 | r2 +RE | Rizin | rizin | rizin +RE | objdump | objdump | objdump +RE | RetDec | retdec-decompiler,retdec-decompiler.py | retdec + +Debuggers | edb | edb | edb +Debuggers | gdb | gdb | gdb + +Packers | upx | upx | upx +Packers | bytehist | bytehist | bytehist +Packers | de4dot | de4dot | de4dot + +Android | apktool | apktool | apktool +Android | droidlysis | droidlysis | droidlysis +Android | androgui | androgui,androgui.py | androgui +Android | baksmali | baksmali | baksmali +Android | dex2jar | dex2jar | dex2jar + +Java | CFR | cfr | cfr +Java | Procyon | procyon | procyon +Java | JAD | jad | jad +Java | jd-gui | jd-gui | jd-gui +Java | idx_parser | idx_parser,idx_parser.py | idx_parser + +Python | pyinstxtractor | pyinstxtractor,pyinstxtractor.py | pyinstxtractor +Python | pycdc | pycdc | pycdc + +JavaScript | js | js | js +JavaScript | js-file | js-file | js-file +JavaScript | objects.js | objects.js | objects.js +JavaScript | box-js | box-js | box-js +JavaScript | JSDetox | jsdetox | jsdetox +JavaScript | de4js | de4js | de4js + +PowerShell | pwsh | pwsh | pwsh + +Flash | swfdump | swfdump | swfdump +Flash | flare | flare | flare +Flash | flasm | flasm | flasm +Flash | swf_mastah | swf_mastah,swf_mastah.py | swf_mastah +Flash | xxxswf | xxxswf | xxxswf + +Shellcode | shellcode2exe | shellcode2exe,shellcode2exe.bat | shellcode2exe +Shellcode | scdbg | scdbg | scdbg + +Office | vmonkey | vmonkey | vmonkey +Office | pcodedmp | pcodedmp | pcodedmp +Office | olevba | olevba | olevba +Office | xlmdeobfuscator | xlmdeobfuscator | xlmdeobfuscator +Office | oledump | oledump,oledump.py | oledump +Office | msoffice-crypt | msoffice-crypt | msoffice-crypt +Office | ssview | ssview | ssview + +PDF | pdfid | pdfid,pdfid.py | pdfid, pdfid.py +PDF | pdfparser | pdfparser,pdf-parser.py | pdfparser,pdf-parser.py +PDF | pdfextract | pdfextract | pdfextract +PDF | pdfdecrypt | pdfdecrypt | pdfdecrypt +PDF | peepdf | peepdf | peepdf +PDF | pdftk | pdftk | pdftk +PDF | pdfresurrect | pdfresurrect | pdfresurrect +PDF | qpdf | qpdf | qpdf +PDF | pdfobjflow | pdfobjflow | pdfobjflow + +RTF | rtfobj | rtfobj | rtfobj +RTF | rtfdump | rtfdump,rtfdump.py | rtfdump + +Email | emldump | emldump,emldump.py | emldump +Email | msgconvert | msgconvert | msgconvert + +NetMon | Burp Suite | burpsuite | burpsuite +NetMon | NetworkMiner | networkminer | networkminer +NetMon | polarproxy | polarproxy | polarproxy +NetMon | mitmproxy | mitmproxy | mitmproxy +NetMon | Wireshark | wireshark | wireshark +NetMon | tshark | tshark | tshark +NetMon | ngrep | ngrep | ngrep +NetMon | tcpxtract | tcpxtract | tcpxtract + +Services | Thug | thug | thug +Services | fakedns | fakedns | fakedns +Services | fakemail | fakemail | fakemail +Services | accept-all-ips | accept-all-ips | accept-all-ips +Services | inetsim | inetsim | inetsim +Services | fakenet | fakenet | fakenet + +NetUtils | netcat | nc,netcat | nc +NetUtils | tor | tor | tor +NetUtils | wget | wget | wget +NetUtils | curl | curl | curl +NetUtils | ssh | ssh | ssh +NetUtils | unfurl | unfurl,unfurl_cli.py | unfurl + +Memory | Volatility 2 | vol.py,volatility | volatility +Memory | Volatility 3 | vol3,volatility3 | vol3 +Memory | Rekall | rekall | rekall +Memory | linux_mem_diff | linux_mem_diff,linux_mem_diff.py | linux_mem_diff +Memory | aeskeyfind | aeskeyfind | aeskeyfind +Memory | rsakeyfind | rsakeyfind | rsakeyfind + +Intel | Automater | automater,automater.py | automater +Intel | Shodan | shodan | shodan +Intel | ipwhois_cli | ipwhois_cli,ipwhois_cli.py | ipwhois_cli +Intel | pdnstool | pdnstool | pdnstool +Intel | malwoverview | malwoverview,malwoverview.py | malwoverview +Intel | nsrllookup | nsrllookup | nsrllookup +Intel | vt | vt | vt +Intel | YARA engine | yara | yara +Intel | Viper | viper | viper +Intel | dexray | dexray | dexray +Intel | time-decode | time-decode,time-decode.py | time-decode +Intel | ioc_writer | ioc_writer,ioc_writer.py | ioc_writer + +Behavior | ltrace | ltrace | ltrace +Behavior | strace | strace | strace +Behavior | frida | frida,frida-trace | frida +Behavior | sysdig | sysdig | sysdig +Behavior | unhide | unhide | unhide + +Files | scalpel | scalpel | scalpel +Files | unzip | unzip | unzip +Files | unrar | unrar | unrar +Files | cabextract | cabextract | cabextract + +View | code | code | code +View | scite | scite | scite +View | xpdf | xpdf | xpdf +View | feh | feh | feh +View | convert | convert | convert +View | tesseract | tesseract | tesseract +EOF +) + +# Counters +missing_cheat=0 +missing_tldr=0 +total=0 + +# Header +print_row "Category" "Tool" "CHEAT" "TLDR" +printf -- "%.0s-" $(seq 1 74); echo + +# Iterate tools +while IFS='|' read -r category name cheat_list tldr_list; do + # skip empty/comment lines + [[ -z "${category// }" ]] && continue + + # trim spaces + category="${category//[$'\t\r\n']}"; category="$(echo "$category" | sed 's/^ *//; s/ *$//')" + name="$(echo "$name" | sed 's/^ *//; s/ *$//')" + cheat_list="$(echo "$cheat_list" | sed 's/^ *//; s/ *$//')" + tldr_list="$(echo "$tldr_list" | sed 's/^ *//; s/ *$//')" + + (( total++ )) + + # CHEAT check: iterate candidates until one resolves + cheat_status="MISS" + IFS=',' read -r -a cheat_candidates <<< "$cheat_list" + for c in "${cheat_candidates[@]}"; do + c="$(echo "$c" | sed 's/^ *//; s/ *$//')" + if resolved="$(resolve_cheat_file "$c")"; then + cheat_status="OK" + break + fi + done + + # TLDR check + tldr_status="MISS" + IFS=',' read -r -a tldr_candidates <<< "$tldr_list" + for t in "${tldr_candidates[@]}"; do + t="$(echo "$t" | sed 's/^ *//; s/ *$//')" + if resolved_tldr="$(resolve_tldr_page "$t")"; then + tldr_status="OK" + break + fi + done + + [[ "$cheat_status" == "MISS" ]] && ((missing_cheat++)) + [[ "$tldr_status" == "MISS" ]] && ((missing_tldr++)) + + print_row "$category" "$name" "$cheat_status" "$tldr_status" + +done <<< "$TOOL_LIST" + +printf -- "%.0s-" $(seq 1 74); echo +printf "Total: %d | Missing CHEAT: %d | Missing TLDR: %d\n" "$total" "$missing_cheat" "$missing_tldr" + +# Hints +echo +if [[ "$missing_cheat" -gt 0 || "$missing_tldr" -gt 0 ]]; then + echo "Hints:" + echo "- Add cheat sheets in $CHEAT_DIR_PERSONAL (names without extension recommended)." + echo "- Add TLDR pages in: ${TLDR_DIRS[*]} (as .md)." + echo "- fhelp cheat uses resolution similar to this script (handles .py and hyphen/underscore variants)." +fi diff --git a/scripts/convert-remnux-cheats.py b/scripts/convert-remnux-cheats.py new file mode 100644 index 0000000..b3f894c --- /dev/null +++ b/scripts/convert-remnux-cheats.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 + +""" +convert-remnux-cheats.py + +Parse a consolidated markdown cheatsheet (e.g., remnux-tldr-cheatsheet.md) +and generate cheat sheets and TLDR pages for tools missing from the system. + +Heuristics: +- A tool section is detected by a heading line (## or ###). The heading text + is the tool display name. If parentheses provide aliases, the first alias is + used as the canonical filename (e.g., "7-Zip (7z, 7za)" => 7z). +- Within a section, bullet lines ("- " or "* ") are treated as descriptions. + Inline code `...` on those lines is extracted as commands. +- Fenced code blocks (``` ... ```) are extracted; each non-empty line becomes a command entry. +- TLDR entries: produce simple bullets with the command in backticks and the description when available. +- Cheat entries: print description as a comment line, followed by the command. + +By default, only missing files are created. Existing cheat or TLDR files are left intact +unless --overwrite is provided. +""" + +import argparse +import os +import re +from pathlib import Path + +HDR_RE = re.compile(r"^(#{2,3})\s+(.+?)\s*$") +FENCE_RE = re.compile(r"^```(.*)$") +INLINE_CODE_RE = re.compile(r"`([^`]+)`") + + +def choose_canonical_name(heading: str) -> str: + """From a heading like "7-Zip (7z, 7za, 7zr, 7zz)" choose the first alias if present, + else sanitize the heading text to a plausible command name.""" + text = heading.strip() + # If parentheses include aliases, pick the first alias + m = re.search(r"\(([^)]+)\)", text) + if m: + inner = m.group(1) + # split on comma or whitespace + cand = re.split(r"[\s,]+", inner.strip())[0] + if cand: + return cand + # Otherwise, drop parentheses and after a dash if looks like "Tool - description" + text = re.sub(r"\(.*?\)", "", text) + text = text.split(" - ")[0] + text = text.strip() + # Replace spaces with hyphens, keep .py if present + # Remove characters not typical for command names + safe = re.sub(r"[^a-zA-Z0-9._+-]", "-", text).strip("-") + # Lowercase for TLDR filenames; cheat files can be exact + return safe + + +def parse_sections(lines): + sections = [] # list of dicts: {name, display, desc_cmds:[(desc, cmd)], cmds:[cmd]} + current = None + in_fence = False + fence_accum = [] + + bullet_desc = None # last bullet description without inline code + + for raw in lines: + line = raw.rstrip("\n") + h = HDR_RE.match(line) + if h and not in_fence: + # start new section + if current: + sections.append(current) + hdr_text = h.group(2).strip() + current = { + "display": hdr_text, + "name": choose_canonical_name(hdr_text), + "desc_cmds": [], # list of (desc, cmd) + "cmds": [] # list of commands (from code fences) + } + bullet_desc = None + continue + + # fence handling + f = FENCE_RE.match(line) + if f: + if not in_fence: + in_fence = True + fence_accum = [] + else: + # closing fence: flush accumulated commands + in_fence = False + if current and fence_accum: + for cmdline in fence_accum: + cmdline = cmdline.strip() + if cmdline: + current["cmds"].append(cmdline) + fence_accum = [] + continue + + if in_fence: + fence_accum.append(line) + continue + + # bullets with potential inline code + if re.match(r"^\s*[-*]\s+", line): + # extract inline code first + codes = INLINE_CODE_RE.findall(line) + # remove code spans for a cleaner description + desc = INLINE_CODE_RE.sub("{}", line) + desc = re.sub(r"^\s*[-*]\s+", "", desc).strip() + if current is not None: + if codes: + for code in codes: + current["desc_cmds"].append((desc, code.strip())) + else: + # bullet with no code -> remember as a description context for next code block lines if desired + bullet_desc = desc + continue + + # plain text lines with inline code, treat similarly + codes = INLINE_CODE_RE.findall(line) + if codes and current is not None: + desc = INLINE_CODE_RE.sub("{}", line).strip() + for code in codes: + current["desc_cmds"].append((desc, code.strip())) + continue + + # otherwise ignore + + if current: + sections.append(current) + return sections + + +def ensure_dir(p: Path): + p.mkdir(parents=True, exist_ok=True) + + +def write_cheat(tool_name: str, section, out_dir: Path, overwrite: bool) -> bool: + # cheat file path chosen as plain name without extension (.cheat not required) + filename = tool_name + cheat_path = out_dir / filename + if cheat_path.exists() and not overwrite: + return False + + lines = [] + # Header as comment + lines.append(f"# {section['display']}") + lines.append("") + + # From desc_cmds (description + single command) + for desc, cmd in section.get("desc_cmds", []): + if desc: + lines.append(f"# {desc}") + lines.append(cmd) + lines.append("") + + # From cmds (code fences) + for cmd in section.get("cmds", []): + lines.append(cmd) + lines.append("") + + # Write + cheat_path.write_text("\n".join(lines).rstrip() + "\n", encoding="utf-8") + return True + + +def write_tldr(tool_name: str, section, out_dir: Path, overwrite: bool) -> bool: + # TLDR pages expect lowercase filenames with .md + filename = tool_name.lower() + tldr_path = out_dir / f"{filename}.md" + if tldr_path.exists() and not overwrite: + return False + + md = [] + md.append(f"# {tool_name}") + md.append("") + md.append("> Generated from remnux-tldr-cheatsheet.md. Review for accuracy.") + md.append("") + + # Emit desc_cmds first + for desc, cmd in section.get("desc_cmds", []): + if desc: + md.append(f"- {desc}:") + else: + md.append(f"- Example:") + md.append("") + md.append(f"`{cmd}`") + md.append("") + + # Emit code-fence commands as generic examples + for cmd in section.get("cmds", []): + md.append(f"- Example:") + md.append("") + md.append(f"`{cmd}`") + md.append("") + + tldr_path.write_text("\n".join(md).rstrip() + "\n", encoding="utf-8") + return True + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--in", dest="infile", required=True, help="Path to remnux-tldr-cheatsheet.md") + ap.add_argument("--cheat-dir", required=True, help="Output directory for cheat sheets") + ap.add_argument("--tldr-dir", required=True, help="Output directory for TLDR pages") + ap.add_argument("--only-missing", action="store_true", help="Only create files that don't already exist") + ap.add_argument("--overwrite", action="store_true", help="Overwrite existing files") + args = ap.parse_args() + + if args.overwrite and args.only_missing: + print("[!] --overwrite and --only-missing are mutually exclusive; using --overwrite") + args.only_missing = False + + text = Path(args.infile).read_text(encoding="utf-8", errors="replace").splitlines() + sections = parse_sections(text) + + cheat_dir = Path(args.cheat_dir) + tldr_dir = Path(args.tldr_dir) + ensure_dir(cheat_dir) + ensure_dir(tldr_dir) + + created_cheat = created_tldr = 0 + skipped_cheat = skipped_tldr = 0 + + for sec in sections: + tool_name = sec["name"] + # Write cheat + c_written = write_cheat(tool_name, sec, cheat_dir, overwrite=not args.only_missing) + if c_written: + created_cheat += 1 + else: + skipped_cheat += 1 + # Write tldr + t_written = write_tldr(tool_name, sec, tldr_dir, overwrite=not args.only_missing) + if t_written: + created_tldr += 1 + else: + skipped_tldr += 1 + + print(f"โœ… Conversion complete: cheat created={created_cheat}, skipped={skipped_cheat}; tldr created={created_tldr}, skipped={skipped_tldr}") + + +if __name__ == "__main__": + main() diff --git a/scripts/create-accurate-tools-db.sh b/scripts/create-accurate-tools-db.sh new file mode 100755 index 0000000..19fde60 --- /dev/null +++ b/scripts/create-accurate-tools-db.sh @@ -0,0 +1,112 @@ +#!/bin/bash + +# Create accurate tools database for REMnux-based container +# This script only includes tools that actually exist in the container +# with their correct command names and practical usage examples + +set -e + +DOCS_DIR="/opt/remnux-docs" +TOOLS_DB="/opt/remnux-docs/tools.db" + +echo "๐Ÿ”ง Creating accurate tools database..." + +# Create documentation directory +mkdir -p "$DOCS_DIR" + +# Create the tools database with verified tools +cat > "$TOOLS_DB" << 'EOF' +# REMnux Enhanced Tools Database +# Format: TOOL_NAME|DESCRIPTION|CATEGORY|USAGE_EXAMPLE|AVAILABLE +# Only includes tools verified to be available in container + +# PDF Analysis Tools +pdf-parser.py|Parse and analyze PDF structure and objects|analyze documents pdf|pdf-parser.py suspicious.pdf|โœ“ +pdfid.py|Identify suspicious elements in PDF files|analyze documents pdf|pdfid.py document.pdf|โœ“ +peepdf|Interactive PDF analysis framework with JavaScript detection|analyze documents pdf|peepdf -i malicious.pdf|โœ“ +pdftk|Manipulate PDF files (merge, split, encrypt)|analyze documents pdf|pdftk input.pdf output output.pdf flatten|โœ“ +qpdf|PDF manipulation tool (merge, convert, transform)|analyze documents pdf|qpdf --decrypt encrypted.pdf output.pdf|โœ“ +pdfresurrect|Extract previous versions from PDF files|analyze documents pdf|pdfresurrect document.pdf|โœ“ +pdftool.py|Analyze PDF incremental updates|analyze documents pdf|pdftool.py document.pdf|โœ“ + +# Malware Analysis Tools +capa|Detect malware capabilities using MITRE ATT&CK framework|malware analysis|capa malware.exe|โœ“ +box-js|JavaScript sandbox for malware analysis|malware analysis|box-js --output-dir=/tmp suspicious.js|โœ“ +oletools|Office document analysis suite (oledump.py, rtfdump.py, etc)|malware analysis|oledump.py document.doc|โœ“ +oledump.py|Analyze OLE files (Office documents)|malware analysis|oledump.py document.xls|โœ“ +rtfdump.py|Analyze RTF documents|malware analysis|rtfdump.py document.rtf|โœ“ +emldump.py|Analyze EML email files|malware analysis|emldump.py message.eml|โœ“ +base64dump.py|Extract and decode base64 strings|malware analysis|base64dump.py document.txt|โœ“ +strings|Extract printable strings from files|malware analysis|strings -n 10 binary.exe|โœ“ +hexdump|Display file content in hexadecimal|malware analysis|hexdump -C binary.dat|โœ“ +xxd|Create hex dump or reverse it|malware analysis|xxd binary.exe|โœ“ + +# Data Analysis Tools +visidata|Interactive data exploration and analysis|data analysis|vd data.csv|โœ“ +unfurl|URL analysis and extraction tool|data analysis|echo "http://example.com/path" | unfurl|โœ“ + +# Origami PDF Tools (Ruby-based) +pdfcop|Examine PDF file structure using Origami|analyze documents pdf|ruby -e "require 'origami'; puts Origami::PDF.read('file.pdf').info"|โš  +pdfextract|Extract objects from PDF using Origami|analyze documents pdf|pdfextract document.pdf|โš  +pdfmetadata|Extract PDF metadata using Origami|analyze documents pdf|pdfmetadata document.pdf|โš  + +# File Format Tools +exiftool|Extract metadata from files|general utilities|exiftool document.pdf|โœ“ +file|Determine file type|general utilities|file suspicious.dat|โœ“ +binwalk|Analyze and extract firmware images|malware analysis|binwalk firmware.bin|โœ“ +foremost|File carving tool|forensics|foremost -t pdf,jpg -i disk.img|โœ“ + +# Text Processing +catdoc|Convert MS Word documents to text|view or edit files|catdoc document.doc|โœ“ +docx2txt|Extract text from DOCX files|view or edit files|docx2txt document.docx|โœ“ +unrtf|Convert RTF to other formats|view or edit files|unrtf --html document.rtf|โœ“ + +# Archive Tools +7z|7-Zip archiver with high compression|general utilities|7z x archive.7z|โœ“ +unzip|Extract ZIP archives|general utilities|unzip archive.zip|โœ“ +tar|Archive files|general utilities|tar -xzf archive.tar.gz|โœ“ + +# Network Analysis +tshark|Network protocol analyzer (Wireshark CLI)|network analysis|tshark -r capture.pcap|โœ“ +tcpdump|Command-line packet analyzer|network analysis|tcpdump -r capture.pcap|โœ“ + +# System Tools +mc|Midnight Commander file manager|general utilities|mc|โœ“ +htop|Interactive process viewer|general utilities|htop|โœ“ +tree|Display directory structure as tree|general utilities|tree /path/to/directory|โœ“ + +# Image Processing +convert|ImageMagick convert tool|general utilities|convert input.pdf output.png|โœ“ +identify|ImageMagick identify tool|general utilities|identify image.jpg|โœ“ + +# OCR Tools +tesseract|OCR text extraction|general utilities|tesseract image.png output|โœ“ + +# Container-Added Tools +curl|Transfer data from servers|network analysis|curl -L http://example.com|โœ“ +wget|Download files from web|network analysis|wget http://example.com/file.pdf|โœ“ +jq|JSON processor|data analysis|cat data.json | jq '.field'|โœ“ +yq|YAML processor|data analysis|cat data.yaml | yq '.field'|โœ“ + +# Text Editors +vim|Vi/Vim text editor|view or edit files|vim filename.txt|โœ“ +nano|Simple text editor|view or edit files|nano filename.txt|โœ“ + +# Python Tools Available +python3|Python 3 interpreter|general utilities|python3 script.py|โœ“ +pip3|Python package installer|general utilities|pip3 install package|โœ“ + +EOF + +echo "โœ… Accurate tools database created at: $TOOLS_DB" + +# Count tools +TOOL_COUNT=$(grep -c "^[^#]" "$TOOLS_DB" 2>/dev/null || echo "0") +echo "๐Ÿ”ง Indexed $TOOL_COUNT verified tools" + +echo "" +echo "๐Ÿ“ Notes:" +echo " โœ“ = Tool verified available" +echo " โš  = Tool may need different invocation or setup" +echo "" +echo "Usage: find-tool [search_term] to search through available tools" \ No newline at end of file diff --git a/scripts/create-enhanced-cheatsheets.sh b/scripts/create-enhanced-cheatsheets.sh new file mode 100755 index 0000000..43ebce5 --- /dev/null +++ b/scripts/create-enhanced-cheatsheets.sh @@ -0,0 +1,439 @@ +#!/bin/bash + +# Create enhanced cheat sheets by combining existing ones with file analysis examples +# This leverages existing resources and adds specific malware/PDF analysis use cases + +set -e + +CHEAT_DIR="/opt/cheatsheets" +NAVI_DIR="/opt/navi-cheats" +TLDR_CACHE="/root/.local/share/tldr" + +echo "๐Ÿ“š Creating enhanced cheat sheet collection..." + +# Create directories +mkdir -p "$CHEAT_DIR" "$NAVI_DIR" "$TLDR_CACHE" + +# Function to download and enhance existing cheat sheets +download_and_enhance() { + local tool="$1" + local base_url="https://raw.githubusercontent.com/cheat/cheatsheets/master" + local output_file="$CHEAT_DIR/$tool.cheat" + + echo "Processing $tool..." + + # Try to download existing cheat sheet + if curl -s "$base_url/$tool" -o "/tmp/${tool}_base.cheat" && [ -s "/tmp/${tool}_base.cheat" ]; then + echo "โœ“ Found existing cheat sheet for $tool" + cp "/tmp/${tool}_base.cheat" "$output_file" + else + echo "โš  No existing cheat sheet for $tool, creating from scratch" + echo "# $tool" > "$output_file" + echo "" >> "$output_file" + fi +} + +# Download base cheat sheets for key tools and fix directory structure +TOOLS=("pdftk" "7z" "tar" "unzip" "exiftool") + +# Create proper cheat directory structure (cheat expects files without extensions in subdirs) +mkdir -p "$CHEAT_DIR/personal" + +for tool in "${TOOLS[@]}"; do + download_and_enhance "$tool" + # Copy to proper cheat structure (without .cheat extension) + if [[ -f "$CHEAT_DIR/${tool}.cheat" ]]; then + cp "$CHEAT_DIR/${tool}.cheat" "$CHEAT_DIR/personal/${tool}" + fi +done + +# Enhance pdftk with file analysis specific examples +cat >> "$CHEAT_DIR/pdftk.cheat" << 'EOF' + +# === FILE ANALYSIS ENHANCEMENTS === + +# Flatten PDF to prevent JavaScript execution (SECURITY) +pdftk suspicious.pdf output safe.pdf flatten + +# Remove form fields and JavaScript (SECURITY) +pdftk malicious.pdf output cleaned.pdf flatten dont_ask + +# Extract pages for isolated analysis +pdftk document.pdf cat 1-3 output first_pages.pdf + +# Combine with metadata removal workflow +pdftk input.pdf output temp.pdf flatten +exiftool -all= temp.pdf +mv temp.pdf_original analysis_ready.pdf + +# Decompress PDF streams for analysis +pdftk compressed.pdf output uncompressed.pdf uncompress + +EOF + +# Create comprehensive PDF analysis cheat sheet (doesn't exist in standard repos) +cat > "$CHEAT_DIR/pdf-analysis.cheat" << 'EOF' +--- +tags: [ pdf, malware, analysis, security ] +--- + +# PDF Analysis Workflow + +# Quick PDF overview and suspicious element detection +pdfid.py document.pdf + +# Detailed PDF structure analysis +pdf-parser.py document.pdf + +# Interactive analysis with JavaScript detection +peepdf -i document.pdf + +# Extract metadata +exiftool document.pdf + +# Remove passwords for analysis +qpdf --password=PASSWORD --decrypt encrypted.pdf decrypted.pdf + +# Flatten PDF to remove interactive elements (SECURITY) +pdftk suspicious.pdf output safe.pdf flatten + +# Extract embedded files +pdf-parser.py --extract document.pdf + +# Convert PDF to images for safe viewing +convert document.pdf[0-2] page-%02d.png + +# OCR text from PDF images +convert document.pdf page.png && tesseract page.png output + +# Check for embedded JavaScript +peepdf -s extract_js document.pdf + +# Analyze PDF with Origami (if available) +ruby -e "require 'origami'; pdf = Origami::PDF.read('document.pdf'); puts pdf.each_object.select{|o| o.is_a? Origami::Stream}.count" + +# Extract strings from PDF +strings document.pdf | grep -i "javascript\|openaction\|aa\|js" + +# Hexdump analysis of PDF structure +hexdump -C document.pdf | head -50 + +EOF + +# Create malware analysis workflow cheat sheet +cat > "$CHEAT_DIR/malware-analysis.cheat" << 'EOF' +--- +tags: [ malware, analysis, security, forensics ] +--- + +# Malware Analysis Workflow + +# File type identification +file suspicious.exe + +# Extract readable strings +strings -n 8 malware.bin + +# Detect capabilities with CAPA +capa malware.exe + +# Analyze JavaScript in sandbox +box-js --output-dir=/tmp/js_analysis suspicious.js + +# Office document analysis +oledump.py document.doc + +# RTF document analysis +rtfdump.py document.rtf + +# Email analysis +emldump.py message.eml + +# Extract base64 content +base64dump.py document.txt + +# Binary analysis and extraction +binwalk malware.bin + +# File carving +foremost -t exe,dll,pdf -i disk.img + +# Hex analysis +xxd malware.exe | head -20 + +# Extract metadata from any file +exiftool malware.exe + +# Archive analysis +7z l suspicious.zip +unzip -l suspicious.zip + +# Network capture analysis (if pcap available) +tshark -r capture.pcap -Y "http.request.uri contains .exe" + +EOF + +# Create NAVI cheat sheets (interactive format) +cat > "$NAVI_DIR/pdf-analysis.cheat" << 'EOF' +% pdf analysis + +# Quick PDF info and suspicious elements +pdfid.py + +# Detailed PDF structure analysis +pdf-parser.py + +# Interactive PDF analysis +peepdf -i + +# Flatten PDF for security (remove JavaScript/forms) +pdftk output flatten + +# Decrypt password-protected PDF +qpdf --password= --decrypt + +# Extract PDF metadata +exiftool + +# Convert PDF pages to images +convert [] + +# Extract strings with JavaScript keywords +strings | grep -i "javascript\|openaction\|aa\|js" + +$ pdf_file: find . -name "*.pdf" -type f +$ input_pdf: find . -name "*.pdf" -type f +$ encrypted_pdf: find . -name "*.pdf" -type f +$ page_range: echo "0-2" +$ output_pattern: echo "page-%02d.png" +$ password: echo "password123" +$ output_pdf: echo "output.pdf" + +EOF + +cat > "$NAVI_DIR/malware-analysis.cheat" << 'EOF' +% malware analysis + +# Identify file type +file + +# Extract printable strings +strings -n + +# Detect malware capabilities +capa + +# Analyze JavaScript in sandbox +box-js --output-dir= + +# Analyze Office documents +oledump.py + +# Analyze RTF documents +rtfdump.py + +# Extract and analyze base64 content +base64dump.py + +# Binary analysis and file extraction +binwalk + +# File carving from disk image +foremost -t -i + +# Hex dump analysis +xxd | head - + +$ suspicious_file: find . -type f ! -name "*.txt" ! -name "*.md" +$ binary_file: find . -type f \( -name "*.exe" -o -name "*.dll" -o -name "*.bin" \) +$ executable_file: find . -type f \( -name "*.exe" -o -name "*.dll" \) +$ js_file: find . -name "*.js" -type f +$ office_file: find . -type f \( -name "*.doc" -o -name "*.xls" -o -name "*.ppt" \) +$ rtf_file: find . -name "*.rtf" -type f +$ file_with_base64: find . -name "*.txt" -o -name "*.log" -type f +$ disk_image: find . -name "*.img" -o -name "*.dd" -type f +$ min_length: echo "8" +$ output_dir: echo "/tmp/analysis" +$ file_types: echo "exe,dll,pdf,jpg" +$ num_lines: echo "20" + +EOF + +# Create custom tldr pages for tools missing from standard tldr +mkdir -p "$TLDR_CACHE/pages/common" + +cat > "$TLDR_CACHE/pages/common/pdfid.py.md" << 'EOF' +# pdfid.py + +> Analyze PDF files and identify potentially suspicious elements. +> Part of Didier Stevens' PDF analysis toolkit. +> More information: . + +- Analyze a PDF file for suspicious elements: + +`pdfid.py {{path/to/document.pdf}}` + +- Show detailed analysis including object counts: + +`pdfid.py {{[-v|--verbose]}} {{path/to/document.pdf}}` + +- Analyze all PDF files in a directory: + +`pdfid.py {{path/to/directory/*.pdf}}` + +- Output results in CSV format: + +`pdfid.py {{[-c|--csv]}} {{path/to/document.pdf}}` + +- Scan for specific keywords in PDF: + +`pdfid.py {{[-k|--keyword]}} {{javascript}} {{path/to/document.pdf}}` + +EOF + +cat > "$TLDR_CACHE/pages/common/pdf-parser.py.md" << 'EOF' +# pdf-parser.py + +> Parse and analyze PDF file structure, extract objects and streams. +> Part of Didier Stevens' PDF analysis toolkit. +> More information: . + +- Parse PDF structure and show all objects: + +`pdf-parser.py {{path/to/document.pdf}}` + +- Extract a specific object by number: + +`pdf-parser.py {{[-o|--object]}} {{object_number}} {{path/to/document.pdf}}` + +- Search for objects containing specific text: + +`pdf-parser.py {{[-s|--search]}} {{javascript}} {{path/to/document.pdf}}` + +- Extract and decode streams: + +`pdf-parser.py {{[-f|--filter]}} {{path/to/document.pdf}}` + +- Dump raw object content: + +`pdf-parser.py {{[-d|--dump]}} {{[-o|--object]}} {{object_number}} {{path/to/document.pdf}}` + +- Generate statistics about PDF structure: + +`pdf-parser.py {{[-a|--stats]}} {{path/to/document.pdf}}` + +EOF + +cat > "$TLDR_CACHE/pages/common/peepdf.md" << 'EOF' +# peepdf + +> Interactive PDF analysis framework with JavaScript analysis capabilities. +> More information: . + +- Analyze PDF file interactively: + +`peepdf {{[-i|--interactive]}} {{path/to/document.pdf}}` + +- Analyze PDF and automatically extract suspicious elements: + +`peepdf {{[-f|--force-mode]}} {{path/to/document.pdf}}` + +- Load PDF and execute peepdf script: + +`peepdf {{[-s|--script]}} {{script.txt}} {{path/to/document.pdf}}` + +- Analyze PDF with specific password: + +`peepdf {{[-p|--password]}} {{password}} {{path/to/document.pdf}}` + +- Generate XML report: + +`peepdf {{[-x|--xml]}} {{path/to/document.pdf}}` + +- Update malicious URL database: + +`peepdf {{[-u|--update]}}` + +EOF + +cat > "$TLDR_CACHE/pages/common/capa.md" << 'EOF' +# capa + +> Detect malware capabilities using the MITRE ATT&CK framework. +> Analyzes executables and maps them to threat behaviors. +> More information: . + +- Analyze an executable for capabilities: + +`capa {{path/to/malware.exe}}` + +- Show verbose analysis with detailed explanations: + +`capa {{[-v|--verbose]}} {{path/to/malware.exe}}` + +- Output results in JSON format: + +`capa {{[-j|--json]}} {{path/to/malware.exe}}` + +- Analyze with custom rules directory: + +`capa {{[-r|--rules]}} {{path/to/rules}} {{path/to/malware.exe}}` + +- Show only capabilities above certain confidence: + +`capa {{[-t|--tag]}} {{communication}} {{path/to/malware.exe}}` + +- Analyze shellcode instead of PE file: + +`capa {{[-f|--format]}} {{shellcode}} {{path/to/shellcode.bin}}` + +EOF + +# Configure cheat to use our cheat sheets directory (for remnux user) +mkdir -p /home/remnux/.config/cheat +cat > /home/remnux/.config/cheat/conf.yml << 'EOF' +--- +cheatpaths: + - name: personal + path: /opt/cheatsheets/personal + tags: [personal] + readonly: false +EOF +chown -R remnux:remnux /home/remnux/.config/ + +# Fix navi configuration to prevent fzf preview errors (for remnux user) +mkdir -p /home/remnux/.config/navi +cat > /home/remnux/.config/navi/config.yaml << 'EOF' +cheats: + paths: + - /opt/navi-cheats + - /opt/cheatsheets + +finder: + command: fzf + preview_window: right:50% + +shell: + command: bash +EOF +chown -R remnux:remnux /home/remnux/.config/ + +# Copy custom cheat sheets to proper cheat structure +cp "$CHEAT_DIR/pdf-analysis.cheat" "$CHEAT_DIR/personal/pdf-analysis" 2>/dev/null || true +cp "$CHEAT_DIR/malware-analysis.cheat" "$CHEAT_DIR/personal/malware-analysis" 2>/dev/null || true + +echo "โœ… Enhanced cheat sheet collection created!" +echo "" +echo "๐Ÿ“Š Summary:" +echo " ๐Ÿ“ Cheat sheets: $CHEAT_DIR" +echo " ๐ŸŽฏ Navi cheats: $NAVI_DIR" +echo " ๐Ÿ“š TLDR cache: $TLDR_CACHE" +echo "" +echo "Available cheat sheets:" +ls -1 "$CHEAT_DIR/personal/" 2>/dev/null | sed 's/^/ โ€ข /' || echo " โ€ข (checking directory structure...)" +echo "" +echo "Usage examples:" +echo " cheat pdftk # Show pdftk cheat sheet" +echo " navi # Interactive cheat browser" +echo " tldr pdfid.py # Quick examples for pdfid.py" +echo " fhelp cheat pdf # File analysis PDF workflow" diff --git a/scripts/create-offline-help-system.sh b/scripts/create-offline-help-system.sh new file mode 100755 index 0000000..ac144b1 --- /dev/null +++ b/scripts/create-offline-help-system.sh @@ -0,0 +1,492 @@ +#!/bin/bash + +# Create comprehensive offline help system focused on tldr and cheat +# Removes navi complexity and ensures reliable offline operation + +set -e + +CHEAT_DIR="/opt/cheatsheets" +TLDR_CACHE="/home/remnux/.local/share/tldr" +TOOLS_DB="/opt/remnux-docs/tools.db" + +echo "๐Ÿ“š Creating streamlined offline help system..." + +# Create directories +mkdir -p "$CHEAT_DIR/personal" "$TLDR_CACHE/pages/common" "/opt/remnux-docs" + +# Configure cheat properly for remnux user +mkdir -p /home/remnux/.config/cheat +cat > /home/remnux/.config/cheat/conf.yml << 'EOF' +--- +cheatpaths: + - name: personal + path: /opt/cheatsheets/personal + tags: [personal] + readonly: false +EOF + +# Download existing cheat sheets and place them correctly +echo "๐Ÿ“ฅ Downloading existing cheat sheets..." +TOOLS=("tar" "7z" "unzip" "grep" "find" "awk" "sed") + +for tool in "${TOOLS[@]}"; do + echo " โ€ข Downloading $tool cheat sheet..." + if curl -s "https://raw.githubusercontent.com/cheat/cheatsheets/master/$tool" -o "$CHEAT_DIR/personal/$tool" && [ -s "$CHEAT_DIR/personal/$tool" ]; then + echo " โœ… Downloaded $tool" + else + echo " โš  Failed to download $tool, creating basic version" + echo "# $tool" > "$CHEAT_DIR/personal/$tool" + echo "# Basic usage examples for $tool" >> "$CHEAT_DIR/personal/$tool" + fi +done + +# Create comprehensive cheat sheets for your original analysis tools +echo "๐Ÿ“‹ Creating analysis-focused cheat sheets..." + +# PDF Analysis cheat sheet +cat > "$CHEAT_DIR/personal/pdf-analysis" << 'EOF' +# PDF Analysis Workflow + +# Quick PDF overview and suspicious element detection +pdfid.py document.pdf + +# Detailed PDF structure analysis +pdf-parser.py document.pdf + +# Interactive analysis with JavaScript detection +peepdf -i document.pdf + +# Extract metadata +exiftool document.pdf + +# Remove passwords for analysis +qpdf --password=PASSWORD --decrypt encrypted.pdf decrypted.pdf + +# Flatten PDF to remove interactive elements (SECURITY) +pdftk suspicious.pdf output safe.pdf flatten + +# Extract embedded files +pdf-parser.py --extract document.pdf + +# Convert PDF to images for safe viewing +convert document.pdf[0-2] page-%02d.png + +# OCR text from PDF images +convert document.pdf page.png && tesseract page.png output + +# Check for embedded JavaScript +peepdf -s extract_js document.pdf + +# Extract strings from PDF +strings document.pdf | grep -i "javascript\|openaction\|aa\|js" + +# Hexdump analysis of PDF structure +hexdump -C document.pdf | head -50 +EOF + +# Individual tool cheat sheets +cat > "$CHEAT_DIR/personal/pdfid" << 'EOF' +# pdfid.py - PDF Analysis Tool + +# Basic PDF analysis +pdfid.py document.pdf + +# Verbose output with detailed object counts +pdfid.py -a document.pdf + +# Scan all PDFs in directory +pdfid.py *.pdf + +# Output in CSV format (requires plugins) +pdfid.py -c document.pdf + +# Force analysis of potentially corrupted PDFs +pdfid.py -f document.pdf +EOF + +cat > "$CHEAT_DIR/personal/pdf-parser" << 'EOF' +# pdf-parser.py - PDF Structure Analysis + +# Parse PDF structure +pdf-parser.py document.pdf + +# Extract specific object by number +pdf-parser.py -o 5 document.pdf + +# Search for objects containing text +pdf-parser.py -s javascript document.pdf + +# Extract and decode streams +pdf-parser.py -f document.pdf + +# Dump raw object content +pdf-parser.py -d -o 5 document.pdf + +# Generate statistics +pdf-parser.py -a document.pdf + +# Extract all objects to files +pdf-parser.py --extract document.pdf +EOF + +cat > "$CHEAT_DIR/personal/peepdf" << 'EOF' +# peepdf - Interactive PDF Analysis + +# Interactive analysis mode +peepdf -i document.pdf + +# Force mode for suspicious PDFs +peepdf -f document.pdf + +# Load PDF with script +peepdf -s script.txt document.pdf + +# Analyze with specific password +peepdf -p password document.pdf + +# Generate XML report +peepdf -x document.pdf + +# Update malicious URL database +peepdf -u + +# Check for vulnerabilities +peepdf -C document.pdf +EOF + +cat > "$CHEAT_DIR/personal/pdftk" << 'EOF' +# pdftk - PDF Manipulation + +# Flatten PDF (remove JavaScript/forms) - SECURITY +pdftk suspicious.pdf output safe.pdf flatten + +# Concatenate PDFs +pdftk file1.pdf file2.pdf cat output combined.pdf + +# Extract specific pages +pdftk document.pdf cat 1-3 output pages1-3.pdf + +# Split PDF into single pages +pdftk document.pdf burst + +# Decrypt password-protected PDF +pdftk encrypted.pdf input_pw password output decrypted.pdf + +# Add password to PDF +pdftk document.pdf output protected.pdf user_pw password + +# Decompress PDF streams for analysis +pdftk compressed.pdf output uncompressed.pdf uncompress +EOF + +# OLE document analysis (oledump) +cat > "$CHEAT_DIR/personal/oledump.py" << 'EOF' +# oledump.py - Analyze OLE files (Office documents) + +# List all streams (basic overview) +oledump.py document.doc + +# Dump the content of a specific stream (e.g., 8) to stdout +oledump.py -s 8 -d document.doc > stream8.bin + +# Analyze a specific stream (e.g., 8) in detail +oledump.py -s 8 document.doc +EOF + +# Provide alias without .py for convenience +cp "$CHEAT_DIR/personal/oledump.py" "$CHEAT_DIR/personal/oledump" 2>/dev/null || true + +# Malware analysis cheat sheet +cat > "$CHEAT_DIR/personal/malware-analysis" << 'EOF' +# Malware Analysis Workflow + +# File identification +file suspicious.exe +exiftool suspicious.exe + +# String analysis +strings -n 8 malware.bin +strings -e l malware.bin # little-endian 16-bit +strings -e b malware.bin # big-endian 16-bit + +# Capability detection +capa malware.exe + +# JavaScript analysis +box-js --output-dir=/tmp/js_analysis suspicious.js + +# Office document analysis +oledump.py document.doc +rtfdump.py document.rtf +emldump.py message.eml + +# Base64 content extraction +base64dump.py document.txt + +# Binary analysis +binwalk malware.bin +hexdump -C malware.exe | head -20 + +# File carving +foremost -t exe,dll,pdf -i disk.img +EOF + +cat > "$CHEAT_DIR/personal/capa" << 'EOF' +# capa - Malware Capability Detection + +# Basic capability analysis +capa malware.exe + +# Verbose output with detailed explanations +capa -v malware.exe + +# Output in JSON format +capa -j malware.exe + +# Use custom rules directory +capa -r /path/to/rules malware.exe + +# Show only specific capability tags +capa -t communication malware.exe + +# Analyze shellcode +capa -f shellcode shellcode.bin + +# Analyze with specific architecture +capa -a x64 malware.exe +EOF + +# Create comprehensive TLDR pages +echo "๐Ÿ“– Creating TLDR pages for analysis tools..." + +cat > "$TLDR_CACHE/pages/common/pdfid.py.md" << 'EOF' +# pdfid.py + +> Analyze PDF files and identify potentially suspicious elements. +> Part of Didier Stevens' PDF analysis toolkit. +> More information: . + +- Analyze a PDF file for suspicious elements: + +`pdfid.py {{path/to/document.pdf}}` + +- Show detailed analysis with object counts: + +`pdfid.py {{-a|--all}} {{path/to/document.pdf}}` + +- Analyze all PDF files in current directory: + +`pdfid.py {{*.pdf}}` + +- Output results in CSV format (requires plugins): + +`pdfid.py {{-c|--csv}} {{path/to/document.pdf}}` + +- Force analysis of potentially corrupted PDF: + +`pdfid.py {{-f|--force}} {{path/to/document.pdf}}` +EOF + +cat > "$TLDR_CACHE/pages/common/pdf-parser.py.md" << 'EOF' +# pdf-parser.py + +> Parse and analyze PDF file structure, extract objects and streams. +> Part of Didier Stevens' PDF analysis toolkit. +> More information: . + +- Parse PDF structure and show all objects: + +`pdf-parser.py {{path/to/document.pdf}}` + +- Extract a specific object by number: + +`pdf-parser.py {{-o|--object}} {{object_number}} {{path/to/document.pdf}}` + +- Search for objects containing specific text: + +`pdf-parser.py {{-s|--search}} {{javascript}} {{path/to/document.pdf}}` + +- Extract and decode streams: + +`pdf-parser.py {{-f|--filter}} {{path/to/document.pdf}}` + +- Dump raw object content to file: + +`pdf-parser.py {{-d|--dump}} {{-o|--object}} {{object_number}} {{path/to/document.pdf}}` + +- Generate statistics about PDF structure: + +`pdf-parser.py {{-a|--stats}} {{path/to/document.pdf}}` + +- Extract all objects to separate files: + +`pdf-parser.py {{--extract}} {{path/to/document.pdf}}` +EOF + +cat > "$TLDR_CACHE/pages/common/peepdf.md" << 'EOF' +# peepdf + +> Interactive PDF analysis framework with JavaScript analysis capabilities. +> More information: . + +- Analyze PDF file interactively: + +`peepdf {{-i|--interactive}} {{path/to/document.pdf}}` + +- Analyze PDF and force processing of suspicious elements: + +`peepdf {{-f|--force-mode}} {{path/to/document.pdf}}` + +- Load PDF and execute peepdf script: + +`peepdf {{-s|--script}} {{script.txt}} {{path/to/document.pdf}}` + +- Analyze PDF with specific password: + +`peepdf {{-p|--password}} {{password}} {{path/to/document.pdf}}` + +- Generate XML analysis report: + +`peepdf {{-x|--xml}} {{path/to/document.pdf}}` + +- Check for known vulnerabilities: + +`peepdf {{-C|--check-vulns}} {{path/to/document.pdf}}` + +- Update malicious URL database: + +`peepdf {{-u|--update}}` +EOF + +cat > "$TLDR_CACHE/pages/common/capa.md" << 'EOF' +# capa + +> Detect malware capabilities using the MITRE ATT&CK framework. +> Analyzes executables and maps them to threat behaviors. +> More information: . + +- Analyze an executable for capabilities: + +`capa {{path/to/malware.exe}}` + +- Show verbose analysis with detailed explanations: + +`capa {{-v|--verbose}} {{path/to/malware.exe}}` + +- Output results in JSON format: + +`capa {{-j|--json}} {{path/to/malware.exe}}` + +- Analyze with custom rules directory: + +`capa {{-r|--rules}} {{path/to/rules}} {{path/to/malware.exe}}` + +- Show only capabilities matching specific tag: + +`capa {{-t|--tag}} {{communication}} {{path/to/malware.exe}}` + +- Analyze shellcode instead of PE file: + +`capa {{-f|--format}} {{shellcode}} {{path/to/shellcode.bin}}` + +- Analyze with specific architecture: + +`capa {{-a|--arch}} {{x64}} {{path/to/malware.exe}}` +EOF + +cat > "$TLDR_CACHE/pages/common/box-js.md" << 'EOF' +# box-js + +> JavaScript sandbox for malware analysis and deobfuscation. +> More information: . + +- Analyze JavaScript file in sandbox: + +`box-js {{suspicious.js}}` + +- Analyze with custom output directory: + +`box-js {{--output-dir}} {{/tmp/analysis}} {{suspicious.js}}` + +- Enable verbose logging: + +`box-js {{--verbose}} {{suspicious.js}}` + +- Analyze with timeout (in seconds): + +`box-js {{--timeout}} {{30}} {{suspicious.js}}` + +- Download and analyze JavaScript from URL: + +`box-js {{--download}} {{http://example.com/malicious.js}}` + +- Analyze with custom user agent: + +`box-js {{--user-agent}} {{"Custom Agent"}} {{suspicious.js}}` +EOF + +# Create accurate tools database +cat > "$TOOLS_DB" << 'EOF' +# REMnux Analysis Tools Database +# Format: TOOL_NAME|DESCRIPTION|CATEGORY|USAGE_EXAMPLE + +# PDF Analysis Tools (Your Original Focus) +pdfid.py|Identify suspicious elements in PDF files|pdf analysis|pdfid.py document.pdf +pdf-parser.py|Parse and analyze PDF structure and objects|pdf analysis|pdf-parser.py document.pdf +peepdf|Interactive PDF analysis framework with JavaScript detection|pdf analysis|peepdf -i malicious.pdf +pdftk|Manipulate PDF files (merge, split, flatten)|pdf analysis|pdftk input.pdf output output.pdf flatten +qpdf|PDF manipulation tool (decrypt, merge, convert)|pdf analysis|qpdf --decrypt encrypted.pdf output.pdf + +# Malware Analysis Tools +capa|Detect malware capabilities using MITRE ATT&CK|malware analysis|capa malware.exe +box-js|JavaScript sandbox for malware analysis|malware analysis|box-js --output-dir=/tmp suspicious.js +oledump.py|Analyze OLE files (Office documents)|malware analysis|oledump.py document.xls +rtfdump.py|Analyze RTF documents|malware analysis|rtfdump.py document.rtf +emldump.py|Analyze EML email files|malware analysis|emldump.py message.eml +base64dump.py|Extract and decode base64 strings|malware analysis|base64dump.py document.txt + +# File Analysis Tools +strings|Extract printable strings from files|file analysis|strings -n 10 binary.exe +hexdump|Display file content in hexadecimal|file analysis|hexdump -C binary.dat +file|Determine file type|file analysis|file suspicious.dat +exiftool|Extract metadata from files|file analysis|exiftool document.pdf +binwalk|Analyze and extract firmware/binary images|file analysis|binwalk firmware.bin + +# Archive and Compression +7z|7-Zip archiver with high compression|utilities|7z x archive.7z +unzip|Extract ZIP archives|utilities|unzip archive.zip +tar|Archive files with compression|utilities|tar -xzf archive.tar.gz + +# Text Processing and Search +grep|Search text using patterns|text processing|grep -r "pattern" directory/ +find|Find files and directories|file search|find /path -name "*.pdf" +awk|Text processing and pattern scanning|text processing|awk '{print $1}' file.txt +sed|Stream editor for text manipulation|text processing|sed 's/old/new/g' file.txt +EOF + +# Initialize tldr cache and ensure offline functionality +echo "๐Ÿ”„ Initializing tldr cache for offline use..." +export HOME=/home/remnux +su - remnux -c "tldr --update" 2>/dev/null || echo "โš  TLDR update attempted" + +# Set proper ownership +chown -R remnux:remnux /home/remnux/.config /home/remnux/.local "$CHEAT_DIR" 2>/dev/null || true + +echo "โœ… Streamlined offline help system created!" +echo "" +echo "๐Ÿ“Š Summary:" +echo " ๐Ÿ“ Cheat sheets: $CHEAT_DIR/personal/" +echo " ๐Ÿ“š TLDR cache: $TLDR_CACHE/pages/common/" +echo " ๐Ÿ” Tools database: $TOOLS_DB" +echo "" +echo "Available help:" +echo " fhelp cheat pdf-analysis # PDF analysis workflow" +echo " fhelp cheat pdfid # pdfid.py examples" +echo " tldr pdfid.py # Quick pdfid.py reference" +echo " tldr capa # Quick capa reference" +echo "" + +# Count resources +cheat_count=$(find "$CHEAT_DIR/personal" -type f 2>/dev/null | wc -l) +tldr_count=$(find "$TLDR_CACHE/pages/common" -name "*.md" 2>/dev/null | wc -l) +echo "๐Ÿ“ˆ Resources: $cheat_count cheat sheets, $tldr_count TLDR pages" \ No newline at end of file diff --git a/scripts/demo-help-coverage.sh b/scripts/demo-help-coverage.sh new file mode 100644 index 0000000..2a6cb41 --- /dev/null +++ b/scripts/demo-help-coverage.sh @@ -0,0 +1,64 @@ +#!/bin/bash + +# Simple demo to show current help coverage status + +echo "๐Ÿ“š Current Help System Status" +echo "==============================" +echo "" + +echo "๐Ÿ“‹ Available Cheat Sheets:" +echo "-------------------------" +ls -1 /opt/cheatsheets/personal/ 2>/dev/null | head -20 | sed 's/^/ โœ… /' +echo "" + +echo "๐Ÿ“– Available TLDR Pages (custom):" +echo "---------------------------------" +ls -1 /home/remnux/.local/share/tldr/pages/common/*.md 2>/dev/null | \ + sed 's|.*/||; s|\.md$||' | head -20 | sed 's/^/ โœ… /' +echo "" + +echo "๐Ÿงช Testing fhelp cheat resolution:" +echo "----------------------------------" +echo " Testing: fhelp cheat pdfid" +fhelp cheat pdfid 2>&1 | head -5 | sed 's/^/ /' +echo "" +echo " Testing: fhelp cheat pdfid.py (with .py)" +fhelp cheat pdfid.py 2>&1 | head -5 | sed 's/^/ /' +echo "" +echo " Testing: fhelp cheat oledump" +fhelp cheat oledump 2>&1 | head -5 | sed 's/^/ /' +echo "" + +echo "๐Ÿ“Š Quick Check - Key Tools:" +echo "--------------------------" + +KEY_TOOLS=("pdfid" "pdf-parser" "peepdf" "pdftk" "capa" "oledump" "binwalk" "strings") + +for tool in "${KEY_TOOLS[@]}"; do + cheat_status="โŒ" + tldr_status="โŒ" + + # Check cheat (simple file test) + if [[ -f "/opt/cheatsheets/personal/$tool" ]] || [[ -f "/opt/cheatsheets/personal/${tool}.py" ]]; then + cheat_status="โœ…" + fi + + # Check TLDR (simple file test) + if [[ -f "/home/remnux/.local/share/tldr/pages/common/${tool}.md" ]] || \ + [[ -f "/home/remnux/.local/share/tldr/pages/common/${tool}.py.md" ]]; then + tldr_status="โœ…" + fi + + printf " %-20s CHEAT: %s TLDR: %s\n" "$tool" "$cheat_status" "$tldr_status" +done + +echo "" +echo "โœจ Summary:" +cheat_count=$(ls -1 /opt/cheatsheets/personal/ 2>/dev/null | wc -l) +tldr_count=$(ls -1 /home/remnux/.local/share/tldr/pages/common/*.md 2>/dev/null | wc -l) +echo " ๐Ÿ“ Cheat sheets: $cheat_count" +echo " ๐Ÿ“š TLDR pages: $tldr_count" +echo "" +echo "๐Ÿ’ก To add more tools:" +echo " 1. Place remnux-tldr-cheatsheet.md in /data" +echo " 2. Run: import-remnux-cheatsheets.sh /data/remnux-tldr-cheatsheet.md" \ No newline at end of file diff --git a/scripts/fhelp b/scripts/fhelp new file mode 100755 index 0000000..d41a376 --- /dev/null +++ b/scripts/fhelp @@ -0,0 +1,358 @@ +#!/bin/bash + +# Enhanced File Analysis Help System +# Integrates multiple help sources: custom cheat sheets, navi, tldr, and tool database + +# Color definitions +RED='\033[0;31m' +GREEN='\033[0;32m' +BLUE='\033[0;34m' +YELLOW='\033[1;33m' +CYAN='\033[0;36m' +MAGENTA='\033[0;35m' +NC='\033[0m' + +# Help system paths +TOOLS_DB="/opt/remnux-docs/tools.db" +CHEAT_DIR="/opt/cheatsheets" +TLDR_CACHE="/home/remnux/.local/share/tldr" + +# Resolve cheat file names from a user-provided tool name +# Tries several variants: exact, without .py, with .py, hyphen/underscore alternatives +resolve_cheat_file() { + local name="$1" + local base=$(echo "$name" | sed 's/\.[Pp][Yy]$//') + + # candidates to try in order + local candidates=( + "$name" + "$base" + "${base}.py" + "${base//_/}" + "${base//-/_}" + "${base//_/-}" + ) + + for cand in "${candidates[@]}"; do + if [[ -f "$CHEAT_DIR/personal/$cand" ]]; then + echo "$CHEAT_DIR/personal/$cand" + return 0 + fi + if [[ -f "$CHEAT_DIR/${cand}.cheat" ]]; then + echo "$CHEAT_DIR/${cand}.cheat" + return 0 + fi + done + return 1 +} + +show_main_help() { + echo -e "${CYAN}๐Ÿ“š File Analysis Container Help System${NC}" + echo "======================================" + echo "" + echo -e "${GREEN}๐Ÿ” Tool Discovery:${NC}" + echo " fhelp tools [term] - Search for analysis tools" + echo " fhelp tools --interactive - Browse tools interactively" + echo " fhelp tools --list - List all available tools" + echo "" + echo -e "${GREEN}๐Ÿ“– Command Examples:${NC}" + echo " fhelp cheat - Show cheat sheet for specific tool" + echo " fhelp examples - Browse all command examples interactively" + echo " fhelp quick - Quick examples (tldr style)" + echo "" + echo -e "${GREEN}๐ŸŽฏ Analysis Workflows:${NC}" + echo " fhelp pdf - PDF analysis workflow" + echo " fhelp malware - Malware analysis workflow" + echo " fhelp forensics - System forensics workflow" + echo "" + echo -e "${GREEN}๐Ÿ’ก Quick Access:${NC}" + echo " fhelp --all - Show everything available" + echo " fhelp --offline - Verify offline capabilities" + echo "" + echo -e "${YELLOW}Examples:${NC}" + echo " fhelp tools pdf # Find PDF analysis tools" + echo " fhelp cheat pdfid.py # Show pdfid.py examples" + echo " fhelp quick tar # Quick tar examples" + echo " fhelp examples # Browse all examples" +} + +show_cheat() { + local tool="$1" + + if [[ -z "$tool" ]]; then + echo -e "${RED}โŒ Please specify a tool name${NC}" + echo "Usage: fhelp cheat " + return 1 + fi + + # Check for specific workflow cheat sheets + local cheat_file="" + case "$tool" in + "pdf"|"pdf-analysis") + cheat_file="$CHEAT_DIR/pdf-analysis.cheat" + ;; + "malware"|"malware-analysis") + cheat_file="$CHEAT_DIR/malware-analysis.cheat" + ;; + *) + cheat_file="$CHEAT_DIR/${tool}.cheat" + ;; + esac + + if [[ -f "$cheat_file" ]]; then + echo -e "${CYAN}๐Ÿ“‹ Cheat Sheet: ${YELLOW}$tool${NC}" + echo "=" $(printf '=%.0s' $(seq 1 ${#tool})) + echo "" + # Skip YAML frontmatter if present and print raw content (no ANSI coloring to avoid artifacts) + awk '/^---$/{if(++c==2) start=1; next} start || !/^---$/ && c!=1' "$cheat_file" + else + # Try resolution of common variants (e.g., pdfid.py -> pdfid, oledump -> oledump.py) + local resolved + resolved=$(resolve_cheat_file "$tool") || true + if [[ -n "$resolved" && -f "$resolved" ]]; then + echo -e "${CYAN}๐Ÿ“‹ Cheat Sheet: ${YELLOW}$tool${NC}" + echo "=" $(printf '=%.0s' $(seq 1 ${#tool})) + echo "" + awk '/^---$/{if(++c==2) start=1; next} start || !/^---$/ && c!=1' "$resolved" + else + echo -e "${YELLOW}โš ๏ธ No cheat sheet found for '$tool'${NC}" + echo "" + echo "Available cheat sheets:" + if [[ -d "$CHEAT_DIR/personal" ]]; then + ls -1 "$CHEAT_DIR/personal/" 2>/dev/null | sed 's/^/ โ€ข /' + fi + return 1 + fi + fi +} + +show_quick() { + local command="$1" + + if [[ -z "$command" ]]; then + echo -e "${RED}โŒ Please specify a command name${NC}" + echo "Usage: fhelp quick " + return 1 + fi + + echo -e "${CYAN}๐Ÿ“– Quick examples for: ${YELLOW}$command${NC}" + echo "" + + if command -v tldr >/dev/null 2>&1; then + if ! tldr "$command" 2>/dev/null; then + echo -e "${YELLOW}โš ๏ธ No tldr page found for '$command'${NC}" + echo "Try: fhelp cheat $command" + fi + else + echo -e "${RED}โŒ tldr command not available${NC}" + return 1 + fi +} + +show_tools() { + local search_term="$1" + local option="$2" + + case "$option" in + "--interactive") + if command -v find-tool >/dev/null 2>&1; then + find-tool --interactive + else + echo -e "${RED}โŒ find-tool not available${NC}" + fi + ;; + "--list") + if command -v find-tool >/dev/null 2>&1; then + find-tool --list + else + echo -e "${RED}โŒ find-tool not available${NC}" + fi + ;; + *) + if [[ -z "$search_term" ]]; then + echo -e "${RED}โŒ Please provide a search term${NC}" + echo "Usage: fhelp tools " + echo " fhelp tools --interactive" + echo " fhelp tools --list" + return 1 + fi + + echo -e "${CYAN}๐Ÿ” Searching analysis tools...${NC}" + if command -v find-tool >/dev/null 2>&1; then + find-tool "$search_term" + else + echo -e "${RED}โŒ find-tool not available${NC}" + fi + ;; + esac +} + +show_examples() { + echo -e "${CYAN}๐ŸŽฏ Available Command Examples${NC}" + echo "" + echo "Available cheat sheets:" + if [[ -d "$CHEAT_DIR/personal" ]]; then + echo -e "${GREEN}Cheat sheets (use: fhelp cheat ):${NC}" + ls -1 "$CHEAT_DIR/personal/" 2>/dev/null | sed 's/^/ โ€ข /' + echo "" + fi + + echo "Available TLDR pages:" + if [[ -d "/home/remnux/.local/share/tldr/pages/common" ]]; then + echo -e "${GREEN}TLDR pages (use: tldr ):${NC}" + ls -1 /home/remnux/.local/share/tldr/pages/common/*.md 2>/dev/null | sed 's|.*/||; s|\.md$||' | sed 's/^/ โ€ข /' + fi +} + +show_offline_status() { + echo -e "${CYAN}๐Ÿ”Œ Offline Capability Check${NC}" + echo "===========================" + echo "" + echo "Documentation Tools:" + + local tools=("find-tool" "cheat" "tldr") + for tool in "${tools[@]}"; do + if command -v "$tool" >/dev/null 2>&1; then + echo -e " ${GREEN}โœ… $tool - available${NC}" + else + echo -e " ${RED}โŒ $tool - missing${NC}" + fi + done + + echo "" + echo "Documentation Files:" + + local files=("$TOOLS_DB" "$CHEAT_DIR/pdf-analysis.cheat" "$CHEAT_DIR/malware-analysis.cheat") + local file_names=("/opt/remnux-docs/tools.db" "/opt/cheatsheets/pdf-analysis.cheat" "/opt/cheatsheets/malware-analysis.cheat") + + for i in "${!files[@]}"; do + if [[ -f "${files[$i]}" ]]; then + echo -e " ${GREEN}โœ… ${file_names[$i]} - available${NC}" + else + echo -e " ${RED}โŒ ${file_names[$i]} - missing${NC}" + fi + done + + # Count available cheat sheets + if [[ -d "$CHEAT_DIR" ]]; then + local cheat_count=$(ls -1 "$CHEAT_DIR"/*.cheat 2>/dev/null | wc -l) + echo "" + echo -e "${CYAN}๐Ÿ“Š $cheat_count cheat sheets available${NC}" + fi + + echo "" + echo -e "${GREEN}๐ŸŽ‰ Offline help system ready!${NC}" +} + +show_pdf_workflow() { + echo -e "${CYAN}๐Ÿ“„ PDF Analysis Workflow${NC}" + echo "========================" + echo "" + echo -e "${GREEN}Step 1: Initial Analysis${NC}" + echo " pdfid.py document.pdf # Quick overview" + echo " file document.pdf # File type check" + echo "" + echo -e "${GREEN}Step 2: Detailed Analysis${NC}" + echo " pdf-parser.py document.pdf # Structure analysis" + echo " peepdf -i document.pdf # Interactive analysis" + echo "" + echo -e "${GREEN}Step 3: Security Measures${NC}" + echo " pdftk document.pdf output safe.pdf flatten # Remove JavaScript" + echo " qpdf --decrypt encrypted.pdf decrypted.pdf # Remove password" + echo "" + echo -e "${GREEN}Step 4: Deep Inspection${NC}" + echo " strings document.pdf | grep -i javascript # Find suspicious strings" + echo " exiftool document.pdf # Extract metadata" + echo " convert document.pdf[0] preview.png # Safe preview" + echo "" + echo -e "${YELLOW}For more examples: fhelp cheat pdf${NC}" +} + +show_malware_workflow() { + echo -e "${CYAN}๐Ÿฆ  Malware Analysis Workflow${NC}" + echo "============================" + echo "" + echo -e "${GREEN}Step 1: File Identification${NC}" + echo " file suspicious.exe # Identify file type" + echo " exiftool suspicious.exe # Extract metadata" + echo "" + echo -e "${GREEN}Step 2: Static Analysis${NC}" + echo " strings -n 8 malware.bin # Extract strings" + echo " capa malware.exe # Detect capabilities" + echo " binwalk malware.bin # Analyze binary structure" + echo "" + echo -e "${GREEN}Step 3: Document Analysis${NC}" + echo " oledump.py document.doc # Office documents" + echo " rtfdump.py document.rtf # RTF documents" + echo " box-js suspicious.js # JavaScript sandbox" + echo "" + echo -e "${GREEN}Step 4: Data Extraction${NC}" + echo " base64dump.py encoded.txt # Base64 content" + echo " foremost -t exe,dll -i image.dd # File carving" + echo "" + echo -e "${YELLOW}For more examples: fhelp cheat malware${NC}" +} + +show_all() { + echo -e "${CYAN}๐Ÿ” Complete Help System Overview${NC}" + echo "=================================" + echo "" + + show_tools "analysis" + echo "" + echo -e "${CYAN}Available Workflows:${NC}" + echo " โ€ข PDF Analysis (fhelp pdf)" + echo " โ€ข Malware Analysis (fhelp malware)" + echo "" + + if [[ -d "$CHEAT_DIR" ]]; then + echo -e "${CYAN}Available Cheat Sheets:${NC}" + ls -1 "$CHEAT_DIR"/*.cheat 2>/dev/null | sed 's|.*/||; s|\.cheat$||' | sed 's/^/ โ€ข /' + echo "" + fi + + show_offline_status +} + +# Main command parsing +case "${1:-}" in + "tools") + shift + show_tools "$@" + ;; + "cheat") + shift + show_cheat "$@" + ;; + "quick") + shift + show_quick "$@" + ;; + "examples") + show_examples + ;; + "pdf") + show_pdf_workflow + ;; + "malware") + show_malware_workflow + ;; + "forensics") + echo -e "${YELLOW}โš ๏ธ Forensics workflow not yet implemented${NC}" + echo "Try: fhelp malware or fhelp pdf" + ;; + "--offline") + show_offline_status + ;; + "--all") + show_all + ;; + "--help"|"-h"|"help"|"") + show_main_help + ;; + *) + echo -e "${RED}Unknown option: $1${NC}" + echo "" + show_main_help + ;; +esac \ No newline at end of file diff --git a/scripts/find-tool b/scripts/find-tool index 2cfbe48..b247f01 100755 --- a/scripts/find-tool +++ b/scripts/find-tool @@ -68,7 +68,7 @@ search_tools() { echo -e "${CYAN}๐Ÿ” Searching for: ${YELLOW}$search_term${NC}" echo "" - # Search in tool names, descriptions, categories, and usage hints + # Search in tool names, descriptions, categories, and usage hints (5 fields now) local matches=$(awk -F'|' -v term="$search_term" ' /^[^#]/ && (tolower($1) ~ tolower(term) || tolower($2) ~ tolower(term) || tolower($3) ~ tolower(term) || tolower($4) ~ tolower(term)) { print $0 @@ -82,17 +82,17 @@ search_tools() { fi local count=0 - echo "$matches" | while IFS='|' read -r name description category usage; do + echo "$matches" | while IFS='|' read -r name description category usage available; do count=$((count + 1)) echo -e "${GREEN}๐Ÿ”ง $name${NC}" - echo -e " ${BLUE}Description:${NC} $description" - echo -e " ${BLUE}Category:${NC} $category" + echo -e " Description: $description" + echo -e " Category: $category" if [[ -n "$usage" && "$usage" != " " ]]; then - echo -e " ${BLUE}Usage:${NC} $usage" + echo -e " Usage: $usage" fi - # Check if tool exists on system - if command -v "$name" >/dev/null 2>&1; then + # Use availability status from database instead of checking command + if [[ "$available" == "โœ“" ]]; then echo -e " ${GREEN}โœ“ Available${NC}" else echo -e " ${YELLOW}โš  Check path or alternative name${NC}" diff --git a/scripts/import-remnux-cheatsheets.sh b/scripts/import-remnux-cheatsheets.sh new file mode 100644 index 0000000..923cddf --- /dev/null +++ b/scripts/import-remnux-cheatsheets.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash + +# import-remnux-cheatsheets.sh +# Wrapper to import a consolidated markdown cheatsheet (remnux-tldr-cheatsheet.md) +# and generate missing cheat sheets (/opt/cheatsheets/personal/) and +# tldr pages (/home/remnux/.local/share/tldr/pages/common/.md) inside the container. +# +# Usage (inside container): +# import-remnux-cheatsheets.sh /data/remnux-tldr-cheatsheet.md +# Or from host: +# docker run --rm -v "$(pwd):/data" tabledevil/file-analysis:streamlined \ +# bash -lc "/usr/local/bin/import-remnux-cheatsheets.sh /data/remnux-tldr-cheatsheet.md && fhelp examples" + +set -euo pipefail + +SCRIPT_DIR="/usr/local/bin" +CONVERTER="$SCRIPT_DIR/convert-remnux-cheats.py" + +if [[ $# -lt 1 ]]; then + echo "Usage: $0 [--dry-run]" >&2 + exit 2 +fi + +INPUT_MD="$1"; shift || true +EXTRA_ARGS=("$@") + +if [[ ! -f "$INPUT_MD" ]]; then + echo "Input file not found: $INPUT_MD" >&2 + exit 2 +fi + +# Ensure directories exist +mkdir -p /opt/cheatsheets/personal +mkdir -p /home/remnux/.local/share/tldr/pages/common + +# Run the converter +python3 "$CONVERTER" --in "$INPUT_MD" \ + --cheat-dir /opt/cheatsheets/personal \ + --tldr-dir /home/remnux/.local/share/tldr/pages/common \ + --only-missing "${EXTRA_ARGS[@]}" + +# Fix ownership for remnux user +chown -R remnux:remnux /opt/cheatsheets /home/remnux/.local/share/tldr || true + +echo "โœ… Import complete. Try: fhelp examples" \ No newline at end of file diff --git a/test-help-system.sh b/test-help-system.sh new file mode 100755 index 0000000..a247594 --- /dev/null +++ b/test-help-system.sh @@ -0,0 +1,164 @@ +#!/bin/bash + +# Test script for enhanced help system +# This validates that all help components work correctly + +echo "๐Ÿงช Testing Enhanced Help System" +echo "===============================" +echo "" + +# Build the container first +echo "๐Ÿ—๏ธ Building container with enhanced help system..." +docker build -f Dockerfile.remnux -t tabledevil/file-analysis:enhanced . || { + echo "โŒ Container build failed!" + exit 1 +} + +echo "โœ… Container built successfully" +echo "" + +# Test basic help functions +echo "๐Ÿ“‹ Testing basic help functions..." + +# Test 1: Main help +echo " โ€ข Testing main help..." +docker run --rm tabledevil/file-analysis:enhanced fhelp | head -10 | grep -q "File Analysis Container Help System" && { + echo " โœ… Main help works" +} || { + echo " โŒ Main help failed" +} + +# Test 2: Tools search +echo " โ€ข Testing tools search..." +docker run --rm tabledevil/file-analysis:enhanced fhelp tools pdf 2>/dev/null | head -20 | grep -q "PDF" && { + echo " โœ… Tools search works" +} || { + echo " โŒ Tools search failed" +} + +# Test 3: PDF workflow +echo " โ€ข Testing PDF workflow..." +docker run --rm tabledevil/file-analysis:enhanced fhelp pdf | grep -q "pdfid.py" && { + echo " โœ… PDF workflow works" +} || { + echo " โŒ PDF workflow failed" +} + +# Test 4: Malware workflow +echo " โ€ข Testing malware workflow..." +docker run --rm tabledevil/file-analysis:enhanced fhelp malware | grep -q "capa" && { + echo " โœ… Malware workflow works" +} || { + echo " โŒ Malware workflow failed" +} + +# Test 5: Cheat sheets +echo " โ€ข Testing cheat sheets..." +docker run --rm tabledevil/file-analysis:enhanced fhelp cheat pdf | grep -q "pdfid.py" && { + echo " โœ… Cheat sheets work" +} || { + echo " โŒ Cheat sheets failed" +} + +# Test 6: Quick examples (tldr) +echo " โ€ข Testing tldr integration..." +docker run --rm tabledevil/file-analysis:enhanced fhelp quick tar 2>/dev/null | head -20 | grep -q "tar" && { + echo " โœ… TLDR integration works" +} || { + echo " โŒ TLDR integration failed" +} + +# Test 7: Offline status +echo " โ€ข Testing offline status check..." +docker run --rm tabledevil/file-analysis:enhanced fhelp --offline | grep -q "Offline help system ready" && { + echo " โœ… Offline status works" +} || { + echo " โŒ Offline status failed" +} + +echo "" + +# Test individual help tools +echo "๐Ÿ”ง Testing individual help tools..." + +# Test find-tool +echo " โ€ข Testing find-tool..." +docker run --rm tabledevil/file-analysis:enhanced find-tool pdf 2>/dev/null | head -20 | grep -q "PDF" && { + echo " โœ… find-tool works" +} || { + echo " โŒ find-tool failed" +} + +# Test fhelp cheat command (internal implementation) +echo " โ€ข Testing fhelp cheat command..." +docker run --rm tabledevil/file-analysis:enhanced fhelp cheat tar 2>/dev/null | head -20 | grep -q "tar" && { + echo " โœ… fhelp cheat command works" +} || { + echo " โŒ fhelp cheat command failed" +} + +# Test tldr command +echo " โ€ข Testing tldr command..." +docker run --rm tabledevil/file-analysis:enhanced tldr tar 2>/dev/null | head -20 | grep -q "tar" && { + echo " โœ… tldr command works" +} || { + echo " โŒ tldr command failed" +} + +# Test navi (just check if it exists and doesn't crash immediately) +echo " โ€ข Testing navi availability..." +docker run --rm tabledevil/file-analysis:enhanced bash -c "command -v navi && echo 'navi available'" | grep -q "navi available" && { + echo " โœ… navi is available" +} || { + echo " โŒ navi not available" +} + +echo "" + +# Test actual analysis tools mentioned in help +echo "๐Ÿ› ๏ธ Testing analysis tools availability..." + +TOOLS=("pdfid.py" "pdf-parser.py" "peepdf" "pdftk" "capa" "strings" "exiftool" "file") + +for tool in "${TOOLS[@]}"; do + echo " โ€ข Testing $tool..." + docker run --rm tabledevil/file-analysis:enhanced bash -c "command -v '$tool' >/dev/null 2>&1" && { + echo " โœ… $tool available" + } || { + echo " โŒ $tool not found" + } +done + +echo "" + +# Test file structure +echo "๐Ÿ“ Testing help system file structure..." + +FILES=( + "/opt/remnux-docs/tools.db" + "/opt/cheatsheets/pdf-analysis.cheat" + "/opt/cheatsheets/malware-analysis.cheat" + "/opt/navi-cheats/pdf-analysis.cheat" + "/usr/local/bin/fhelp" + "/usr/local/bin/find-tool" +) + +for file in "${FILES[@]}"; do + echo " โ€ข Testing $file..." + docker run --rm tabledevil/file-analysis:enhanced test -f "$file" && { + echo " โœ… $file exists" + } || { + echo " โŒ $file missing" + } +done + +echo "" + +# Interactive test (requires manual validation) +echo "๐ŸŽฏ Manual Test Available:" +echo " Run: docker run -it tabledevil/file-analysis:enhanced" +echo " Then try: fhelp, fhelp tools pdf, fhelp examples, etc." +echo "" + +echo "๐Ÿ Help system testing complete!" +echo " Review the results above to ensure all components work correctly." \ No newline at end of file diff --git a/test-streamlined-help.sh b/test-streamlined-help.sh new file mode 100755 index 0000000..3e55263 --- /dev/null +++ b/test-streamlined-help.sh @@ -0,0 +1,166 @@ +#!/bin/bash + +# Test script for streamlined help system (tldr + cheat, no navi) + +echo "๐Ÿงช Testing Streamlined Help System (TLDR + Cheat)" +echo "=================================================" +echo "" + +# Build the container +echo "๐Ÿ—๏ธ Building streamlined container..." +docker build -f Dockerfile.remnux -t tabledevil/file-analysis:streamlined . || { + echo "โŒ Container build failed!" + exit 1 +} + +echo "โœ… Container built successfully" +echo "" + +# Test basic help functions +echo "๐Ÿ“‹ Testing basic help functions..." + +# Test 1: Main help +echo " โ€ข Testing main help..." +docker run --rm tabledevil/file-analysis:streamlined fhelp | head -10 | grep -q "File Analysis Container Help System" && { + echo " โœ… Main help works" +} || { + echo " โŒ Main help failed" +} + +# Test 2: PDF workflow +echo " โ€ข Testing PDF workflow..." +docker run --rm tabledevil/file-analysis:streamlined fhelp pdf | grep -q "pdfid.py" && { + echo " โœ… PDF workflow works" +} || { + echo " โŒ PDF workflow failed" +} + +# Test 3: Malware workflow +echo " โ€ข Testing malware workflow..." +docker run --rm tabledevil/file-analysis:streamlined fhelp malware | grep -q "capa" && { + echo " โœ… Malware workflow works" +} || { + echo " โŒ Malware workflow failed" +} + +# Test 4: Examples listing +echo " โ€ข Testing examples listing..." +docker run --rm tabledevil/file-analysis:streamlined fhelp examples | grep -q "pdfid" && { + echo " โœ… Examples listing works" +} || { + echo " โŒ Examples listing failed" +} + +echo "" + +# Test fhelp cheat functionality +echo "๐Ÿ“‹ Testing fhelp cheat sheets..." + +CHEAT_TOOLS=("pdfid" "pdf-parser" "pdftk" "capa" "tar") + +for tool in "${CHEAT_TOOLS[@]}"; do + echo " โ€ข Testing fhelp cheat $tool..." + docker run --rm tabledevil/file-analysis:streamlined fhelp cheat "$tool" 2>/dev/null | head -5 | grep -q "Cheat Sheet" && { + echo " โœ… fhelp cheat $tool works" + } || { + echo " โŒ fhelp cheat $tool failed" + } +done + +echo "" + +# Test TLDR functionality +echo "๐Ÿ“– Testing TLDR functionality..." + +echo " โ€ข Testing standard tldr (tar)..." +docker run --rm tabledevil/file-analysis:streamlined bash -c "export HOME=/home/remnux && tldr tar" 2>/dev/null | head -5 | grep -q "tar" && { + echo " โœ… Standard tldr works" +} || { + echo " โŒ Standard tldr failed" +} + +# Test custom analysis tools TLDR pages +TLDR_TOOLS=("pdfid.py" "capa" "peepdf") + +for tool in "${TLDR_TOOLS[@]}"; do + echo " โ€ข Testing tldr $tool..." + docker run --rm tabledevil/file-analysis:streamlined bash -c "export HOME=/home/remnux && tldr '$tool'" 2>/dev/null | head -5 | grep -q "$tool" && { + echo " โœ… tldr $tool works" + } || { + echo " โŒ tldr $tool failed" + } +done + +echo "" + +# Test tool availability +echo "๐Ÿ› ๏ธ Testing analysis tools availability..." + +TOOLS=("pdfid.py" "pdf-parser.py" "peepdf" "pdftk" "capa" "strings" "exiftool" "file") + +for tool in "${TOOLS[@]}"; do + echo " โ€ข Testing $tool availability..." + docker run --rm tabledevil/file-analysis:streamlined bash -c "command -v '$tool' >/dev/null 2>&1" && { + echo " โœ… $tool available" + } || { + echo " โŒ $tool not found" + } +done + +echo "" + +# Test file structure +echo "๐Ÿ“ Testing help system file structure..." + +FILES=( + "/opt/remnux-docs/tools.db" + "/opt/cheatsheets/personal/pdfid" + "/opt/cheatsheets/personal/pdf-analysis" + "/home/remnux/.local/share/tldr/pages/common/pdfid.py.md" + "/usr/local/bin/fhelp" + "/usr/local/bin/find-tool" +) + +for file in "${FILES[@]}"; do + echo " โ€ข Testing $file..." + docker run --rm tabledevil/file-analysis:streamlined test -f "$file" && { + echo " โœ… $file exists" + } || { + echo " โŒ $file missing" + } +done + +echo "" + +# Test offline capabilities +echo "๐Ÿ”Œ Testing offline capabilities..." + +echo " โ€ข Testing fhelp --offline..." +docker run --rm tabledevil/file-analysis:streamlined fhelp --offline | grep -q "Offline help system ready" && { + echo " โœ… Offline status check works" +} || { + echo " โŒ Offline status check failed" +} + +echo "" + +# Summary +echo "๐Ÿ“Š Resource Summary:" +docker run --rm tabledevil/file-analysis:streamlined bash -c " +echo ' Cheat sheets:' \$(find /opt/cheatsheets/personal -type f 2>/dev/null | wc -l) +echo ' TLDR pages:' \$(find /home/remnux/.local/share/tldr/pages/common -name '*.md' 2>/dev/null | wc -l) +echo ' Analysis tools in PATH:' \$(echo 'pdfid.py pdf-parser.py peepdf pdftk capa' | tr ' ' '\n' | xargs -I {} sh -c 'command -v {} >/dev/null && echo found' | wc -l) +" + +echo "" +echo "๐ŸŽฏ Interactive Test Available:" +echo " Run: docker run -it tabledevil/file-analysis:streamlined" +echo " Then try:" +echo " fhelp # Main help" +echo " fhelp cheat pdfid # pdfid.py cheat sheet" +echo " tldr capa # capa quick reference (if working)" +echo " fhelp examples # List all available help" +echo "" + +echo "๐Ÿ Streamlined help system testing complete!" +echo " Review results above - navi removed, focusing on tldr + cheat reliability" \ No newline at end of file