Major repository cleanup and enhancement

- Reorganize documentation: moved old docs to docs/ directory
- Add comprehensive README.md with build options and usage guide
- Add detailed CONTRIBUTING.md with help content management guide
- Create Makefile for automated building and testing
- Add Dockerfile.scratch for building from Ubuntu 20.04 base
- Enhance all Dockerfiles with PowerShell + PSScriptAnalyzer
- Add modern shells: zsh (with plugins) and fish (with config)
- Add modern CLI tools: fd-find, ripgrep, fzf
- Create comprehensive help system with cheat/TLDR/fish completions
- Add helper scripts for help content management and coverage checking
- Fix Dockerfile.remnux script references
- Support three build variants: upstream (REMnux), scratch (Ubuntu), kali

Build options:
  - make build-upstream: Fast, uses REMnux upstream (recommended)
  - make build-scratch: Full control, builds from Ubuntu 20.04
  - make build-kali: Legacy Kali Linux base

Features:
  - PowerShell with PSScriptAnalyzer module
  - Modern shells (zsh, fish) with custom configurations
  - Enhanced help system (cheat sheets, TLDR pages, fish completions)
  - Help coverage checking and bulk import tools
  - Comprehensive documentation for users and contributors
This commit is contained in:
Tobias Kessels
2025-10-01 11:45:56 +02:00
parent 6bfcfd7935
commit b98aaee3e0
27 changed files with 5000 additions and 62 deletions

146
docs/HELP_SYSTEM.md Normal file
View File

@@ -0,0 +1,146 @@
# File Analysis Container - Help System Guide
The enhanced container includes a comprehensive offline help system for malware and file analysis.
## 🎯 Main Help Commands
### Core Help System
```bash
fhelp # Complete help system overview
fhelp --offline # Verify all offline capabilities work
fhelp --all # Show everything available
```
### Tool Discovery (Fuzzy Search)
```bash
fhelp tools pdf # Find all PDF-related tools
fhelp tools malware # Find malware analysis tools
fhelp tools javascript # Find JavaScript analysis tools
fhelp tools --interactive # Browse all tools with fzf
fhelp tools --list # List all available tools
fhelp tools --categories # Show tool categories
```
### Command Examples & Cheat Sheets
```bash
fhelp cheat pdf # PDF analysis commands
fhelp cheat malware # Malware analysis commands
fhelp cheat system # System utilities commands
fhelp cheat pdfid.py # Specific tool examples
fhelp examples # Browse all examples interactively (navi)
```
### Analysis Workflows
```bash
fhelp pdf # Complete PDF analysis workflow
fhelp malware # Malware analysis workflow
fhelp forensics # System forensics workflow
```
### Quick Examples (tldr style)
```bash
fhelp quick tar # Quick tar examples
fhelp quick grep # Quick grep examples
fhelp quick find # Quick find examples
```
## 🛠️ Direct Tool Access
### Interactive Cheat Sheet Browser
```bash
navi --fzf # Browse all cheat sheets interactively
navi --print # Print cheat sheet for current directory
```
### Standard Help Tools
```bash
cheat tar # Show tar cheat sheet
tldr grep # Quick grep examples
find-tool pdf # Direct tool search (same as fhelp tools)
```
## 📚 Available Cheat Sheet Categories
- **PDF Analysis** - pdfid.py, pdf-parser.py, peepdf, pdftk, origami tools
- **Malware Analysis** - capa, box-js, oledump.py, unfurl, visidata
- **System Utilities** - 7z, mc, grep, find, file operations
## 🎨 Example Usage Sessions
### Find PDF Tools
```bash
remnux@container:/data$ fhelp tools pdf
🔍 Searching for: pdf
🔧 peepdf
Description: Examine elements of the PDF file.
Category: analyze documents pdf
✓ Available
🔧 pdfid.py
Description: Identify suspicious elements of the PDF file.
✓ Available
```
### Get PDF Analysis Commands
```bash
remnux@container:/data$ fhelp cheat pdf
📄 PDF Analysis Commands:
pdfid.py <pdf_file>
pdf-parser.py <pdf_file>
peepdf -i <pdf_file>
pdftk <pdf_file> cat output flattened_<pdf_file>
```
### View Analysis Workflow
```bash
remnux@container:/data$ fhelp pdf
📄 PDF Analysis Workflow:
1. Initial Analysis:
pdfid.py sample.pdf # Quick overview
file sample.pdf # Verify file type
2. Deep Analysis:
pdf-parser.py sample.pdf # Structure analysis
peepdf -i sample.pdf # Interactive analysis
```
## 🔧 Troubleshooting
### If `fhelp` doesn't work:
```bash
# Use the full path
/usr/local/bin/fhelp
# Check if it's installed
which fhelp
ls -la /usr/local/bin/fhelp
```
### If tools are missing:
```bash
# Check offline capabilities
fhelp --offline
# Verify tool database
ls -la /opt/remnux-docs/tools.db
```
### Alternative Commands
- Use `find-tool` instead of `fhelp tools`
- Use `navi --fzf` for interactive browsing
- Use `cheat` or `tldr` for standard command help
## ⚡ Quick Reference
| Command | Purpose |
|---------|---------|
| `fhelp` | Main help system |
| `fhelp tools [term]` | Search for tools |
| `fhelp cheat [tool]` | Show examples |
| `fhelp [workflow]` | Show analysis workflow |
| `navi --fzf` | Interactive cheat sheets |
| `find-tool [term]` | Alternative tool search |
The entire help system works **completely offline** with local documentation!

View File

@@ -0,0 +1,154 @@
# Enhanced File Analysis Help System
## 🎯 Overview
This document describes the comprehensive improvements made to the container's help system, addressing all the issues you encountered and providing a robust, offline-capable help experience.
## ✅ Problems Fixed
### 1. **Navi Configuration Issues**
- **Problem**: `invalid preview window layout: up:2:nohidden` errors
- **Solution**: Created proper navi configuration file at `/root/.config/navi/config.yaml` with correct fzf settings
- **Result**: Navi now works without preview window errors
### 2. **Inaccurate Tool Information**
- **Problem**: Tools listed that don't exist (evince, ioc_parser), unclear command names
- **Solution**: Created curated tools database with only verified, available tools and their exact command names
- **Result**: `fhelp tools` now shows accurate, actionable information
### 3. **Missing Cheat Sheets**
- **Problem**: Limited or missing practical examples for PDF/malware analysis
- **Solution**: Downloaded existing cheat sheets from popular repos and enhanced them with analysis-specific examples
- **Result**: Comprehensive cheat sheets for analysis workflows with exact commands
### 4. **TLDR Cache Issues**
- **Problem**: TLDR not finding analysis-specific tools
- **Solution**: Created custom TLDR pages for pdfid.py, pdf-parser.py, peepdf, capa
- **Result**: `fhelp quick <tool>` now works for specialized analysis tools
### 5. **Command Clarity**
- **Problem**: Users couldn't find the exact command to run (e.g., confusion about ioc_parser)
- **Solution**: All help now shows exact command syntax with practical examples
- **Result**: Clear, copy-pasteable commands for all analysis scenarios
## 🛠️ Enhanced Components
### 1. **Accurate Tools Database** (`/opt/remnux-docs/tools.db`)
- Only verified, available tools
- Exact command names (not just descriptions)
- Usage examples for each tool
- Availability status indicators
### 2. **Enhanced Cheat Sheets** (`/opt/cheatsheets/`)
- Downloads existing cheat sheets from `cheat/cheatsheets` repo
- Adds analysis-specific examples on top
- Covers PDF analysis workflow, malware analysis workflow
- Security-focused examples (PDF flattening, password removal)
### 3. **Interactive Navi Sheets** (`/opt/navi-cheats/`)
- Dynamic command completion with file discovery
- Interactive parameter selection
- Context-aware suggestions
### 4. **Custom TLDR Pages** (`/root/.local/share/tldr/pages/common/`)
- Analysis-specific tools not in standard TLDR
- Follows standard TLDR format
- Quick reference examples
### 5. **Unified fhelp Command**
- Single interface to all help systems
- Clear command structure with examples
- Workflow-based help (PDF, malware analysis)
- Offline capability verification
## 📋 Usage Examples
### Find Tools
```bash
fhelp tools pdf # Find PDF analysis tools
fhelp tools malware # Find malware analysis tools
fhelp tools --interactive # Interactive tool browser
```
### Get Command Examples
```bash
fhelp cheat pdf # PDF analysis workflow
fhelp cheat pdftk # pdftk-specific examples
fhelp quick tar # Quick tar examples
fhelp examples # Interactive example browser
```
### Analysis Workflows
```bash
fhelp pdf # Step-by-step PDF analysis
fhelp malware # Step-by-step malware analysis
```
### Verification
```bash
fhelp --offline # Check all help systems work offline
fhelp --all # Complete overview of capabilities
```
## 🎯 Key Improvements for File Analysis
### PDF Security Analysis
- **Flattening**: `pdftk suspicious.pdf output safe.pdf flatten`
- **Password Removal**: `qpdf --password=PASSWORD --decrypt encrypted.pdf decrypted.pdf`
- **JavaScript Detection**: `strings document.pdf | grep -i "javascript"`
- **Safe Preview**: `convert document.pdf[0] preview.png`
### Malware Analysis Workflow
- **Capability Detection**: `capa malware.exe`
- **JavaScript Sandbox**: `box-js --output-dir=/tmp suspicious.js`
- **Office Document Analysis**: `oledump.py document.doc`
- **Binary Analysis**: `binwalk malware.bin`
### Data Analysis
- **Interactive Exploration**: `vd data.csv`
- **URL Analysis**: `unfurl http://suspicious.com/path`
## 🔧 Testing
Run comprehensive tests with:
```bash
./test-help-system.sh
```
This validates:
- All help commands work correctly
- Tool availability matches documentation
- File structure is correct
- Integration between help systems works
## 📦 Build Instructions
The enhanced help system is automatically included when building:
```bash
docker build -f Dockerfile.remnux -t tabledevil/file-analysis:enhanced .
```
## 🎉 Benefits
1. **No More Missing Tools**: Only lists tools that actually exist with correct names
2. **Practical Examples**: Real-world analysis commands, not just descriptions
3. **Workflow Guidance**: Step-by-step analysis procedures
4. **Offline Capable**: Works completely offline with comprehensive documentation
5. **Multiple Interfaces**: Choose between fhelp, navi, cheat, tldr based on preference
6. **Security Focus**: Emphasizes safe analysis practices (PDF flattening, sandboxing)
## 📱 Quick Reference Card
| Command | Purpose | Example |
|---------|---------|---------|
| `fhelp` | Main help system | `fhelp tools pdf` |
| `fhelp cheat <tool>` | Tool examples | `fhelp cheat pdftk` |
| `fhelp pdf` | PDF workflow | Shows step-by-step PDF analysis |
| `fhelp malware` | Malware workflow | Shows malware analysis steps |
| `navi` | Interactive browser | Browse all examples with fzf |
| `find-tool <term>` | Tool search | `find-tool javascript` |
| `tldr <tool>` | Quick reference | `tldr tar` |
| `cheat <tool>` | Detailed examples | `cheat 7z` |
This enhanced system provides comprehensive, accurate, offline-capable help for file analysis workflows while leveraging existing community resources and adding specialized analysis examples.

View File

@@ -0,0 +1,148 @@
# Streamlined File Analysis Help System
## 🎯 **Mission Accomplished!**
Based on your test feedback and requirements, I've created a **streamlined, reliable help system** that focuses on what works: **TLDR + Cheat**, removing the problematic navi component.
## ✅ **Issues Fixed**
### **Problems Resolved:**
1. **❌ Navi Removed** - No more broken encoding, preview errors, or crashes
2. **✅ TLDR Working Offline** - Pre-populated cache with custom analysis tool pages
3. **✅ Cheat Working Perfectly** - Proper configuration, fhelp cheat integration
4. **✅ All Your Original Tools Covered** - pdfid.py, pdf-parser.py, peepdf, pdftk, capa, etc.
## 🛠️ **What Works Now**
### **PDF Analysis Tools (Your Original Focus):**
```bash
# Quick reference
tldr pdfid.py # Standard tldr format
tldr pdf-parser.py # Command examples
tldr peepdf # Interactive analysis
# Detailed cheat sheets
fhelp cheat pdfid # pdfid.py examples
fhelp cheat pdf-parser # PDF structure analysis
fhelp cheat pdftk # PDF manipulation (flatten, etc.)
fhelp cheat pdf-analysis # Complete workflow
```
### **Malware Analysis Tools:**
```bash
# Quick reference
tldr capa # Malware capabilities
tldr box-js # JavaScript sandbox
# Detailed workflows
fhelp cheat capa # Capability detection
fhelp cheat malware-analysis # Complete workflow
```
### **Core Utilities:**
```bash
# Downloaded from popular repos
fhelp cheat tar # Archive examples
fhelp cheat 7z # Compression
fhelp cheat grep # Text search
fhelp cheat find # File search
```
## 🎯 **Key Features**
### **1. Focused on Your Original Tools**
- **pdfid.py** - PDF suspicious element detection
- **pdf-parser.py** - PDF structure analysis
- **peepdf** - Interactive PDF analysis
- **pdftk** - PDF flattening for security
- **capa** - Malware capability detection
- Plus all standard utilities
### **2. Security-Focused Examples**
```bash
# PDF Security (Your Priority)
pdftk suspicious.pdf output safe.pdf flatten # Remove JavaScript
qpdf --decrypt encrypted.pdf decrypted.pdf # Remove passwords
convert document.pdf[0] preview.png # Safe preview
```
### **3. Offline-Capable**
- **No Internet Required** - Everything works offline
- **Pre-populated TLDR Cache** - Standard + custom analysis tools
- **Local Cheat Sheets** - Downloaded from quality repos + custom ones
### **4. Multiple Interfaces**
- **fhelp** - Main help system with workflows
- **tldr** - Quick command references
- **fhelp cheat** - Detailed examples
- **fhelp examples** - List all available help
## 📦 **Usage**
### **Build & Run:**
```bash
# Build streamlined container
docker build -f Dockerfile.remnux -t tabledevil/file-analysis:streamlined .
# Run with help system
docker run -it tabledevil/file-analysis:streamlined
```
### **Inside Container:**
```bash
# Main help and workflows
fhelp # Complete help menu
fhelp pdf # PDF analysis workflow
fhelp malware # Malware analysis workflow
# Quick references
tldr pdfid.py # Quick pdfid.py examples
tldr capa # Quick capa examples
# Detailed examples
fhelp cheat pdfid # Comprehensive pdfid.py usage
fhelp cheat pdf-analysis # Complete PDF workflow
fhelp examples # List all available help
```
## 📊 **Resources Included**
- **15+ Cheat Sheets** - Including your original analysis tools
- **5+ Custom TLDR Pages** - For analysis-specific tools
- **Complete Offline Cache** - No network dependencies
- **Security-Focused Examples** - PDF flattening, password removal
- **Workflow Guidance** - Step-by-step analysis procedures
## 🏆 **Test Results**
Run comprehensive tests:
```bash
./test-streamlined-help.sh
```
**Expected Results:**
- ✅ All basic help functions work
- ✅ All fhelp cheat sheets work
- ✅ TLDR functionality works (standard + custom)
- ✅ All analysis tools available
- ✅ Complete offline operation
## 🎉 **Benefits Achieved**
1. **Reliability** - No more crashes or encoding issues
2. **Simplicity** - Focused on tools that work consistently
3. **Completeness** - All your original tools covered with examples
4. **Security Focus** - Practical commands for safe PDF analysis
5. **Offline Operation** - Works without internet connection
6. **Real Examples** - Copy-pasteable commands for actual analysis
## 🚀 **Ready for Production**
The streamlined container is now **production-ready** with:
- **Removed problematic navi** - No more crashes
- **Working TLDR integration** - Both standard and custom pages
- **Reliable cheat system** - Via fhelp cheat integration
- **Complete coverage** - All your original analysis tools
- **Security examples** - PDF flattening, password removal, etc.
**Perfect for your file analysis workflow!** 🎯

View File

@@ -0,0 +1,53 @@
# This Docker image encapsulates the REMnux v7 distro on Ubuntu 20.04 (focal).
# For details about REMnux, including how you can run it on a physical system
# or as a virtual machine, see https://REMnux.org.
#
# You can run this image as a container using a command such as:
#
# docker run --rm -it remnux/remnux-distro /bin/bash
#
# To map a local directory into the container's /home/remnux/files directory,
# you could use a command lile this by supplying the appropriate directory name:
#
# docker run --rm -it -v <local_directory>:/home/remnux/files remnux/remnux-distro /bin/bash
#
# If you'd like to access the container using SSH, you can invoke it like this by
# mapping your local TCP port 22 to the container's internal TCP port 22. In this example,
# the container will remain active in the background:
#
# docker run -d -p 22:22 remnux/remnux-distro
#
# If you're going to run this container in a remote cloud, be sure to change the default
# password and otherwise harden the system according to your requirements.
#
# If you're planning to use Cutter inside the container, you'll need to include the
# --privileged parameter when invoking Docker.
#
FROM ubuntu:20.04
LABEL description="REMnux® is a Linux toolkit for reverse-engineering and analyzing malicious software."
LABEL maintainer="Lenny Zeltser (@lennyzeltser, zeltser.com)"
LABEL version="v2025.30.1"
ARG CAST_VER=0.16.22
USER root
WORKDIR /tmp
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get update && \
apt-get install -y wget gnupg git && \
wget https://github.com/ekristen/cast/releases/download/v${CAST_VER}/cast-v${CAST_VER}-linux-amd64.deb && \
dpkg -i /tmp/cast-v${CAST_VER}-linux-amd64.deb && \
cast install --mode cloud --user remnux remnux && \
rm -rf /root/.cache/* && \
unset DEBIAN_FRONTEND
RUN rm /tmp/cast-v${CAST_VER}-linux-amd64.deb
ENV TERM=linux
WORKDIR /home/remnux
RUN mkdir /var/run/sshd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

74
docs/tool-comparison.md Normal file
View File

@@ -0,0 +1,74 @@
# Tool Comparison: Current Kali vs REMnux
## Current Tools in Kali-based Container
### PDF Analysis Tools
- **pdfid.py** - Schnelle Übersicht über PDF-Aufbau
- **pdf-parser.py** - Zerlegen und extrahieren von PDF-Elementen
- **peepdf** - PDF-Analyse Framework mit Javascript Analyse
- **pdftk** - Tool um das PDF zu "flatten"
- **origami** (Ruby gems) - pdfcop, pdfextract, pdfmetadata
### Malware Analysis Tools
- **capa** - Malware capability detection
- **box-js** - JavaScript sandbox analysis
- **oletools** (oledump.py, rtfdump.py, emldump.py, etc.)
- **visidata** - Data exploration and analysis
- **unfurl** - URL/data analysis
### General Tools
- **exiftool** - Metadata extraction
- **catdoc**, **docx2txt** - Document conversion
- **unrtf** - RTF processing
- **ImageMagick** - Image processing
- **DidierStevens suite** - many analysis tools in /opt/didierstevenssuite/
### System Tools
- **mc** - Midnight Commander
- **wget**, **git**, **p7zip-full**, **npm**
## REMnux PDF Tools (Already Available)
### PDF Analysis Suite (✅ Already have most)
- **peepdf** ✅ - same tool
- **origami** ✅ - same Ruby gem suite
- **pdf-parser.py** ✅ - same Didier Stevens tool
- **pdfid.py** ✅ - same Didier Stevens tool
- **pdftk-java** ✅ - same as pdftk but Java version
- **qpdf** ❌ - PDF manipulation tool we don't have
- **pdfresurrect** ❌ - Extract previous PDF versions, we don't have
- **pdftool** ❌ - Analyze PDF incremental updates, we don't have
### General Document Analysis (REMnux has)
- **base64dump** ❌ - Didier Stevens Base64 decoder, we don't have
- **tesseract** ❌ - OCR tool, we don't have
## Analysis: What We Need to Add
### Tools REMnux has that we don't:
1. **qpdf** - PDF manipulation (merge, convert, transform)
2. **pdfresurrect** - Extract previous versions from PDFs
3. **pdftool** - PDF incremental update analysis
4. **base64dump.py** - Base64 decoder (Didier Stevens)
5. **tesseract** - OCR tool
### Tools we have that REMnux doesn't explicitly list:
1. **capa** - Malware capability detection
2. **box-js** - JavaScript sandbox
3. **visidata** - Data exploration
4. **unfurl** - URL/data analysis
5. **ImageMagick** - Image processing
6. **catdoc/docx2txt** - Document conversion
## Migration Strategy
REMnux base image will provide:
- All our current PDF tools (peepdf, origami, pdf-parser.py, pdfid.py, pdftk)
- Plus additional tools (qpdf, pdfresurrect, pdftool, base64dump, tesseract)
We need to add:
- capa (malware analysis)
- box-js (JavaScript analysis)
- visidata (data exploration)
- unfurl (URL analysis)
- Our German documentation files