TODO.txt

# InvOCR Refactoring Project Plan

## 1. Code Structure Refactoring

### 1.1. Module Size Reduction (max 300 lines per file)
- [x] Split `formats/pdf/extractor.py` into smaller modules in `formats/pdf/extractors/`
  - [x] Create `extractors/patterns.py` with common regex patterns
  - [x] Create `extractors/date_utils.py` for date extraction
  - [x] Create `extractors/numeric_utils.py` for number/currency handling
  - [x] Create `extractors/item_utils.py` for line item extraction
  - [x] Create `extractors/totals_utils.py` for totals extraction
  - [x] Create `extractors/base_extractor.py` with common functionality
- [ ] Split `formats/pdf/rule_based_extractor.py` (>600 lines) into:
  - [ ] `extractors/rules/rule_loader.py` - for rule loading
  - [ ] `extractors/rules/rule_extractor.py` - for rule-based extraction logic
  - [ ] `extractors/rules/validation.py` - for rule validation
- [ ] Split other large modules (>300 lines) in core/ and formats/ directories

### 1.2. Extraction System Restructuring
- [x] Implement decision tree approach via `extractor_factory.py`
- [ ] Create organized module structure:
  - [ ] `invocr.extractors` - Base extraction logic
  - [ ] `invocr.extractors.formats` - Format-specific extractors
  - [ ] `invocr.extractors.specialized` - Special case extractors
  - [ ] `invocr.extractors.rules` - Rule-based extraction systems

## 2. Code Organization & CLI Interface

### 2.1. Script Migration
- [x] Move standalone scripts to organized package structure:
  - [x] `invocr/cli/__init__.py` - CLI entry point
  - [x] `invocr/cli/commands/` - Individual command modules
  - [x] `invocr/cli/common.py` - Shared CLI utilities

### 2.2. Command Line Interface Enhancement
- [x] Create `invocr/__main__.py` for `poetry run invocr` commands
- [x] Implement subcommands in CLI:
  - [x] `invocr convert` - Convert documents between formats
  - [x] `invocr extract` - Extract data from documents
  - [x] `invocr batch` - Process multiple files
  - [x] `invocr validate` - Validate extraction results
  - [x] `invocr debug` - Debug extraction process
  - [x] `invocr ocr-text` - View OCR text from documents
  - [x] `invocr workflow` - End-to-end invoice processing workflow

### 2.3. Fix Import Chain
- [ ] Update `formats/pdf/__init__.py` re-exports to fix `extract_document_number` and other import errors
- [ ] Fix all other cross-module imports for refactored code

## 3. Configuration System

### 3.1. YAML Configuration
- [ ] Create YAML schema for extraction configuration
- [ ] Implement `invocr/config/__init__.py` for config loading
- [ ] Add sample config files in `config/` directory

### 3.2. Configuration-driven Extraction Pipeline
- [ ] Create `invocr/pipeline/__init__.py` with high-level pipeline
- [ ] Implement pipeline stages:
  - [ ] Document loading
  - [ ] Format detection
  - [ ] Text extraction
  - [ ] Data extraction
  - [ ] Validation
  - [ ] Output formatting

## 4. Documentation

### 4.1. Split Documentation
- [ ] Reorganize README.md into separate files:
  - [ ] `docs/installation.md` - Installation instructions
  - [ ] `docs/examples.md` - Usage examples
  - [ ] `docs/cli.md` - CLI documentation
  - [ ] `docs/config.md` - Configuration documentation
  - [ ] `docs/api.md` - API documentation

### 4.2. Example Updates
- [ ] Update all examples to use refactored code
- [ ] Add YAML configuration examples

## 5. Current Issues to Address

### 5.1. Data Extraction Fixes
- [x] Fix Adobe JSON extraction (`invocr/extractors/specialized/adobe_extractor.py`)
  - [x] Implement multi-level detection
  - [x] Add OCR text verification
- [ ] Fix total calculation in line items
  - [ ] Ensure totals are properly calculated from items when there's only one item

### 5.2. PDF Processing
- [x] Update `pdf_to_json.py` to support year/month processing like `html_to_png.py`:
  - [x] Add --month/--year arguments
  - [x] Process files in specified directories
  - [x] Save JSON in same directory with matching name

## 6. Testing & Integration

### 6.1. Test Coverage
- [ ] Add tests for all refactored modules
- [ ] Test YAML configuration system
- [ ] Test CLI commands

### 6.2. Continuous Integration
- [ ] Update CI pipeline for refactored code
- [ ] Add linting rules to enforce file size limits

## 7. Import Chain Resolution (Current Priority)
- [ ] Fix `extract_document_number` import error in `formats/pdf/__init__.py`
- [ ] Verify `invocr convert` CLI works end-to-end

## 8. Intelligent Extraction & Validation

### 8.1. Dynamic Decision Tree Implementation
- [x] Create document type classifier
- [x] Implement dynamic extractor selection based on document type
- [x] Build workflow orchestration using previously written generic functions
- [x] Integrate workflow into CLI command system
- [ ] Design fallback mechanisms for unrecognized document types
- [ ] Improve document type classification accuracy (reduce 'unknown' results)

### 8.2. Specialized Extraction Functions
- [x] Develop regex-based extractors with strict validation
- [x] Create unit tests for all extraction functions
- [x] Implement ML/LLM-based extraction for complex documents
- [x] Add step-by-step extraction logging for debugging and validation

### 8.3. OCR and HTML Generation
- [x] Generate HTML from PDF with embedded OCR text
- [x] Split each page into 4 quadrants (top/bottom, left/right) for targeted analysis
- [ ] Implement image analysis for table detection and extraction
- [ ] Create specialized extractors for different document regions

### 8.4. Data Validation Framework
- [x] Implement business rule validation (percentage limits, rounding rules)
- [x] Add context-aware cross-field consistency checks
- [ ] Create validation for specific document types (invoices, receipts, etc.)
- [ ] Implement automatic correction for common formatting issues
- [ ] Fix tax_amount extraction issues (prevent VAT ID misinterpretation)
- [ ] Improve seller information extraction accuracy
- [ ] Tune validation confidence thresholds for better accuracy reporting
- [ ] Add robust error handling for extraction edge cases

### 8.5. Machine Learning Integration
- [x] Integrate ML models for document classification
- [x] Implement LLM-based extraction for unstructured text
- [ ] Create feedback loop for continuous model improvement
- [ ] Add confidence scores for extracted values

### 8.6. Complete Document Generation
- [ ] Create invoice generator from extracted data
- [ ] Implement templates for different document types
- [ ] Add validation before document generation
- [ ] Support multiple output formats (PDF, JSON, CSV)