Skip to content

GEMINI.md

Latest commit

 

History

History
48 lines (38 loc) · 3.89 KB

GEMINI.md

File metadata and controls

48 lines (38 loc) · 3.89 KB

Project Context: python-sentinel-pipeline (Status: May 30, 2026 - ROI & Notification Enhancement Phase)

Today's Achievements ("ROI & Notification Enhancement" Phase)

  • Fixed ROI Transparency: Resolved a regression where dark pixels (value 0) were incorrectly treated as transparent by removing harmful srcNodata=0 settings and ensuring proper dstAlpha handling in gdal.Warp.
  • Multi-Tile ROI Mosaicing: Finalized the roi_manager.py refactor to group input products by orbit and date, enabling seamless mosaicing of ROIs that span multiple tiles.
  • Enhanced Orbit Metadata: Updated metadata_engine.py to extract and store relative_orbit and orbit_direction for both Sentinel-1 and Sentinel-2, improving product traceability.
  • Enhanced ROI Notifications:
    • Integrated apprise into roi_manager.py to send targeted alerts for ROI crops.
    • Implemented decoupled image generation: downscaled previews for Bluesky and full-size high-quality JPEGs for direct notifications (Signal, etc.).
  • Universal ROI Support:
    • Refactored ROI logic to be sensor-agnostic; any product (S1, S2, FUSED) can now be targetted for ROI cropping.
    • Hardened filename parsing and timestamp extraction using robust "pinch" logic to handle compound ROI names (e.g., Ust-Luga) and complex product types (e.g., RADAR-BURN).
  • Centralized Metadata Architecture:
    • Refactored metadata_engine.py as the single source of truth for product identification, resolution mapping, and sidecar generation.
    • Simplified rebuild_metadata.py to delegate all heavy lifting to the engine, ensuring consistent metadata across the entire pipeline.
    • ROI crops now correctly inherit parent metadata like cloud_cover.
  • Viewer UI/UX Polish:
    • Streamlined ROI display with dedicated "Regions of Interest" section and subheaders.
    • Added dual-mode sorting for ROIs ("By Product" vs "By ROI") with full persistence.
    • Corrected "Invalid Date" issues by fixing ISO timestamp formatting in sidecars.
    • Updated translations across all supported languages (FI, SE, EN, DE).
  • Download Progress Indicators:
    • Implemented a real-time ASCII progress bar in copernicus/_class.py for large Sentinel-1 downloads, providing immediate visual feedback on data volume and speed.

Historical Achievements

("Performance & Reliability" Phase - April 2026)

  • Multi-BBOX Support: Modified functions.get_boxes() to support semicolon-separated lists and JSON lists.
  • Footprint Optimization: Implemented 10x mask downsampling and recursive hole-filling.
  • Memory Safety: Introduced "Balanced Parallelism" limiting ThreadPoolExecutor and GDAL sub-processes.
  • Alpha Integrity: Unified alpha mask generation across all S2 products.
  • Viewer Enhancements: Robust zIndex management for Overlays and UI consistency improvements.

Performance Lessons

  • Vectorization Scaling: Attempting to vectorize 10m masks with speckle noise is a CPU death trap. 100m downsampling with hole-filling provides identical UI utility for 1% of the compute cost.
  • Sub-process Concurrency: When parallelizing tasks that invoke GDAL, the product of Finalizer_Workers * GDAL_NUM_THREADS must be carefully managed to avoid exponential memory spikes.
  • Variable Interpolation: python-dotenv has limits with complex string interpolation. Manual parsing and fallback resolution is sometimes necessary for robust configuration management.

Future stuff

  • Docker Production: Finalize the Nginx + Pipeline production container suite. Needs conceptualizing.
  • Verification: Monitor long-running queue for any remaining OOM edge cases.

Final Thoughts

The pipeline is now highly autonomous and resilient. It not only searches, downloads, processes, and manages metadata, but it now proactively predicts its next data intake, automatically generates targeted regional crops, and gracefully handles corrupted historical data.