Add Processing Engine protection for air-gapped Kubernetes deployments

[dburton-influxdata]

Summary

InfluxDB 3 Enterprise has a binary bug (influxdata/influxdb_pro#3607) where Processing Engine initialization runs on ALL pods (ingester, querier, compactor) even when processingEngine.enabled: false. This causes critical failures in air-gapped Kubernetes environments with errors like:

Serve command failed: Python environment initialization failed: Virtual environment error: 
Failed to initialize virtualenv: Activation script not found at '/plugins/.venv/bin/activate'

Impact

• Severity: Critical - completely blocks air-gapped Kubernetes deployments
• Affected Pods: All non-processor pods (ingester, querier, compactor)
• Trigger: Pods cannot access internet to download Python packages for virtualenv
• User Impact: Cannot deploy InfluxDB 3 Enterprise in air-gapped environments

Official InfluxData Workaround

Per official documentation:

⚠️ IMPORTANT: Setting plugin-dir="" or INFLUXDB3_PLUGIN_DIR="" (empty string) does not disable the Processing Engine. You must comment out, remove, or unset the configuration — not set it to empty.

The docs explicitly recommend using INFLUXDB3_UNSET_VARS for Docker/Kubernetes deployments:

docker run -e INFLUXDB3_UNSET_VARS="INFLUXDB3_PLUGIN_DIR" influxdb:3-enterprise

Proposed Helm Chart Enhancement

Add automatic INFLUXDB3_UNSET_VARS to non-processor pods when processingEngine.enabled: false (the default):

{{- if not .Values.processingEngine.enabled }}
# Prevent Processing Engine initialization on non-processor pods (air-gapped deployments)
# Official docs: https://docs.influxdata.com/influxdb3/enterprise/reference/config-options/#disable-the-processing-engine
- name: INFLUXDB3_UNSET_VARS
  value: "INFLUXDB3_PLUGIN_DIR INFLUXDB3_PACKAGE_MANAGER INFLUXDB3_PLUGIN_REPO"
{{- end }}

Files to Modify

• templates/ingester-statefulset.yaml
• templates/querier-statefulset.yaml
• templates/compactor-statefulset.yaml

Why This Enhancement Belongs in Helm Chart

Even after the binary bug is fixed in influxdb_pro:

1. Backward compatibility: Supports older binary versions that have the bug
2. Defense in depth: Explicitly prevents Processing Engine on non-processor pods by design
3. Air-gap safety: Guarantees deployments work regardless of binary version
4. Official best practice: Follows InfluxData's documented recommendation
5. Zero configuration: Users get protection automatically
6. Clear intent: Declaratively states "these pods should NEVER initialize Processing Engine"

Benefits

• ✅ Unblocks air-gapped customer deployments immediately
• ✅ Follows official InfluxData documentation
• ✅ Zero breaking changes - fully backward compatible
• ✅ Minimal code change - low risk
• ✅ Works with all binary versions (old and new)
• ✅ Users don't need manual workaround configuration
• ✅ Protects against future regressions

Customer Validation

A customer in a production air-gapped environment has confirmed this workaround works:

"The work-around by unsetting the VARs seems to allow the Pods to start up now."

Related Issues

• Binary bug: https://github.com/influxdata/influxdb_pro/issues/3607
• Workaround validated: https://github.com/influxdata/influxdb_pro/issues/3607#issuecomment-4453074298
• Official docs: https://docs.influxdata.com/influxdb3/enterprise/reference/config-options/#disable-the-processing-engine

Implementation

I have patches ready for all three StatefulSet templates. Happy to submit a PR once this issue is reviewed and approved.

---

Priority: High
Risk: Minimal (tested workaround, backward compatible, follows official docs)
Impact: Critical for air-gapped deployments