“Mycelium” – A Hierarchical Framework for Distributed Neural Network Coordination
Note: This project was conceived and designed independently by Shasank Prasad (shasankp.14@gmail.com).
It draws on concepts from modular/adaptive deep learning research (e.g., Alippi & Cambria, 2022) (https://link.springer.com/article/10.1007/s10618-022-00890-9) but introduces new mechanisms such as tag-based routing, patch networks, and cold storage for long-term retention.
NOTE: To current readers, the README.md content (project architecture parts) are deprecrated and shall be updated soon shortly with the latest implementation details
The motivation for this project stems from the growing complexity of modern AI systems, where monolithic neural networks often face significant scalability, maintainability, and adaptability challenges. As AI models expand to handle diverse and evolving problem domains, retraining an entire large-scale network for every new task or data distribution becomes computationally expensive, time-consuming, and prone to catastrophic forgetting.
Project Mycelium aims to address these issues by introducing a hierarchically organized, distributed neural architecture inspired by the decentralized yet highly connected nature of biological mycelial networks. At its core is a meta controller that oversees and coordinates multiple domain expert models, each trained to excel in a specific subdomain.
These domain experts can spawn patch networks—lightweight, temporary submodels designed to rapidly adapt to novel challenges or data anomalies without disrupting the stability of the broader system.
The primary purpose of this architecture is to enable modular learning, parallel specialization, and continuous adaptation. By compartmentalizing expertise and allowing localized updates, the system can evolve in real time while preserving existing knowledge. This reduces training overhead, enhances fault tolerance, and improves the overall robustness of the network. The ultimate aim is to create an AI framework capable of dynamically reallocating resources, adapting to shifting operational requirements, and scaling seamlessly across complex, multi-faceted problem spaces—making it especially valuable for domains such as autonomous systems, large-scale data analytics, and real-time decision-making environments.
The architecture of Project Mycelium is composed of three core layers: the meta controller layer, the domain expert layer, and the patch network layer.
The meta controller layer functions as the system’s central routing and coordination mechanism, maintaining a mapping of input types to their respective domain experts. Rather than being a traditional neural network itself, the meta controller is implemented as a lightweight, dynamically updatable mapping structure (e.g., JSON or binary hash mapping), ensuring that the addition of new experts or patches does not require retraining of the central layer. The domain expert layer contains specialized neural networks—potentially of different architectures such as CNNs, RNNs, or Transformers—each dedicated to a specific domain of knowledge.
When new data relevant to an existing domain is introduced, instead of retraining the original expert (which risks catastrophic forgetting), a patch network is trained in isolation on the new data. Once trained, the patch is linked to the original expert within the meta controller’s mapping, effectively merging their capabilities without overwriting prior knowledge. Communication between subgraphs is possible, allowing outputs from one domain expert to feed into another where cross-domain reasoning is required.
In cases where a domain is completely new, a fresh expert is trained independently and integrated into the mapping. This design supports incremental, non-destructive learning, scalable expansion, and fine-grained modularity, making it adaptable to long-term, evolving AI systems.
A brief outline of how Project Mycelium’s architecture would work in theory.
If Project Mycelium proves successful, it could lay the foundation for an even more advanced neural network architecture—Project Mycelium-X—designed to incorporate secure, modular intelligence layers and enhanced adaptability.
One of the most promising directions involves Encrypted Neural Modules (ENMs)—self-contained, encrypted neural components that can be securely transmitted, integrated, or swapped between systems without exposing proprietary architectures or learned weights. This would enable:
-
Federated Secure Intelligence Sharing: Systems could exchange specialized knowledge without risking sensitive data leaks.
-
Plug-and-Play Domain Expertise: Modular, encrypted components could be attached to a base system to instantly provide new capabilities (e.g., language translation, domain-specific problem solving).
-
Secure Collaborative Learning: Multiple AI agents could cooperatively evolve by exchanging encrypted growth patches, allowing a distributed "hive mind" effect without compromising privacy.
In addition, the future system could integrate:
-
Hierarchical Growth Controllers: Multi-level meta-controllers to coordinate the evolution of multiple subnetworks simultaneously.
-
Self-Repairing Mechanisms: Automatic detection and replacement of underperforming or corrupted subnetworks.
-
Adaptive Encryption Levels: Dynamic cryptographic strength depending on the sensitivity of the learned knowledge.
-
Cross-Domain Fusion Learning: Merging multiple encrypted modules to produce emergent capabilities beyond the scope of the original individual components.
If realized, this evolution of Project Mycelium would mark the convergence of self-growing neural architectures, secure AI intelligence exchange, and distributed collaborative learning, potentially redefining how AI systems communicate, scale, and evolve in secure, interconnected environments
To optimize resource usage and allow domain-specialized components to scale without inflating active memory or GPU load and also tackle hard disk storage problems, the architecture can introduce a Cold Storage System for subgraphs. This system enables low-priority or infrequently accessed domain models to be stored in compressed form and dynamically reactivated when needed.
-
Subgraphs not actively required are quantized to lower precision (e.g., INT8 or FP16) and serialized to disk or a distributed storage node.
-
Each stored subgraph is accompanied by:
- Domain Metadata (type, last usage, performance metrics)
- Mini-Buffers of Historical Samples (500–1000 domain-specific samples from recent interactions)
- Synthetic Sample Generation Rules (e.g., SMOTE parameters, domain-specific augmentation scripts)
-
Storage uses a versioning system to avoid overwriting valuable states, allowing rollbacks if fine-tuning produces regressions.
-
An LRU (Least Recently Used) or weighted activity-based scoring determines when a subgraph becomes eligible for cold storage.
-
Additional criteria may include domain deprecation (user no longer interacts with that domain) or seasonal load balancing (e.g., offloading rarely used image processing during NLP-heavy workloads).
-
The Meta-Mapping Layer detects incoming inputs belonging to a cold-stored domain.
-
The system enters a new operational state called "Remembering" — displayed to the end-user to set expectations about a short warm-up period.
-
The stored quantized subgraph is dequantized back to higher precision (FP16 or FP32 or even FP64 or FP128) and loaded into GPU memory.
-
Associated mini-buffers are retrieved and staged for potential quick fine-tuning.
-
If recent input patterns differ significantly from the mini-buffer distribution, a few-shot fine-tuning phase is triggered:
- Base Samples: Loaded from the mini-buffer.
- Synthetic Samples: Generated via augmentation or SMOTE-style oversampling for data balance.
- Fine-Tuning: Performed at low learning rates to prevent catastrophic forgetting.
-
This phase ensures the subgraph adapts to current context without requiring full retraining.
- Memory Efficiency: Keeps GPU VRAM free for high-priority active domains.
- Scalability: Supports large numbers of specialized subgraphs without resource explosion.
- Adaptability: Retains ability to learn from recent trends while preserving historical performance.
- User Transparency: "Remembering" phase communicates recovery process to maintain user trust.
This project connects to a growing body of research on modular and adaptive deep learning.
In particular, it relates to work such as:
-
Adaptive Modular Deep Learning for Data Streams
(M. Alippi, E. Cambria, et al., Data Mining and Knowledge Discovery, 2022)
[DOI: 10.1007/s10618-022-00890-9]https://link.springer.com/article/10.1007/s10618-022-00890-9
That research explores modularization and dynamic expert routing for adapting to evolving data distributions.
Mycelium builds upon the same broad direction but introduces several unique contributions:
- Semantic tag-based meta-controller: Uses tags to dynamically route inputs, track temporal locality, and preserve context.
- Patch networks: Rather than retraining experts directly, new “patches” are spawned to capture novel knowledge while freezing prior subnetworks.
- Cold storage abstraction: A scalable memory mechanism that allows long-term retention, quantization/dequantization, and retrieval of dormant subnetworks.
The core idea and system design for Mycelium originated independently by Shasank Prasad, prior to reviewing related literature.
Citations are included here for completeness and academic grounding.