feat: implement global namespace with caching and smart S3 routing

This commit overhauls the bucket lookup and region handling logic to support a true global namespace, while significantly improving performance and maintaining data locality constraints.

 **Key Changes:**

 *   **Global Bucket Lookup:** Modified `Persistence::get_bucket_by_name` to remove the `region` constraint. Nodes can now look up any bucket in the global database regardless of its home region.
 *   **Metadata Caching:** Introduced `MetadataCache` using `moka`. This caches bucket, tenant, and policy metadata locally on each node to reduce load on the global database.
 *   **Cache Invalidation via P2P:** Implemented a pub/sub mechanism over the existing `libp2p` cluster mesh. Nodes broadcast `MetadataEvent` (e.g., `BucketUpdated`) when modifying metadata, allowing peers to
      invalidate their local caches near-instantly.
 *   **S3 Gateway Smart Routing:** Updated the S3 Gateway to support `301 Moved Permanently` / `307 Temporary Redirect` responses when a client accesses a bucket located in a different region. This enables AWS
      SDKs to automatically retry against the correct endpoint.
 *   **S3 Location Constraint:** Updated `CreateBucket` to correctly parse and respect the `LocationConstraint` XML body, allowing S3 clients to specify the target region for new buckets.
 *   **Internal Logic Updates:** Refactored `ObjectManager` to verify bucket region *after* lookup, ensuring requests are still processed by the correct regional node (or redirected at the gateway layer).
 *   **Admin CLI & Tests:** Updated the `admin` CLI to accommodate the new configuration structure. Fixed integration tests to account for eventual consistency when using the direct-to-DB admin tool by
      implementing retry logic and configurable cache TTLs.

 **Components Touched:**

 *   `anvil-core`: Persistence layer (SQL updates), Caching logic, P2P Event definitions, ObjectManager refactoring.
 *   `anvil`: S3 Gateway routing/redirection, Service startup/wiring.
 *   `anvil-cli` / `admin`: Configuration updates.
 *   `tests`: Enhanced robustness for async cache updates.

 **Performance Impact:**
   23 Massively reduced read load on the Global Postgres instance for hot paths (e.g., checking bucket existence on every object request).

Author: zcourts

Cargo.lock

@@ -1,104 +1,93 @@
-# Anvil: Open‑Source Object Storage in Rust
+# Anvil: An Open-Source Object Store for AI/ML Research
 
-**Anvil** is an open‑source, S3‑compatible object storage server written in Rust. Built by the team behind Worka, Anvil is designed to host large files—such as open‑source model weights—with high performance and reliability. It exposes a familiar S3 HTTP gateway, a high‑performance gRPC API, multi‑tenant isolation, and the ability to scale from a single development node to a multi‑region cluster.
+[![Build Status](https://github.com/worka-ai/anvil-enterprise/actions/workflows/ci.yml/badge.svg)](https://github.com/worka-ai/anvil-enterprise/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![JOSS Submission](https://joss.theoj.org/papers/10.21105/joss.XXXXX/status.svg)](https://joss.theoj.org/papers/10.21105/joss.XXXXX)
 
----
-
-## 🔥 Why Anvil?
-
-- **Written in Rust**: Modern, memory-safe, and highly concurrent.
-- **S3-Compatible**: Works out of the box with AWS SDKs, CLI, and third-party tools.
-- **gRPC API**: For low-latency, high-throughput access.
-- **Multi-Tenant**: Serve different model groups or clients in isolation.
-- **Clusterable**: Run standalone or as a horizontally-scalable distributed system.
-- **Model Hosting Friendly**: Built to serve billions of tokens efficiently.
+**Anvil** is a high-performance, open-source distributed object store built in Rust. It is designed to address the data management and storage challenges inherent in modern computational research, particularly for large-scale Artificial Intelligence (AI) and Machine Learning (ML) workloads. By providing an S3-compatible interface, a native high-throughput gRPC API, and first-class support for content-addressing, Anvil serves as a foundational infrastructure layer for reproducible and efficient research.
 
 ---
 
-## 🚀 Quick Start (Standalone)
+## Key Features
 
-```bash
-cargo install anvil
-anvil server --root ./data --port 9000
-```
-
-Now test it:
-
-```bash
-aws --endpoint-url http://localhost:9000 s3 ls
-```
+-   **Content-Addressable Storage:** Automatically deduplicates identical data using BLAKE3 hashing, dramatically reducing storage costs for versioned models and datasets.
+-   **High-Performance gRPC Streaming:** A native gRPC API with bidirectional streaming, ideal for high-throughput ML data loaders that feed GPUs directly from storage.
+-   **S3-Compatible Gateway:** Provides drop-in compatibility with the vast ecosystem of existing research tools and SDKs that support the S3 API (Boto3, MLflow, Rclone, etc.).
+-   **Built for the ML Ecosystem:** Includes features like the `anvil hf ingest` command to import model repositories directly from the Hugging Face Hub.
+-   **Modern, Resilient Architecture:** Built in Rust for memory safety and high concurrency, with a SWIM-like gossip protocol over QUIC for clustering and failure detection.
+-   **Multi-Tenant by Design:** Provides strong logical isolation between different users, teams, or projects.
 
 ---
 
-## 🧪 Example: Upload and Fetch via S3
+## 🚀 Quick Start
 
-```bash
-# Upload a file
-aws --endpoint-url http://localhost:9000 s3 cp weights.gguf s3://mymodels/weights.gguf
+The fastest way to get a single-node Anvil instance running is with Docker Compose.
 
-# Fetch the file
-curl http://localhost:9000/mymodels/weights.gguf
-```
-
----
-
-## 🏗️ Building From Source
+1.  **Save the `docker-compose.yml`:**
+    Save the example `docker-compose.yml` from the [Getting Started Guide](./docs/01-getting-started.md) to a local file.
 
-Anvil uses [Rust](https://www.rust-lang.org/tools/install) and requires at least version 1.72.
+2.  **Launch Anvil:**
+    ```bash
+    docker-compose up -d
+    ```
 
-```bash
-git clone https://github.com/worka-ai/anvil
-cd anvil
-cargo build --release
-```
-
----
+3.  **Create Your First Tenant and App:**
+    Use the `admin` tool to create a tenant and an app with API credentials.
+    ```bash
+    # Create a region and a tenant
+    docker compose exec anvil1 admin region create europe-west-1
+    docker compose exec anvil1 admin tenant create my-first-tenant
 
-## ⚙️ Running in Cluster Mode
+    # Create an app and save the credentials
+    docker compose exec anvil1 admin app create --tenant-name my-first-tenant --app-name my-cli-app
+    ```
 
-Start multiple nodes with a shared cluster config (see [docs](https://worka.ai/docs/anvil/operational-guide/scaling)).
-
----
-
-## 📡 gRPC API
-
-See full [API reference](https://worka.ai/docs/anvil/user-guide/grpc-api). Example client use:
-
-```bash
-anvil grpc-client --list-buckets
-```
-
----
-
-## 🔐 Authentication
-
-Supports API key-based tenant isolation. See [Auth docs](https://worka.ai/docs/anvil/user-guide/auth-permissions).
+4.  **Configure the Anvil CLI:**
+    Use the credentials from the previous step to configure your local `anvil` CLI.
+    ```bash
+    anvil configure --host http://localhost:50051 --client-id YOUR_CLIENT_ID --client-secret YOUR_CLIENT_SECRET
+    ```
 
 ---
 
 ## 📘 Documentation
 
-- [Getting Started](https://worka.ai/docs/anvil/getting-started)
-- [Deployment](https://worka.ai/docs/anvil/operational-guide/deployment)
-- [S3 Gateway](https://worka.ai/docs/anvil/user-guide/s3-gateway)
-- [Cluster Scaling](https://worka.ai/docs/anvil/operational-guide/scaling)
-- [Contributing](https://worka.ai/docs/anvil/developer-guide/contributing)
+For complete guides on deployment, architecture, and usage, please see the [**Full Documentation**](./docs/index.md).
+
+-   [Getting Started](./docs/01-getting-started.md)
+-   [Authentication & Permissions](./docs/03-user-guide-authentication.md)
+-   [Using the S3 Gateway](./docs/04-user-guide-s3-gateway.md)
+-   [Deployment Guide](./docs/06-operational-guide-deployment.md)
 
 ---
 
 ## 🤝 Contributing
 
-We welcome PRs! Check out [CONTRIBUTING.md](https://worka.ai/docs/anvil/developer-guide/contributing) and start with [good first issues](https://github.com/worka-ai/anvil/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
+We welcome contributions of all kinds! Please read our [**Contributing Guide**](./CONTRIBUTING.md) to get started. All participation in the Anvil community is governed by our [**Code of Conduct**](./CODE_OF_CONDUCT.md).
 
 ---
 
-## 📣 Community
-
-- [Discord](https://discord.gg/uCWVg5STGh) — Chat with the team
-- [Product Hunt](https://www.producthunt.com/products/worka-anvil)
+## 📜 Citing Anvil
+
+If you use Anvil in your research, please cite it. Once published in JOSS, a BibTeX entry will be provided here.
+
+```bibtex
+@article{Anvil2025,
+  doi = {10.21105/joss.XXXXX},
+  url = {https://doi.org/10.21105/joss.XXXXX},
+  year = {2025},
+  publisher = {The Open Journal},
+  volume = {X},
+  number = {XX},
+  pages = {XXXXX},
+  author = {Your Name and Other Authors},
+  title = {Anvil: An Open-Source Object Store for AI/ML Research},
+  journal = {Journal of Open Source Software}
+}
+```
 
 ---
 
 ## License
 
-Licensed under [Apache 2.0](LICENSE).
+Anvil is licensed under the [Apache 2.0 License](./LICENSE).

README.md

@@ -107,6 +107,7 @@ aes-gcm = "0.10.3"
 constant_time_eq = "0.4.2"
 http-body-util = "0.1.1"
 subtle = "2.6.1"
+moka = { version = "0.12.11", features = ["future"] }
 
 [build-dependencies]
 tonic-prost-build = { version = "0.14.2" }

anvil-core/Cargo.toml

@@ -0,0 +1,99 @@
+use crate::persistence::{Bucket, Tenant};
+use moka::future::Cache;
+use std::time::Duration;
+
+#[derive(Clone, Debug)]
+pub struct MetadataCache {
+    // (tenant_id, bucket_name) -> Bucket
+    buckets: Cache<(i64, String), Bucket>,
+    // bucket_name -> Bucket (for public/S3 lookups without tenant_id context initially)
+    // This might need to handle conflicts if bucket names aren't globally unique, but 
+    // for S3 compat they should be. Assuming global uniqueness for now.
+    buckets_by_name: Cache<String, Bucket>,
+    
+    // api_key -> Tenant
+    tenants: Cache<String, Tenant>,
+    
+    // (app_id, resource, action) -> bool (authorized)
+    // Or perhaps cache the list of policies? 
+    // Let's cache the policies list for an app as that's what `get_policies_for_app` returns.
+    // app_id -> Vec<String> (policies)
+    app_policies: Cache<i64, Vec<String>>,
+}
+
+impl MetadataCache {
+    pub fn new(config: &crate::config::Config) -> Self {
+        let ttl = Duration::from_secs(config.metadata_cache_ttl_secs);
+        Self {
+            buckets: Cache::builder()
+                .max_capacity(10_000)
+                .time_to_live(ttl) 
+                .build(),
+            buckets_by_name: Cache::builder()
+                .max_capacity(10_000)
+                .time_to_live(ttl)
+                .build(),
+            tenants: Cache::builder()
+                .max_capacity(5_000)
+                .time_to_live(ttl * 2) 
+                .build(),
+            app_policies: Cache::builder()
+                .max_capacity(5_000)
+                .time_to_live(ttl)
+                .build(),
+        }
+    }
+
+    pub async fn get_bucket(&self, tenant_id: i64, name: &str) -> Option<Bucket> {
+        self.buckets.get(&(tenant_id, name.to_string())).await
+    }
+
+    pub async fn insert_bucket(&self, tenant_id: i64, name: String, bucket: Bucket) {
+        self.buckets.insert((tenant_id, name.clone()), bucket.clone()).await;
+        self.buckets_by_name.insert(name, bucket).await;
+    }
+
+    pub async fn invalidate_bucket(&self, tenant_id: i64, name: &str) {
+        self.buckets.invalidate(&(tenant_id, name.to_string())).await;
+        self.buckets_by_name.invalidate(name).await;
+    }
+    
+    // For when we only know the name (e.g. deleting by name, or cross-tenant lookup if allowed)
+    pub async fn get_bucket_by_name_only(&self, name: &str) -> Option<Bucket> {
+        self.buckets_by_name.get(name).await
+    }
+
+    pub async fn invalidate_bucket_by_name(&self, name: &str) {
+        self.buckets_by_name.invalidate(name).await;
+        // Note: We can't easily invalidate the (tenant_id, name) key without scanning 
+        // or knowing the tenant_id. This is a trade-off. 
+        // For strict consistency, the caller should provide tenant_id if possible.
+        // However, P2P events usually contain enough info.
+    }
+
+    pub async fn get_tenant(&self, api_key: &str) -> Option<Tenant> {
+        self.tenants.get(api_key).await
+    }
+
+    pub async fn insert_tenant(&self, api_key: String, tenant: Tenant) {
+        self.tenants.insert(api_key, tenant).await;
+    }
+
+    pub async fn invalidate_tenant(&self, api_key: &str) {
+        self.tenants.invalidate(api_key).await;
+    }
+
+    pub async fn get_app_policies(&self, app_id: i64) -> Option<Vec<String>> {
+        self.app_policies.get(&app_id).await
+    }
+
+    pub async fn insert_app_policies(&self, app_id: i64, policies: Vec<String>) {
+        self.app_policies.insert(app_id, policies).await;
+    }
+
+    pub async fn invalidate_app_policies(&self, app_id: i64) {
+        self.app_policies.invalidate(&app_id).await;
+    }
+}
+
+