feat(ragfs): add CachedFileSystem and Redis/Mooncake/Yuanrong cache providers#2520
Conversation
add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
Mijamind719
changed the title
|
感谢在 Rust 文件系统中实现分布式缓存能力，这为我们的系统整体性能提供了巨大的帮助。但是能否增加使用说明文档和配置实例。如:
|
|
Findings P1 crates/ragfs/src/cache/wrapper.rs:139-217 P2 .cargo/config.toml:1-5 P2 crates/ragfs-cache-yuanrong/src/client.rs:16-76 与 crates/ragfs-cache-yuanrong-sys/native/yuanrong_bridge.cpp:24-27,170,185,211,224,238,276,351,391,415 Open Questions / Assumptions PR 标题里写了 Redis/Mooncake/Yuanrong cache providers，但我在这版 diff 里没有看到 Redis adapter 或 Redis 测试代码；如果 Redis 还没进这个 PR，标题和描述现在是 over-claim。 |
1、current_generation() 改为 provider-first。provider 有 generation 时永远以 provider 为准，commit |
add CachedFileSystem + CacheProvider trait and Mooncake/Yuanrong Provide
…ragfs-add-cache # Conflicts: # crates/ragfs-python/src/lib.rs # crates/ragfs/src/core/mountable.rs # openviking/utils/agfs_utils.py # tests/misc/test_config_validation.py
qin-ctx
left a comment
qin-ctx
left a comment
There was a problem hiding this comment.
这轮主要看新增 cache 层和现有 RAGFS 语义的边界。provider 架构、通用缓存层、native provider 隔离方向都不错，但当前 head 仍有两个需要合入前修复的问题：
- 加密挂载启用 cache 后，cache 包在
EncryptionWrappedFS外层，且 key 只按 namespace/path 区分，可能把某个 account 解密后的明文提供给另一个 account 或缺失 context 的读请求。这是安全边界问题。 - 同 mount 的
copy_within_mountfast path 直接写 raw backend，不会经过CachedFileSystem::write()，因此目的文件和父目录缓存不会失效，后续读可能返回旧内容。
另外，PR 的压测/描述里提到 fs/stat 或元数据 cache 收益，但当前 stat() 直接透传 backend，建议同步修正文档或实现语义。
我没有本地跑测试；GitHub 上当前 pr_review check 是通过的。
| #[cfg(feature = "cache")] | ||
| fn maybe_wrap_cache(&self, fs: Arc<dyn FileSystem>, mount_path: &str) -> Arc<dyn FileSystem> { | ||
| match &self.cache { | ||
| Some(cache) => Arc::new(CachedFileSystem::new( |
There was a problem hiding this comment.
这里把 cache 包在已经构造好的 inner_fs 外层，而 inner_fs 在加密挂载时已经是 EncryptionWrappedFS。结果是 cache 存储和返回的是解密后的明文，但 CachedFileSystem 的 key 只包含 namespace/kind/path hash，不包含 FsContext、account_id、root key 或任何安全域。
具体场景：tenant A 读取 /local/shared/doc.md 时 cache miss，会先经过 EncryptionWrappedFS::read() 解密，然后把明文填入 cache；tenant B 或缺失 account context 的读请求后续命中同一个 path key，就会在 cache 层直接拿到 tenant A 的明文，绕过 EncryptionWrappedFS 的 account_id 校验和解密流程。
建议把 cache 放到加密层下面，只缓存 ciphertext；或者把安全域作为 cache contract 的一部分，并保证 cache hit 仍会重新应用 account/decryption 规则。这里需要补一个 encryption + shared cache provider 的回归测试：tenant A 填充 cache 后，tenant B / missing context 不能读到 tenant A 明文。
There was a problem hiding this comment.
-
单 backend 改为 Stats(Encryption(Cache(Backend)))，共享 cache 仅保存 ciphertext。
-
加密 multi-write 暂停 mount-level cache，并输出 warning，避免缓存明文。
-
新增跨 tenant、missing context、ciphertext payload 回归测试。
| data | ||
| } | ||
|
|
||
| async fn write(&self, path: &str, data: &[u8], offset: u64, flags: WriteFlag) -> Result<u64> { |
There was a problem hiding this comment.
这个 write() 路径会删除目的文件 cache 并失效父目录，但同 mount 的 copy_within_mount fast path 不会经过这里。Python cp() 会优先调用 copy_within_mount()，而 MountableFS::copy_raw_within_mount() 直接对 raw_backend.write() 写目的文件。
因此 cache 开启时可以复现 stale read：先缓存 /local/b.md，再把 /local/a.md copy 到 /local/b.md，raw backend 已经写入新内容，但目的文件 cache 和父目录 cache 仍有效，下一次正常读 /local/b.md 可能返回旧内容。
建议在 mount/cache 边界处理这个语义：cache 开启时不要绕过 mounted filesystem stack，或者给 copy_within_mount 增加 cache-aware invalidation/copy 能力，至少在 raw copy 后失效 dst_path 和目的父目录。请补一个 overwrite copy 后再次 read/list 的回归测试。
There was a problem hiding this comment.
-
wrapper.rs: 给 CachedFileSystem 增加 crate 内部的 invalidate_external_write()，用于失效被外部 raw write 绕过的目的文件 cache 和父目录 cache。
-
mountable.rs: copy_within_mount() raw copy 成功后，递归从 mount stack 中找到 CachedFileSystem，对 dst_rel_path 做 cache invalidation。
-
同文件新增回归测试 copy_within_mount_overwrite_invalidates_cached_destination，覆盖 overwrite copy 后再次 read 和 read_dir。
| entries | ||
| } | ||
|
|
||
| async fn stat(&self, path: &str) -> Result<FileInfo> { |
There was a problem hiding this comment.
这里的 stat() 仍然直接透传 backend，没有走 cache。PR 描述和压测 workload 里提到 fs/stat / 元数据 cache 收益，容易让读者理解为 stat 已被缓存。
如果暂时不打算缓存 stat，建议把描述和 benchmark 结论改成 list/tree/read_dir 相关收益；如果 stat 是目标能力，则需要在这里补实际缓存语义和失效测试。
There was a problem hiding this comment.
stat_ls 混合元数据 workload 包含stat/ls/tree，在 cache on 下的直接 cache 收益主要来自 ls/tree 的目录缓存；
fs/stat 本身当前仍直透 backend，已修改benchmark描述
- pass runtime cache config to the Rust binding - cache ciphertext instead of decrypted content - disable cache for encrypted multi-write mounts - update tests and provider build documentation
Ensure copy_within_mount invalidates the cached destination file and parent directory after the raw backend fast path writes data directly. This prevents stale reads and stale directory metadata when cache is enabled and Python cp() uses the same-mount copy optimization. Add a regression test covering overwrite copy followed by read/list.
| ); | ||
| arc | ||
| } else { | ||
| self.maybe_wrap_cache(arc, &normalized_path) |
There was a problem hiding this comment.
这里会让 cache-enabled 的未加密 multi-write mount 变成 Stats(CachedFileSystem(MultiWriteWrappedFS))，但 as_multiwrite() 目前只会解包 StatsWrappedFS 和 EncryptionWrappedFS，不会穿过 CachedFileSystem。因此 system_sync_status() / system_sync_retry() 会误报 mounted filesystem is not multi-write，unmount() 也会跳过 MultiWriteWrappedFS::shutdown()，同挂载 copy 也不会走 copy_within_primary()。建议给 CachedFileSystem 增加一个受控的 inner FS 访问方法，并让 as_multiwrite() 解开 cache 层；同时补 cached + unencrypted multi-write 的 status/retry/unmount/copy 回归测试。
There was a problem hiding this comment.
- 给 CachedFileSystem 加一个 pub(crate) 的受控 inner 访问方法，然后让 as_multiwrite() 递归穿过 cache 层；在 mountable.rs 里让递归能穿过 ArcFileSystem 回到原始 Arc
- 增加cached + unencrypted multi-write回归测试
commit
Allow MountableFS::as_multiwrite() to unwrap CachedFileSystem when discovering the underlying MultiWriteWrappedFS. This preserves multi-write admin paths and same-mount copy behavior for cache-enabled, unencrypted multi-write mounts. Add a regression test covering sync status, sync retry, same-mount copy, and unmount behavior for cached unencrypted multi-write mounts.
- add configurable tree traversal mode to cache policy - keep default tree behavior delegated to backend - allow cached traversal to reuse read_dir directory cache - bypass cached traversal for multi-write backends - add regression coverage for tree cache behavior and fallbacks
Introduce a shared cache traversal mode for recursive APIs and use it to optionally run grep through CachedFileSystem. - add CacheTraversalMode with backend and cached_traversal modes - keep CacheTreeMode as a compatibility alias - route tree and grep through cached traversal only when explicitly enabled - reuse cached read_dir entries and full-file reads during grep traversal - keep multi-write traversal on the backend path - expose storage.agfs.cache.traversal_mode in Python config - raise max cached directory entries threshold to 4096 - add regression tests for grep cache traversal and traversal config
5 participants
Related Discussion
add CachedFileSystem + CacheProvider trait and Redis/Mooncake/Yuanrong Provide
Summary
This PR adds the RAGFS cache provider architecture and introduces native cache provider adapters for Mooncake and Yuanrong.
Implemented functionality:
CacheProviderabstraction used byCachedFileSystem.remove_all/ directory rename subtree invalidationMooncakeProviderMooncakeClientMooncakeConfigYuanrongProviderYuanrongClientYuanrongConfigragfs-cache-yuanrong-sysFFI crateTesting
Mooncake validation:
1/1passed.1/1passed.8/8passed.5/5passed.18/18passed.111/111passed.1/1passed.Yuanrong validation:
6/6passed.3/3passed.5/5passed.4/4passed.1/1passed.Covered behavior:
Note: Yuanrong native validation used the currently available Docker image, whose installed
openyuanrong-datasystemversion is0.6.3压测
测试一种后端形态：
目标是在后端存储和向量库都固定为本地实现的前提下，对比 cache off / cache on(redis) 对 OpenViking 业务接口的端到端收益和额外开销。
规模：
content/read、fs/stat总共1600个文件
1. Workload 矩阵
warm_readcold_readwarm_readhot_coldread_writels_treeinvalidationsearch_supplementwarm_read2. 执行摘要
cold_read、warm_read、hot_cold、read_write和ls_tree的聚合平均延迟均有改善。read_write中读请求显著改善，但写请求 P95 与 cache off 持平，平均写延迟略有增加。3. 主 Workload A/B 收益
4. 并发 Sweep