feat(git-sync): one-way GitHub → Wiki sync

[NagariaHussain]

Why?

Let a Wiki Space mirror a GitHub repo's docs one way (repo → wiki): the repo is the source of truth and the space is read-only in the wiki. See specs/github_one_way_sync.md.

What?

Shipping in tracer-bullet slices:

• TB1a — backend sync engine. Wiki Space "Git Sync" tab (git_synced immutable after insert, repo_full_name, branch, docs_subdir, last_sync_*), whitelisted sync_now(), Wiki Document.source_path identity, and wiki/wiki/git_sync.py (fetch repo tree, infer structure, apply).
• TB1b-i — backend read-only enforcement. A git-synced space is read-only in the wiki: every content-mutation entry point is blocked server-side, only the sync engine gets through.
• TB1b-ii — frontend read-only + create-space dialog. The authoring SPA renders a synced space read-only (non-CR tree, synced banner, non-editable viewer, no mutation affordances) and managers can create a synced space from the New Space dialog.
• TB2 — Edit on GitHub. Every synced page's three-dots menu gains an "Edit on GitHub" action opening the source file in GitHub's web editor (/edit/{branch}/{source_path}); group landing pages point at their README.md/index.md.
• TB3 — .wiki.json structure override. A repo can ship a root .wiki.json (docs_dir + ordered, nestable nav) to drive sidebar order, nesting, and titles instead of alphabetical folder inference; absent → inference unchanged.
• TB4a — GitHub App settings + token-minter + listing API. Backend auth plumbing: a Wiki Settings "GitHub App" tab and wiki/api/github.py that signs an App JWT, mints short-lived installation tokens (so private repos work, no per-space secrets), and lists installations/repos.

How?

TB1a: the engine synthesizes a target Wiki Revision outside a Change Request and drives the existing merge applier (_apply_merge_changes_only) under in_apply_merge_revision, reusing doc_key per source_path. Head-SHA short-circuit → idempotent re-syncs.

TB1b-i: new is_git_synced_space() / assert_space_writable() helpers in wiki/permissions.py (bypass via in_apply_merge_revision), wired into get_or_create_draft_change_request, create_change_request, apply_cr_operations, and reorder_wiki_documents; plus a write-deny in wiki_document_has_permission. The public reader (get_wiki_tree) never creates a CR, so browsing synced content is unaffected.

TB1b-ii: SpaceDetails.vue branches on isGitSynced — the CR-hydrate watch early-returns and the sidebar is sourced from a non-CR get_wiki_tree instead. ContributionBanner becomes an inline "Git synced — read only" bar (repo@branch link + "Sync now"). A readonly prop threads through WikiDocumentList → NestedDraggable (no drag/add/row-actions) and WikiDocumentPanel → WikiEditor (editable:false, no toolbar, save short-circuited). SpaceList.vue's create dialog gains a "Git synced" switch + repo/branch inputs. New e2e git-sync-readonly.spec.ts.

TB3: load_wiki_config() reads the root .wiki.json from the already-fetched tree (raises on malformed JSON); build_nodes_from_config() walks nav and emits the same node shape as the inference path, so the merge applier is untouched — hierarchy rides synthetic dir keys, nav order rides a zero-padded sort key, titles come from nav. 7 unit tests.

TB2: new pure helper frontend/src/lib/github.js (buildGithubEditUrl) + an "Edit on GitHub" item in WikiDocumentPanel built from the cached Wiki Space repo/branch and the doc's source_path. The engine now stamps a group's source_path as its README/index landing file so the link opens an editable file. New e2e git-sync-edit-on-github.spec.ts.

TB4a: _app_jwt() signs a short-lived RS256 JWT (iss=app id, 9-min exp) with the stored private key; installation_access_token() exchanges it at POST /app/installations/{id}/access_tokens for a never-stored installation token (the engine's future token source); installations()/repositories() list the connecting user's installations/repos (paginated) from a user OAuth token. HTTP rides the module-level requests import so tests redirect it. 5 unit tests. TB4 was carved into TB4a (this, backend) and TB4b (OAuth round-trip + repo picker UI + engine token threading).

Progress

• TB1a — Backend skeleton: fields + engine + sync_now
• TB1b-i — Backend read-only enforcement (CR/reorder block + permission deny), 6 unit tests
• TB1b-ii — Frontend read-only render (non-CR tree) + create-space dialog, e2e spec
• TB2 — Edit on GitHub: menu action + group landing source_path; unit + e2e
• TB3 — .wiki.json structure override: load_wiki_config + build_nodes_from_config, 7 unit tests
• TB4a — GitHub App settings + token-minter + listing API; 5 unit tests
• TB4b — OAuth connect + repo picker + engine token threading (private repos)
• TB5 — Sync log & status panel: Wiki Git Sync Log doctype (per-run counts) + GitSyncPanel.vue history; 4 unit tests
• TB6 — Real-time webhook sync

[greptile-apps]

Confidence Score: 3/5

The auth and permission layers are solid, but the sync engine has a correctness bug that will silently break re-syncs for any space with more than 20 pages.

The missing limit=0 on frappe.get_all in _sync_to_live means the src_to_key map is incomplete for spaces with more than 20 documents. Every page beyond the 20th gets a fresh doc_key on each re-sync, causing the merge applier to treat them as new pages rather than updates — producing duplicate documents silently.

wiki/wiki/git_sync.py — the live_docs query at line 517 and the source_path stamping loop at lines 638-643 both need fixes before this ships to production.

Important Files Changed

Filename	Overview
wiki/wiki/git_sync.py	Core sync engine; contains a P1 bug where frappe.get_all without limit=0 silently truncates live_docs at 20, breaking stable doc_key identity for spaces with >20 pages on re-sync. Also has N+1 queries in the source_path stamping loop.
wiki/api/github.py	GitHub App auth, OAuth, and webhook plumbing. HMAC signature verification is correct (constant-time). OAuth CSRF state handling is sound. Minor style issue: save(ignore_permissions=True) in store_app_credentials lacks inline comment.
wiki/permissions.py	New is_git_synced_space / assert_space_writable helpers and write-deny in wiki_document_has_permission. Bypass via in_apply_merge_revision flag is correct and well-contained.
wiki/wiki/doctype/wiki_space/wiki_space.py	Adds git_synced immutability check and sync_now doc method. Missing write-level permission guard on sync_now() — any reader can enqueue a sync via the API.
wiki/www/github/redirect.py	OAuth callback handler; verifies CSRF state, exchanges code for token, persists it, and redirects. Login check and state verification are correct.
wiki/www/github/new_app.py	App manifest creation flow; checks login and write permission on Wiki Settings before proceeding. Correct permission gating.
wiki/www/github/manifest_redirect.py	Manifest redirect handler; verifies state and permission before calling store_app_credentials. Correct.
frontend/src/pages/SpaceDetails.vue	Adds isGitSynced branching to prevent CR hydration and serve non-CR wiki tree for synced spaces. Read-only enforcement in frontend, correctly paired with server-side guards.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant GH as GitHub
    participant WH as /api/method/wiki.api.github.webhook
    participant Q as Background Queue
    participant SE as sync_space()
    participant API as GitHub REST API
    participant DB as Frappe DB

    GH->>WH: POST push event (HMAC-signed)
    WH->>WH: _verify_signature (HMAC-SHA256)
    WH->>DB: get_all Wiki Space (repo+branch match)
    WH->>Q: frappe.enqueue(sync_space, space)

    Note over Q,SE: Background worker picks up job

    SE->>DB: get_doc Wiki Space
    SE->>API: "GET /repos/{repo}/git/ref (head SHA)"
    SE->>SE: short-circuit if SHA unchanged
    SE->>API: "GET /repos/{repo}/git/trees/{sha}?recursive=1"
    SE->>SE: load_wiki_config / build_nodes
    SE->>API: GET blobs per page (content fetch)
    SE->>DB: create_revision_from_live_tree (live snapshot)
    SE->>DB: "frappe.get_all Wiki Document (live_docs — missing limit=0!)"
    SE->>DB: insert Wiki Revision + Wiki Revision Items
    SE->>DB: _apply_merge_changes_only (under in_apply_merge_revision)
    SE->>DB: stamp source_path per node (N+1 loop)
    SE->>DB: "set_value Wiki Space last_sync_*"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant GH as GitHub
    participant WH as /api/method/wiki.api.github.webhook
    participant Q as Background Queue
    participant SE as sync_space()
    participant API as GitHub REST API
    participant DB as Frappe DB

    GH->>WH: POST push event (HMAC-signed)
    WH->>WH: _verify_signature (HMAC-SHA256)
    WH->>DB: get_all Wiki Space (repo+branch match)
    WH->>Q: frappe.enqueue(sync_space, space)

    Note over Q,SE: Background worker picks up job

    SE->>DB: get_doc Wiki Space
    SE->>API: "GET /repos/{repo}/git/ref (head SHA)"
    SE->>SE: short-circuit if SHA unchanged
    SE->>API: "GET /repos/{repo}/git/trees/{sha}?recursive=1"
    SE->>SE: load_wiki_config / build_nodes
    SE->>API: GET blobs per page (content fetch)
    SE->>DB: create_revision_from_live_tree (live snapshot)
    SE->>DB: "frappe.get_all Wiki Document (live_docs — missing limit=0!)"
    SE->>DB: insert Wiki Revision + Wiki Revision Items
    SE->>DB: _apply_merge_changes_only (under in_apply_merge_revision)
    SE->>DB: stamp source_path per node (N+1 loop)
    SE->>DB: "set_value Wiki Space last_sync_*"

Loading

Prompt To Fix All With AI

Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
wiki/wiki/git_sync.py:517-521
**`frappe.get_all` missing `limit` — re-syncs silently truncate at 20 docs**

`frappe.get_all` defaults to `limit_page_length=20`, so `live_docs` will only contain the first 20 Wiki Documents in the space. On a re-sync of any space with more than 20 pages, every page beyond the 20th has no entry in `src_to_key`, so `node["doc_key"]` is reassigned a fresh `frappe.generate_hash(length=12)` each time — breaking the stable identity guarantee. The merge applier will then create duplicate pages instead of updating the existing ones.

Add `limit=0` (or a sufficiently large value) to the query.

### Issue 2 of 4
wiki/wiki/git_sync.py:638-643
**N+1 query pattern in `source_path` stamping**

Two DB calls (`get_value` + `set_value`) are made per node inside this loop. For a repo with hundreds of pages this creates significant DB churn on every sync run. The `doc_key → name` mapping is already computable from the `_apply_merge_changes_only` result, or a single bulk `get_all(filters={"doc_key": ["in", ...]})` lookup could build the map before the loop.

### Issue 3 of 4
wiki/api/github.py:?
`save(ignore_permissions=True)` is used here without an explanatory inline comment, which obscures why the permission bypass is safe. The caller verifies `frappe.has_permission("Wiki Settings", "write")`, but that context is invisible at the call site.

```suggestion
	settings.save(ignore_permissions=True)  # caller verified write permission on Wiki Settings
	frappe.clear_cache(doctype="Wiki Settings")
```

### Issue 4 of 4
wiki/wiki/doctype/wiki_space/wiki_space.py:68-82
**`sync_now()` lacks an explicit write-permission check**

Frappe only validates `read` permission when a `@frappe.whitelist()` document method is invoked via the API. Any user with read access to the Wiki Space can therefore call `sync_now()` directly, potentially hammering the GitHub API rate limits for the installation token on every request. Adding `frappe.only_for(["Wiki Manager", "Wiki Approver"])` or a `can_write_space` guard would restrict triggering syncs to authorised users.

_{Reviews (1): Last reviewed commit: "fix(git-sync): test mock infinite loop h..." | Re-trigger Greptile}

[greptile-apps]

frappe.get_all missing limit — re-syncs silently truncate at 20 docs

frappe.get_all defaults to limit_page_length=20, so live_docs will only contain the first 20 Wiki Documents in the space. On a re-sync of any space with more than 20 pages, every page beyond the 20th has no entry in src_to_key, so node["doc_key"] is reassigned a fresh frappe.generate_hash(length=12) each time — breaking the stable identity guarantee. The merge applier will then create duplicate pages instead of updating the existing ones.

Add limit=0 (or a sufficiently large value) to the query.

Prompt To Fix With AI

This is a comment left during a code review.
Path: wiki/wiki/git_sync.py
Line: 517-521

Comment:
**`frappe.get_all` missing `limit` — re-syncs silently truncate at 20 docs**

`frappe.get_all` defaults to `limit_page_length=20`, so `live_docs` will only contain the first 20 Wiki Documents in the space. On a re-sync of any space with more than 20 pages, every page beyond the 20th has no entry in `src_to_key`, so `node["doc_key"]` is reassigned a fresh `frappe.generate_hash(length=12)` each time — breaking the stable identity guarantee. The merge applier will then create duplicate pages instead of updating the existing ones.

Add `limit=0` (or a sufficiently large value) to the query.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

N+1 query pattern in source_path stamping

Two DB calls (get_value + set_value) are made per node inside this loop. For a repo with hundreds of pages this creates significant DB churn on every sync run. The doc_key → name mapping is already computable from the _apply_merge_changes_only result, or a single bulk get_all(filters={"doc_key": ["in", ...]}) lookup could build the map before the loop.

Prompt To Fix With AI

This is a comment left during a code review.
Path: wiki/wiki/git_sync.py
Line: 638-643

Comment:
**N+1 query pattern in `source_path` stamping**

Two DB calls (`get_value` + `set_value`) are made per node inside this loop. For a repo with hundreds of pages this creates significant DB churn on every sync run. The `doc_key → name` mapping is already computable from the `_apply_merge_changes_only` result, or a single bulk `get_all(filters={"doc_key": ["in", ...]})` lookup could build the map before the loop.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

sync_now() lacks an explicit write-permission check

Frappe only validates read permission when a @frappe.whitelist() document method is invoked via the API. Any user with read access to the Wiki Space can therefore call sync_now() directly, potentially hammering the GitHub API rate limits for the installation token on every request. Adding frappe.only_for(["Wiki Manager", "Wiki Approver"]) or a can_write_space guard would restrict triggering syncs to authorised users.

Prompt To Fix With AI

This is a comment left during a code review.
Path: wiki/wiki/doctype/wiki_space/wiki_space.py
Line: 68-82

Comment:
**`sync_now()` lacks an explicit write-permission check**

Frappe only validates `read` permission when a `@frappe.whitelist()` document method is invoked via the API. Any user with read access to the Wiki Space can therefore call `sync_now()` directly, potentially hammering the GitHub API rate limits for the installation token on every request. Adding `frappe.only_for(["Wiki Manager", "Wiki Approver"])` or a `can_write_space` guard would restrict triggering syncs to authorised users.

How can I resolve this? If you propose a fix, please make it concise.