Skip to content

fix: stop re-running the annotation finder per annotation when writing metadata#719

Merged
daniel-ji merged 1 commit into
mainfrom
daniel-ji/fix-annotation-import
Jun 26, 2026
Merged

fix: stop re-running the annotation finder per annotation when writing metadata#719
daniel-ji merged 1 commit into
mainfrom
daniel-ji/fix-annotation-import

Conversation

@daniel-ji

Copy link
Copy Markdown
Contributor

To be merged before #676.

AnnotationImporter.import_metadata() writes one metadata.json per logical
annotation (identifier), whose files list must include every source that shares
that identifier (e.g. a mask + its oriented-points + a mesh). To find those siblings
it re-ran the global annotation finder and filtered by identifier:

anno_files = [
    item
    for item in AnnotationImporter.finder(self.config, **self.parents)
    if item.identifier == self.identifier
]

The finder is global: one call walks all annotation source blocks, globs
each, instantiates it, and calls is_valid() — which for masks loads + numpy-scans
the full MRC file. So one finder pass reads all source files. import_metadata runs
once per annotation, so the whole pass is repeated N annotation times just to keep the 1–2
siblings of each annotation.

Fix (this PR)

Don't call the finder in import_metadata. Accumulate each annotation's metadata
incrementally as its sources are imported, into a class-level map keyed by the
fully-qualified output path:

identifier_metadata_map: dict[str, dict] = {}   # on AnnotationImporter

def import_metadata(self):
    ...
    meta = self.identifier_metadata_map.setdefault(filename, {...})
    meta["files"].extend(self.get_metadata(path))
    meta["object_count"] = max(meta["object_count"], self.get_object_count(dest_prefix))
    self.annotation_metadata.write_metadata(filename, meta)

Each source adds its own files + object_count to the shared doc and rewrites it;
the last sibling's write contains everything. The finder is no longer called from
import_metadata at all.

Notes / scope

  • Single file: ingestion_tools/scripts/importers/annotation.py (~40/33 +/-).
  • Moved get_object_count + abstract get_metadata up from BaseAnnotationSource
    to AnnotationImporter (import_metadata now lives on the base and calls them).
  • Removed the now-dead written_metadata_files guard. IMPORTANT: do not re-add
    written_metadata_files.append(filename) — with incremental writes it would make
    the 2nd+ sibling early-return and silently drop its files.
  • import_metadata sets self.local_metadata = meta so the per-instance accessor
    still reflects the written doc (keeps existing tests' assertions valid).
  • Behavior preserved for the "points defined after non-points sources" case
    (object_count = max across siblings, files accumulated) — no longer relies on
    the old FileNotFoundError "wait for all siblings" retry.

Validation

  • py_compile passes.
  • Local-fs end-to-end tests pass: test_ingest_triangular_mesh / _hff
    (exercise import_item + import_metadata + local_metadata assertions).
  • No regression vs baseline in local moto run (7 passed / 39 failed identical with
    and without the change; the 39 are bucket does not exist from an incomplete
    local moto harness, not the change — they pass in CI).
  • Traced test_import_annotation_metadata_with_multiple_sources by hand: yields
    object_count == 3, 3 files {zarr, ndjson, mrc}, shapes {Point, SegmentationMask}.

@daniel-ji daniel-ji merged commit 6884ab9 into main Jun 26, 2026
11 checks passed
@daniel-ji daniel-ji deleted the daniel-ji/fix-annotation-import branch June 26, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants