fix: stop re-running the annotation finder per annotation when writing metadata#719
Merged
Merged
Conversation
This was referenced Jun 25, 2026
uermel
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To be merged before #676.
AnnotationImporter.import_metadata()writes onemetadata.jsonper logicalannotation (
identifier), whosefileslist must include every source that sharesthat identifier (e.g. a mask + its oriented-points + a mesh). To find those siblings
it re-ran the global annotation finder and filtered by identifier:
The finder is global: one call walks all annotation source blocks, globs
each, instantiates it, and calls
is_valid()— which for masks loads + numpy-scansthe full MRC file. So one finder pass reads all source files.
import_metadatarunsonce per annotation, so the whole pass is repeated
Nannotation times just to keep the 1–2siblings of each annotation.
Fix (this PR)
Don't call the finder in
import_metadata. Accumulate each annotation's metadataincrementally as its sources are imported, into a class-level map keyed by the
fully-qualified output path:
Each source adds its own files +
object_countto the shared doc and rewrites it;the last sibling's write contains everything. The finder is no longer called from
import_metadataat all.Notes / scope
ingestion_tools/scripts/importers/annotation.py(~40/33 +/-).get_object_count+ abstractget_metadataup fromBaseAnnotationSourceto
AnnotationImporter(import_metadatanow lives on the base and calls them).written_metadata_filesguard. IMPORTANT: do not re-addwritten_metadata_files.append(filename)— with incremental writes it would makethe 2nd+ sibling early-return and silently drop its files.
import_metadatasetsself.local_metadata = metaso the per-instance accessorstill reflects the written doc (keeps existing tests' assertions valid).
(
object_count= max across siblings,filesaccumulated) — no longer relies onthe old
FileNotFoundError"wait for all siblings" retry.Validation
py_compilepasses.test_ingest_triangular_mesh/_hff(exercise
import_item+import_metadata+local_metadataassertions).and without the change; the 39 are
bucket does not existfrom an incompletelocal moto harness, not the change — they pass in CI).
test_import_annotation_metadata_with_multiple_sourcesby hand: yieldsobject_count == 3, 3 files{zarr, ndjson, mrc}, shapes{Point, SegmentationMask}.