V2 dev

[coretl]

scanspec 2.0 — Gap Analysis, Code Review & Phased Plan

Fixes #196

A. Gap Analysis

Spec subclasses

1.x feature	Status in 2.0	Notes
Linspace	✅ Done	Axis, start, stop, num. Compiles to linear WindowGenerator.
Linspace.bounded	❌ Missing	Classmethod constructing Linspace from extreme bounds. Nice-to-have for 1.x parity.
Range	❌ Missing	Step-size parameterisation equivalent to Linspace. Required per thoughts.md ("All others will be implemented").
Range.bounded	❌ Missing	Classmethod on Range for extreme-bound construction. Comes with Range.
Line	❌ Missing	Alias for Linspace (Line = Linspace). Trivial once Linspace exists — add to exports.
Static	✅ Done	Single fixed position, optional num.
Spiral	✅ Done	Archimedean spiral; _num_points auto-computes num. Uses non-linear WindowGenerator.
Ellipse	❌ Missing	Grid masked to elliptical footprint. Required per thoughts.md.
Polygon	❌ Missing	Grid masked to polygonal footprint. Required per thoughts.md.
Product	✅ Done	Outer × inner via * operator. Compiles to prepended generators.
Zip	✅ Done	Merge two specs into shared dimension. Compiles to merged innermost generator.
Snake	✅ Done	~ operator sets snake flag on innermost generator.
Concat	✅ Done	Two modes: pure-motion merge or children-based concat for Acquire streams.
Repeat	✅ Done	Prepends outer dimension of length num.
Fly	🚫 Intentionally replaced	Replaced by Acquire(fly=True). Fly was a boolean wrapper in 1.x.
ConstantDuration	🚫 Intentionally replaced	Duration in 2.0 is part of Acquire.duration & DetectorGroup timing.
Squash	🚫 Intentionally dropped	Per thoughts.md: "Squash will not be migrated."
Acquire	✅ Done	New in 2.0. Outermost spec binding fly/step, detectors, continuous streams, monitors.

Core data structures

1.x feature	Status in 2.0	Notes
Dimension (1.x)	✅ Replaced	1.x stores midpoints/lower/upper arrays, gap, duration. 2.0 Dimension stores axes/length/snake + lazy setpoints().
SnakedDimension	✅ Replaced	Snake is a boolean flag on Dimension/WindowGenerator in 2.0.
Path	✅ Replaced	Replaced by Scan.__iter__ yielding Window objects.
Midpoints	✅ Replaced	Replaced by Dimension.setpoints() + Scan.__iter__.
Slice	🚫 Dropped	Single-use wrapper in 1.x. No counterpart needed.
AxesPoints	✅ Replaced	2.0 uses dict[AxisT, float] on Window.static_axes / Window.moving_axes.
gap_between_frames	✅ Replaced	Gap is implicit in window previous linkage.
squash_frames	🚫 Dropped	Accompanies Squash.
stack2dimension	✅ Replaced	Iteration logic in Scan.__iter__ / _iter_with_outer.
discriminated_union_of_subclasses	✅ Replaced	2.0 uses AnySpec runtime class + PosargsMeta + pydantic Discriminator.
StrictConfig	✅ Replaced	model_config = ConfigDict(frozen=True) on Spec base.
WindowGenerator	✅ New	Internal engine: linear (axis_ranges) or non-linear (position_fn).
Window	✅ Done	Yielded by Scan.__iter__; carries static_axes, moving_axes, AxisMotion, trigger_groups.
AxisMotion	✅ Done	Boundary kinematics for fly-scan axes.
TriggerPattern	✅ Done	Repeats + livetime + deadtime.
TriggerGroup	✅ Done	Detectors + trigger_patterns.
DetectorGroup	✅ Done	exposures_per_collection, collections_per_event, livetime, deadtime, detectors.
WindowedStream	✅ Done	name, dimensions, detector_groups.
ContinuousStream	✅ Done	name, detector_groups.
MonitorStream	✅ Done	name, detector.
Scan	✅ Done	generators, windowed_streams, continuous_streams, monitors, __iter__, with_start.

Auxiliary modules

1.x feature	Status in 2.0	Notes
plot_spec / plot.py	❌ Missing	Nice-to-have. Not required for API completeness.
cli.py	❌ Missing	Nice-to-have. plot and service commands.
service.py	❌ Missing	Nice-to-have. REST service for spec validation and point generation.
sphinxext.py	❌ Missing	Nice-to-have. Sphinx example_spec directive.
__main__.py	❌ Missing	Nice-to-have. Entry point for python -m scanspec.
__init__.py exports	❌ Missing	Currently just a docstring. Should export core, specs, __version__.

Deprecated 1.x features (not migrated)

Feature	Status	Notes
fly() function	🚫 Dropped	Deprecated in 1.x; replaced by Acquire(fly=True).
step() function	🚫 Dropped	Deprecated in 1.x; replaced by Acquire.
get_constant_duration()	🚫 Dropped	Deprecated in 1.x.
VARIABLE_DURATION	🚫 Dropped	Duration model redesigned in 2.0.
Mask	🚫 Dropped	"Mask was deleted a couple of releases back" (thoughts.md).

B. Phased Plan

Phase C: Ellipse, Polygon, Range, Line

Goal: Implement remaining Spec subclasses required by thoughts.md.

Files to create/modify:

• src/scanspec2/specs.py — add Ellipse, Polygon, Range, Line alias, Linspace.bounded, Range.bounded
• tests/scanspec2/test_specs.py — add construction tests for new classes
• tests/scanspec2/test_compile.py — add compile/setpoint tests for new classes

Implementation details:

• Range: Like Linspace but parameterised by (axis, start, stop, step). Compute num from step. Compile to linear WindowGenerator.
• Range.bounded(axis, lower, upper, step): Classmethod converting bounds to midpoints.
• Linspace.bounded(axis, lower, upper, num): Classmethod converting bounds to start/stop.
• Line = Linspace: Simple alias.
• Ellipse: (x_axis, x_centre, x_diameter, x_step, y_axis, y_centre, y_diameter, y_step, snake, vertical). Build Range grid, apply elliptical mask, compile to non-linear WindowGenerator.
• Polygon: (x_axis, y_axis, vertices, x_step, y_step, snake, vertical). Build Range grid, apply polygon ray-cast mask, compile to non-linear WindowGenerator.

Tests to add:

• Range construction, compile, setpoints, bounded
• Linspace.bounded construction
• Ellipse construction, compile, mask correctness, point count
• Polygon construction, compile, mask correctness, concave shape
• Line alias identity
• JSON round-trip for Range, Ellipse, Polygon

Acceptance criteria:

• All new specs constructable, compilable, iterable
• pytest tests/scanspec2/ -v passes
• pyright src/scanspec2/ tests/scanspec2/ — 0 errors
• ruff check src/scanspec2/ tests/scanspec2/ — 0 errors

Phase D: Scan.fly property, __init__.py exports, final validation

Goal: Align with API_SPEC on remaining points; clean up exports.

Files to create/modify:

• src/scanspec2/core.py — add Scan.fly property
• src/scanspec2/__init__.py — export core, specs
• tests/scanspec2/test_compile.py — test scan.fly property
• tests/scanspec2/test_core.py — test scan.fly property

Implementation details:

• Scan.fly → @property returning self.generators[-1].fly if self.generators else False
• __init__.py → export core and specs modules; add __version__ if _version.py exists

Tests to add:

• scan.fly returns True for fly scans, False for step scans
• scan.fly returns False for empty generator list
• Import smoke test for scanspec2.core and scanspec2.specs

Acceptance criteria:

• scan.fly accessible as property
• from scanspec2 import core, specs works
• All tests pass, 0 pyright, 0 ruff

Phase E (optional): plot.py, cli.py, service.py, sphinxext.py, __main__.py

Goal: Restore non-essential 1.x parity for tooling.

These modules are nice-to-have and not required for the core API. They can be ported incrementally after the core is complete.

Priority order:

1. plot.py — most useful for development/debugging
2. cli.py + __main__.py — depends on plot.py and service.py
3. service.py — REST API for external consumers
4. sphinxext.py — docs infrastructure

Each of these should be its own sub-phase or conversation.

---

Pre-merge checklist

• Delete API_SPEC.md and thoughts.md — these are working documents that should not ship in the final package.
• Update documentation (docs/) to reflect the scanspec 2.0 API and verify it reads well end-to-end.

[coretl]

We seem to have lost the number of frames for the detector from the windowed_stream. We should add a use case that takes a Scan object and produces a TriggerInfo object to prepare() an ophyd-async StandardDetector with.

I think all the information is there:

• livetime, deadtime, exposures_per_collection, collections_per_event for each detector are available on scan.windowed_stream[n].detector_groups[n]
• number_of_events is the product of dim.length for dim in scan.windowed_stream[n].dimensions, but we could probably do with a property on the scan with this name for ease of use

[hyperrealist]

@coretl I've been spending some time trying to understand the new syntax. Currently, continuous_streams and monitors live on Acquire, which means for a multi-phase scan you need a wrapper Acquire that exists purely as a container with no real meaning:

# The outer Acquire has no detectors or fly — it's just scaffolding
spec = Acquire(
    Concat(
        Acquire(motion_a, fly=True, detectors=[...]),
        Acquire(motion_b, fly=False, detectors=[...]),
    ),
    continuous_streams=[ContinuousStream("cameras", [...])],
)

The issue is that continuous_streams and monitors are conceptually scan-level (run from start to finish, not tied to any window), but they're attached to a node that is semantically a collection phase. When there's only one Acquire it's fine, but with Concat I find it becomes misleading.

Wouldn't introduce a ScanConfig(Spec) that holds only scan-level config, keeping Acquire purely about per-phase collection be conceptually simpler? I am imagining something like:

spec = ScanConfig(
    Concat(
        Acquire(motion_a, fly=True, detectors=[...]),
        Acquire(motion_b, fly=False, detectors=[...]),
    ),
    continuous_streams=[ContinuousStream("cameras", [...])],
    monitors=[MonitorStream("ring_current", ...)],
)
scan = spec.compile()

ScanConfig would be a Spec subclass (so it's serialisable and round-trippable), but combinators (Concat, Product, etc.) would reject it at .compile() time just as they currently reject Acquire with continuous_streams. I know we would be adding another class and a bit more complexity, but I think the resulting syntax would offer a bit more conceptual clarity from user perspective.

[coretl]

I debated this with claude:

1. Acquire can add continuous and monitor streams, but classes like product and concat refuse to work with it if it does
2. Acquire(spec, fly, dets) -> Sync, Acquire(continuous) -> Continuous(name, dets), Acquire(monitor) -> Monitor(name, det)

Claude settled on 1. I was on the fence. If you prefer 2 as well then see how that comes out.

[hyperrealist]

Three changes discussed in meetings with @coretl and @shihab-dls:

1. TriggerPattern execution semantics changed: livetime is now always centred — execution is ½·deadtime → livetime → ½·deadtime, not livetime → deadtime. The struct fields are unchanged; only how consumers must interpret them changes.
2. livetime=0.0 is explicitly valid: a pure dead-gap spacer TriggerPattern(repeats, 0.0, deadtime) serves as the trigger pattern semantic for gaps in the variable-gap use case (ptychography/tomography).
3. Window.positions(dt: float, ...) becomes Window.positions(dt_or_pattern: float | TriggerPattern, ...):
   • Pass a float dt to get positions at a fixed servo-cycle rate.
   • Pass a TriggerPattern to get positions at each trigger instant (equivalent to what 1.x scanspec returned).
   • Both return the same iterator interface yielding chunked dict[axis → np.ndarray].

[coretl]

That is the correct understanding (plus pass max_size to positions() to limit np array length, not time length).

However, I have since thought of some issues with this: PCOMP actually requires the first position in a window at 1/2 deadtime, then all subsequent positions at livetime + 1/2 deadtime. I don't think we should push that down to scanspec.

We need the following consumer patterns:

1. PMAC: dt spaced positions starting at start_time
2. PandA time based: TriggerPattern starting at start_time (the current implementation starts at time=0)
3. PandA position based: positions at arbitrary times specified by start_time.

I suggest the following API instead:

• .trigger_groups is modified to return groups that have truncated TriggerPatterns that start at start_time, this is the data source for 2 and 3
• .time_chunks(dt: float, max_size: float | None = None) -> Iterator[np.ndarray] to give us iterator of time chunks for 1
• .positions(time_array: np.ndarray) -> dict[AxisT, np.ndarray] to give us positions for an arbitrary time array generated from trigger_groups or time_chunks