Future of ARPES data analysis with python

[kmnhan]

Although open-source contributions to ARPES data analysis like PyARPES, peaks, and piva are a huge step forward from Igor Pro based proprietary code bases, they still face challenges in standardization. The diversity of experimental setups and the specific needs of different research groups make it difficult to create a unified package or standard data formats. Each group tends to develop its own tools tailored to their specific requirements, leading to decentralized efforts and a lack of interoperability. This trend will likely continue, as each group holds their own preferences and priorities, which can conflict with those of others.

Hence comes the question: is it impossible to create a standard package, or at least a standard on data formats for ARPES data analysis? Can multiple groups agree to develop a common package that can be used by all, with open-source contributions and open discussion? The answer is not straightforward. While it is technically possible to create a standard package, the challenge lies in the willingness of different groups to adopt and contribute to it.

This issue has been filed as a discussion topic, and it is hoped that it will lead to a constructive dialogue among the community.

[phrgab]

Thanks for opening this important issue @kmnhan and for the nice development of erlabpy here. For reference, I am one of the developers of peaks.

I too really like the idea of standardisation here in principle (cf. what xarray - which several open-source ARPES packages already rely heavily on - has achieved for the geosciences community and beyond). But I also agree that this is quite challenging to achieve for a few reasons. The most important seems to be around conventions and requirements: as you write, each lab will have bespoke requirements and also often have a strong preference for different convention choices, naming choices etc. Also I think there is a question of visibility. I'm afraid that I was unaware of your nice development until we were a long way through our development. Probably I could have found it sooner, but did not. And when things are already quite far developed, conventions and approaches can already be quite embedded.

But that all said, I believe there are several aspects in our community that really encourage a collaborative development here: many of us use the same international facilities, and of course students and postdocs from one lab frequently move to another. Having a common software stack would certainly ease such transitions, aid reproducibility, etc. etc. There are also ongoing discussions regarding common data formats (e.g. linked with FAIRmat) which are very relevant here.

So, while I agree this is a complex question, I certainly think it warrants continued discussions!

[arafune]

Although I think it's an interesting issues, I cannot help but recall this.
https://xkcd.com/927/

[kmnhan]

@phrgab, thanks a lot for the thoughtful reply and for your work on peaks! I wasn't aware of the FAIRmat format discussions, and it is great to hear that there are ongoing efforts in the community to address common data formats.

@arafune, thank you for your comment! I understand the sentiment behind the xkcd comic you shared, and I think that it is quite appropriate. I also don't think that we can create a new “one package to rule them all,” but I think we could try to create a common ground for some interoperability between existing packages.

As a first step, since xarray is already a common dependency, I'm considering writing compatibility layers between erlab, PyARPES, and peaks that would enable converting coordinate naming conventions between these packages. This would allow users to easily try out different packages and use features from each without being locked into one specific tool. Compatibility layers should also help in the future if we decide to change our coordinate naming conventions in erlab to align with those of other packages, easing the transition for users by keeping compatibility with legacy code. Additionally, I try to refrain from assuming too much about the data structure when implementing new functions, so that they should work with any generic DataArray object.

If you have any thoughts on this approach or suggestions for how we could further improve interoperability, I would love to hear them!

[phrgab]

Thanks @kmnhan. I think that trying to write compatibility layers is a nice idea for ensuring package interoperability. One question I would have there is about stability of all of the packages. peaks has just been through a major refactor, and so I hope will be more stable from now on, but is still in a 0.x version and so certainly subject to some potential further change. From the version numbers of PyARPES and erlab, I guess maybe the API is a bit more stable, but still I think the nature of our area is that these things do develop over time (e.g. when adding in new detectors that have some different metadata requirements etc.). So if we tried to have such adaptors, I wonder (a) how we make sure that they stay up to date with the latest versions and (b) handle version-by-version changes there. One possibility might be to have clearly-defined schema for the data formats to help make it transparent what is required, but it could still end up some work to keep things aligned...?

[kmnhan]

@phrgab, that is a valid concern. While having a clearly defined schema will help a lot, it would add maintenance burden and will require some work to settle on the schema itself.

What I was planning to do in the short term is to write duck-typed tests along with the compatibility layers that checks whether the transformed data works well with key functions in the target package. This way, if there are any breaking changes in the target package, the tests will fail and we can update the compatibility layer accordingly.

erlab already has a weekly CI that runs tests with the latest versions of its dependencies; I can add tests for the compatibility layers there as well. I think this should help to keep the compatibility layers up to date without too much maintenance burden.

For version-by-version changes, I was thinking of just supporting the latest version of each package, as that is what most users would be using.

I'll come up with a draft implementation in the next few weeks and share it here once I have something working.

[kmnhan]

Additionally, I've looked into the peaks documentation and tinkered with it a bit. It seems to be a well-designed package with a lot of useful features, and I have to say that it looks much easier to learn for new users compared to erlab. Although they diverge on many design choices, I think there are some good ideas in peaks that I would like to consider adopting in erlab.

For future reference, here are some of the key differences between erlab, peaks, and PyARPES I've noticed. I haven't used the other two packages extensively, so I may have missed some aspects or misunderstood some design choices. Please correct me if I am wrong about anything.

• Different coordinate naming conventions

  erlab uses different names for the same physical axis for different experimental geometries, and requires an attribute to specify the geometry. In contrast, peaks uses a consistent naming scheme for the physical axes regardless of the experimental geometry, and adds analyzer angles ana_azi and ana_polar to specify the geometry. PyARPES takes a similar approach to peaks but with different names.

  I'm leaning towards adopting the peaks coordinate naming conventions in erlab since it seems more intuitive, but I would have to think about how to handle backward compatibility with existing code.

• Unit handling

  peaks is well integrated with pint. IMO this is the correct way to handle physical data, but it does add a dependency and some complexity when performing calculations and serialization.

• Metadata handling

  erlab relies on xarray's built-in attribute handling, and most backends flatten nested metadata structures to store them. erlab tries to respect the original metadata as much as possible, and tries to avoid renaming metadata fields from the raw data. This is to allow users to use xarray's built-in serialization methods like to_netcdf and to_zarr without losing metadata. However, this can be quite limiting.

  PyARPES takes a similar approach, but tries to have a more stringent metadata structure and eagerly renames metadata fields to have a consistent structure.

  peaks has a dedicated metadata accessor that can store nested dictionaries and non-string values (with pretty-printing!). I also like that peaks uses pydantic for metadata validation. However, peaks has to use its own serialization methods to preserve metadata, which may hinder interoperability with other xarray-based packages. This is a trade-off that I need to think more about, since peaks's approach seems more user-friendly. I will probably draft a separate issue on this later.

• Data provenance

  Both PyARPES and peaks stores the processing history of the data, and this can be very useful for reproducibility and tracking the analysis steps once you get used to it. This should also help Igor Pro users who are accustomed to having a processing history in their data files.

  In contrast, erlab deliberately avoids storing the processing history of the data, and instead encourages users to use version control systems like Git to track changes in their analysis scripts. This allows for more flexibility in data processing, and encourages users to use transdisciplinary functions. However, I can see the lack of data provenance being a barrier from new users from other packages trying out erlab.

In some ways, erlab tries to behave like an independent GUI application that tries to do as much as possible without powering on a jupyter notebook. This design choice was made with the goal of making it easy to use for users who are accustomed to Igor Pro, by allowing users to conduct most of their analysis with Igor Pro and only use erlab for specific GUI-based tasks like derivative methods for visualization. This has also influenced some design choices mentioned above (like unit and metadata handling). In this sense, I think erlab may serve as a complementary visualization tool for peaks and PyARPES rather than a direct competitor.

[phrgab]

Thanks for the careful assessment here; I think this is a very useful comparison. Few small additional comments:

• coordinates: peaks also implements sign conventions in the loaders, which allow having a transparent $k$-conversion while maintaining signs the same as the experimental setup being used to aid the experimenter on matching between the raw data and the values that should be set on the physical system. The GUIs try to help by supplying local co-ordinate names etc. This has some pros and cons, but we found it the most useful in the end for our workflow. ana_polar doesn't just define the geometry, but also allows a moveable analyser separate to the sample manipulator. In principle ana_tilt does the same. ana_azi is currently used just to set the co-ordinate system, but could be extended for e.g. Fermi mapping by azimuthally rotating the analyser, but we didn't yet implement that.
• Units: pint is great, but does add some complexity for sure, and isn't fully implemented yet in all parts of peaks (e.g. not properly integrated in the fitting)
• Agree with the complementarity comment.