Index Map

The index_map_type Fortran module provides core capabilities that support the use of distributed arrays in SPMD parallel programs through the index_map derived type, which describes the mapping of an array's index set to parallel processes. The mapping allows for overlap between processes, and provides collective array gather procedures and scatter-reduction procedures associated with that overlap.

There are two implementations of the module: one using MPI located in src/mpi, and one using Fortran coarrays located in src/caf. The interfaces are virtually the same, making it relatively simple for client code to switch between the two versions.

• The MPI version is stable and in production use (see Truchas).

• The coarray version is beta quality. It is fully tested, but its current performance is highly variable. Performance of the example programs is comparable to the MPI version when using the NAG Fortran compiler and its shared memory-only implementation of coarrays. However performance is very poor when using GFortran / OpenCoarrays, and worse still with the Intel Fortran compilers and their implementation of coarrays. Links to performance results can be found in this README. This is a work in progress.

Documentation

See the Reference Manual.

The doc directory README has links to some additional information.

Examples

The example directory contains several example programs that illustrate the usage of the index_map_type module. These programs are automatically compiled as part of the build process. See its README file for details on the examples and how to run them.

Building and installing

First clone the repository:

git clone https://github.com/nncarlson/index-map.git
cd index-map

The project uses CMake (version 3.19 or later) to compile and install. You may need to set your FC environment variable to the path of your Fortran compiler before running cmake to ensure that CMake finds the correct compiler.

The build has two mutually exclusive backends:

• MPI, the default and principal backend.
• CAF, the experimental Fortran coarray backend.

An install includes a Fortran library, module files, and a CMake config package. Configure with --install-prefix if you want to install somewhere other than CMake's default prefix. To build without installing, omit the cmake --install step.

MPI version

cmake -S . -B build-mpi --install-prefix /path/to/index_map/mpi
cmake --build build-mpi --parallel
cmake --install build-mpi

By default, CMake will configure an optimized "Release" build; add the option -DCMAKE_BUILD_TYPE=Debug to configure a debug build.

If CMake does not automatically find the MPI installation that matches your Fortran compiler, you can help it by setting the environment variable MPI_HOME to the root directory of the MPI installation. Note that the MPI Fortran 2008 interface (the mpi_f08 module) must have been compiled with the same Fortran compiler being used to compile this project.

The MPI version has been tested with recent versions of the NAG, Intel OneAPI, GNU Fortran, and LLVM Fortran compilers.

Fortran coarray version

This CMake setup understands how to build the coarray version when using one of these Fortran compilers (LLVM flang does not yet support coarrays):

• NAG 7.1 or later with its built-in coarray support for shared-memory systems.
• Intel OneAPI ifx with its built-in coarray support. The companion Intel MPI package must be installed and Intel's setup script run to configure your environment. The Intel coarray implementation uses MPI as its backend.
• GFortran with OpenCoarrays. OpenCoarrays supplies the coarray runtime used by gfortran and it also uses MPI as its backend. Be sure the bin directory of the opencoarrays installation is in your path so that the compiler wrapper caf and runner cafrun can be found. Set FC=caf before running cmake. OpenCoarrays 2.10.3 is known to work with MPICH 5.0, and at the time of this writing, index-map requires the development version 17.0 of gfortran due to a regression effecting versions 15 and 16. When/if the fix is back-ported to 15 and 16, those versions should work as well.

cmake -S . -B build-caf -DINDEX_MAP_BACKEND=CAF --install-prefix /path/to/index_map/caf
cmake --build build-caf --parallel
cmake --install build-caf

The installed package provides the CMake package name index_map, the target index_map::index_map, and backend-specific targets index_map::index_map_mpi or index_map::index_map_caf. A single application should use only one backend.

The installed tree has this general form:

include/index_map/*.mod
lib/libindex_map.a
lib/cmake/index_map/index_map-config.cmake
lib/cmake/index_map/index_map-config-version.cmake
lib/cmake/index_map/index_map-targets.cmake

An alternative fpm build is available under the fpm directory. It stages the fypp-generated sources before invoking fpm.

Using index-map in your project

Client CMake projects should use config-mode find_package and request the backend they were built to use.

For an MPI application:

find_package(index_map REQUIRED COMPONENTS MPI)

add_executable(my_app my_app.F90)
target_link_libraries(my_app PRIVATE index_map::index_map_mpi)

For a CAF application:

find_package(index_map REQUIRED COMPONENTS CAF)

add_executable(my_app my_app.F90)
target_link_libraries(my_app PRIVATE index_map::index_map_caf)

If the install prefix is not in CMake's default search path, pass it when configuring the consuming project:

cmake -S . -B build -DCMAKE_PREFIX_PATH=/path/to/index_map/mpi

The generic target index_map::index_map is also available after a successful find_package(index_map ...) call. The backend-specific targets make the consumer's intended backend explicit and allow CMake to reject a mismatched installation through the requested component.

Testing

After building simply run ctest:

ctest --build-dir build-mpi # or build-caf

Note: These parallel tests use 4 processes. Without additional options / environment settings, MPI may refuse to run "over-subscribed" when testing on a system with fewer than 4 cores or hardware threads.

Packaging test

The test/packaging directory contains a small standalone consumer project that verifies an installed index_map package can be found, linked, and run from another CMake project. Configure it with CMAKE_PREFIX_PATH pointing to an installed MPI or CAF backend:

cmake -S test/packaging -B test/packaging/build \
  -DCMAKE_PREFIX_PATH=/path/to/index_map/mpi \
  -DINDEX_MAP_BACKEND=MPI
cmake --build test/packaging/build
ctest --test-dir test/packaging/build

Use -DINDEX_MAP_BACKEND=CAF and the CAF install prefix to test a CAF installation.

License

This project is distributed under the terms of the MIT license. See LICENSE.md for details.