.F90 File Format – Parallel features

Fortran source code

Fortran has parallelism of two kinds in its source. The language itself has coarrays and do concurrent. Outside the standard, directives (OpenMP, OpenACC) and libraries (MPI) are written into the same files. Directives are on Directives and preprocessor.

Coarrays and images (Fortran 2008)

A program runs as one or more images, all executing the same code, each with its own copy of every variable. A variable with a codimension in square brackets is a coarray, and an image can read or write another image’s copy by naming it in brackets.

real :: x[*]                       ! one scalar on every image
integer :: a(100)[4, *]              ! corank 2: images arranged in a grid
real, codimension[*] :: y(10)

x = this_image()                    ! number of this image, 1 to num_images()
sync all                            ! wait for every image
if (this_image() == 1) then
  total = 0.0
  do i = 1, num_images()
    total = total + x[i]            ! read the copy on image i
  end do
end if
a(1:10)[2, 1] = a(11:20)            ! write to another image
  • cobounds in brackets: the last is *; [2, *] is a corank-2 coarray; a codimension may be given by the attribute codimension[…]
  • this_image(), num_images(), image_index, lcobound, ucobound, coshape inquire about images and cobounds
  • synchronisation: sync all, sync images(list), sync memory, critical … end critical, lock(l) and unlock(l) with a lock_type variable, and error stop to end all images with an error. They take stat= and errmsg=
  • allocatable coarrays are allocated on all images together: allocate(a(n)[*])
  • GNU Fortran needs OpenCoarrays and -fcoarray=lib (or single); Intel uses -coarray; Cray, NAG and others build them in

Teams, events and failed images (Fortran 2018)

use iso_fortran_env, only: team_type, event_type, stat_failed_image
type(team_type) :: half
type(event_type) :: ready[*]

form team (1 + mod(this_image(), 2), half)
change team (half)
  sync all                           ! synchronises only the team
end team

event post (ready[2])                ! tell image 2
event wait (ready)                   ! wait for a post to my copy
if (failed_images() /= 0) print *, 'an image failed'
fail image
  • form team (number, team-variable) splits the images; change team (t) … end team runs a block inside the new team, where image numbers and sync all refer to the team; sync team (t) and team_number(), get_team() work with teams
  • event post, event wait and event_query give one-sided notification without a barrier
  • fail image (for testing), failed_images(), stopped_images() and image_status() support programs that carry on when an image dies
  • collective subroutines: co_sum, co_min, co_max, co_reduce and co_broadcast work on a variable on every image without synchronisation statements
  • atomic subroutines (atomic_define and atomic_ref in Fortran 2008; atomic_add, atomic_cas and the others in Fortran 2018) on variables of atomic_int_kind or atomic_logical_kind

DO CONCURRENT

do concurrent (Fortran 2008) declares that the passes of a loop do not depend on each other, so a compiler may run them in any order or at once, on threads or on a GPU. Fortran 2018 added locality specifiers and Fortran 2023 a reduce specifier,. The body may call only pure procedures (Fortran 2018 relaxed some of the restrictions of Fortran 2008).

do concurrent (i = 1:n, j = 1:m) local(t) shared(a, b) default(none) reduce(+: s)
  t = a(i, j) * b(i, j)
  s = s + t
end do
The four specifiers are local, local_init (a copy of the value on entry), shared and reduce, with default(none) requiring every variable to be named.

NVIDIA nvfortran (-stdpar) can offload it to a GPU, and other compilers can run it on threads. The standard only promises that the order does not matter.

Directive-based models

  • OpenMP: !$omp parallel do and the rest, shared-memory threads; the runtime in omp_lib. Compiled with -fopenmp (GNU, Flang), -qopenmp or -fiopenmp (Intel), -mp (NVIDIA)
  • OpenACC: !$acc parallel loop and the rest, for accelerators. Compiled with -fopenacc (GNU), -acc (NVIDIA)
  • CUDA Fortran (.cuf or -cuda): attributes(global) subroutine k(a) for a kernel, device, managed, constant, shared and pinned variables, launches such as call k<<<grid, block>>>(a), built-in variables threadidx, blockidx, blockdim, griddim, and the !$cuf kernel do directive
  • MPI: use mpi_f08 or use mpi (or include 'mpif.h'), with call mpi_init(ierr) and so on. MPI is a library, so the source is ordinary Fortran
!$omp parallel do private(i) reduction(+:s) schedule(static)
do i = 1, n
  s = s + a(i)
end do
!$omp end parallel do

attributes(global) subroutine axpy(n, a, x, y)
  integer, value :: n
  real, value :: a
  real, device :: x(n), y(n)
  integer :: i
  i = (blockidx%x - 1) * blockdim%x + threadidx%x
  if (i <= n) y(i) = a * x(i) + y(i)
end subroutine axpy

call axpy<<<(n + 255)/256, 256>>>(n, 2.0, dx, dy)
An OpenMP loop (the sentinel line is a directive) and a CUDA Fortran kernel with its launch.

In the viewer, a !$omp or !$acc line is a directive from the sentinel to a trailing ! comment; the code of a CUDA kernel is coloured as Fortran, with attributes, global, device, managed, value and the other attributes as keywords. The chevrons stay plain.