- •Contents
- •List of Figures
- •List of Tables
- •Acknowledgments
- •Introduction to MPI
- •Overview and Goals
- •Background of MPI-1.0
- •Background of MPI-1.1, MPI-1.2, and MPI-2.0
- •Background of MPI-1.3 and MPI-2.1
- •Background of MPI-2.2
- •Who Should Use This Standard?
- •What Platforms Are Targets For Implementation?
- •What Is Included In The Standard?
- •What Is Not Included In The Standard?
- •Organization of this Document
- •MPI Terms and Conventions
- •Document Notation
- •Naming Conventions
- •Semantic Terms
- •Data Types
- •Opaque Objects
- •Array Arguments
- •State
- •Named Constants
- •Choice
- •Addresses
- •Language Binding
- •Deprecated Names and Functions
- •Fortran Binding Issues
- •C Binding Issues
- •C++ Binding Issues
- •Functions and Macros
- •Processes
- •Error Handling
- •Implementation Issues
- •Independence of Basic Runtime Routines
- •Interaction with Signals
- •Examples
- •Point-to-Point Communication
- •Introduction
- •Blocking Send and Receive Operations
- •Blocking Send
- •Message Data
- •Message Envelope
- •Blocking Receive
- •Return Status
- •Passing MPI_STATUS_IGNORE for Status
- •Data Type Matching and Data Conversion
- •Type Matching Rules
- •Type MPI_CHARACTER
- •Data Conversion
- •Communication Modes
- •Semantics of Point-to-Point Communication
- •Buffer Allocation and Usage
- •Nonblocking Communication
- •Communication Request Objects
- •Communication Initiation
- •Communication Completion
- •Semantics of Nonblocking Communications
- •Multiple Completions
- •Non-destructive Test of status
- •Probe and Cancel
- •Persistent Communication Requests
- •Send-Receive
- •Null Processes
- •Datatypes
- •Derived Datatypes
- •Type Constructors with Explicit Addresses
- •Datatype Constructors
- •Subarray Datatype Constructor
- •Distributed Array Datatype Constructor
- •Address and Size Functions
- •Lower-Bound and Upper-Bound Markers
- •Extent and Bounds of Datatypes
- •True Extent of Datatypes
- •Commit and Free
- •Duplicating a Datatype
- •Use of General Datatypes in Communication
- •Correct Use of Addresses
- •Decoding a Datatype
- •Examples
- •Pack and Unpack
- •Canonical MPI_PACK and MPI_UNPACK
- •Collective Communication
- •Introduction and Overview
- •Communicator Argument
- •Applying Collective Operations to Intercommunicators
- •Barrier Synchronization
- •Broadcast
- •Example using MPI_BCAST
- •Gather
- •Examples using MPI_GATHER, MPI_GATHERV
- •Scatter
- •Examples using MPI_SCATTER, MPI_SCATTERV
- •Example using MPI_ALLGATHER
- •All-to-All Scatter/Gather
- •Global Reduction Operations
- •Reduce
- •Signed Characters and Reductions
- •MINLOC and MAXLOC
- •All-Reduce
- •Process-local reduction
- •Reduce-Scatter
- •MPI_REDUCE_SCATTER_BLOCK
- •MPI_REDUCE_SCATTER
- •Scan
- •Inclusive Scan
- •Exclusive Scan
- •Example using MPI_SCAN
- •Correctness
- •Introduction
- •Features Needed to Support Libraries
- •MPI's Support for Libraries
- •Basic Concepts
- •Groups
- •Contexts
- •Intra-Communicators
- •Group Management
- •Group Accessors
- •Group Constructors
- •Group Destructors
- •Communicator Management
- •Communicator Accessors
- •Communicator Constructors
- •Communicator Destructors
- •Motivating Examples
- •Current Practice #1
- •Current Practice #2
- •(Approximate) Current Practice #3
- •Example #4
- •Library Example #1
- •Library Example #2
- •Inter-Communication
- •Inter-communicator Accessors
- •Inter-communicator Operations
- •Inter-Communication Examples
- •Caching
- •Functionality
- •Communicators
- •Windows
- •Datatypes
- •Error Class for Invalid Keyval
- •Attributes Example
- •Naming Objects
- •Formalizing the Loosely Synchronous Model
- •Basic Statements
- •Models of Execution
- •Static communicator allocation
- •Dynamic communicator allocation
- •The General case
- •Process Topologies
- •Introduction
- •Virtual Topologies
- •Embedding in MPI
- •Overview of the Functions
- •Topology Constructors
- •Cartesian Constructor
- •Cartesian Convenience Function: MPI_DIMS_CREATE
- •General (Graph) Constructor
- •Distributed (Graph) Constructor
- •Topology Inquiry Functions
- •Cartesian Shift Coordinates
- •Partitioning of Cartesian structures
- •Low-Level Topology Functions
- •An Application Example
- •MPI Environmental Management
- •Implementation Information
- •Version Inquiries
- •Environmental Inquiries
- •Tag Values
- •Host Rank
- •IO Rank
- •Clock Synchronization
- •Memory Allocation
- •Error Handling
- •Error Handlers for Communicators
- •Error Handlers for Windows
- •Error Handlers for Files
- •Freeing Errorhandlers and Retrieving Error Strings
- •Error Codes and Classes
- •Error Classes, Error Codes, and Error Handlers
- •Timers and Synchronization
- •Startup
- •Allowing User Functions at Process Termination
- •Determining Whether MPI Has Finished
- •Portable MPI Process Startup
- •The Info Object
- •Process Creation and Management
- •Introduction
- •The Dynamic Process Model
- •Starting Processes
- •The Runtime Environment
- •Process Manager Interface
- •Processes in MPI
- •Starting Processes and Establishing Communication
- •Reserved Keys
- •Spawn Example
- •Manager-worker Example, Using MPI_COMM_SPAWN.
- •Establishing Communication
- •Names, Addresses, Ports, and All That
- •Server Routines
- •Client Routines
- •Name Publishing
- •Reserved Key Values
- •Client/Server Examples
- •Ocean/Atmosphere - Relies on Name Publishing
- •Simple Client-Server Example.
- •Other Functionality
- •Universe Size
- •Singleton MPI_INIT
- •MPI_APPNUM
- •Releasing Connections
- •Another Way to Establish MPI Communication
- •One-Sided Communications
- •Introduction
- •Initialization
- •Window Creation
- •Window Attributes
- •Communication Calls
- •Examples
- •Accumulate Functions
- •Synchronization Calls
- •Fence
- •General Active Target Synchronization
- •Lock
- •Assertions
- •Examples
- •Error Handling
- •Error Handlers
- •Error Classes
- •Semantics and Correctness
- •Atomicity
- •Progress
- •Registers and Compiler Optimizations
- •External Interfaces
- •Introduction
- •Generalized Requests
- •Examples
- •Associating Information with Status
- •MPI and Threads
- •General
- •Initialization
- •Introduction
- •File Manipulation
- •Opening a File
- •Closing a File
- •Deleting a File
- •Resizing a File
- •Preallocating Space for a File
- •Querying the Size of a File
- •Querying File Parameters
- •File Info
- •Reserved File Hints
- •File Views
- •Data Access
- •Data Access Routines
- •Positioning
- •Synchronism
- •Coordination
- •Data Access Conventions
- •Data Access with Individual File Pointers
- •Data Access with Shared File Pointers
- •Noncollective Operations
- •Collective Operations
- •Seek
- •Split Collective Data Access Routines
- •File Interoperability
- •Datatypes for File Interoperability
- •Extent Callback
- •Datarep Conversion Functions
- •Matching Data Representations
- •Consistency and Semantics
- •File Consistency
- •Random Access vs. Sequential Files
- •Progress
- •Collective File Operations
- •Type Matching
- •Logical vs. Physical File Layout
- •File Size
- •Examples
- •Asynchronous I/O
- •I/O Error Handling
- •I/O Error Classes
- •Examples
- •Subarray Filetype Constructor
- •Requirements
- •Discussion
- •Logic of the Design
- •Examples
- •MPI Library Implementation
- •Systems with Weak Symbols
- •Systems Without Weak Symbols
- •Complications
- •Multiple Counting
- •Linker Oddities
- •Multiple Levels of Interception
- •Deprecated Functions
- •Deprecated since MPI-2.0
- •Deprecated since MPI-2.2
- •Language Bindings
- •Overview
- •Design
- •C++ Classes for MPI
- •Class Member Functions for MPI
- •Semantics
- •C++ Datatypes
- •Communicators
- •Exceptions
- •Mixed-Language Operability
- •Problems With Fortran Bindings for MPI
- •Problems Due to Strong Typing
- •Problems Due to Data Copying and Sequence Association
- •Special Constants
- •Fortran 90 Derived Types
- •A Problem with Register Optimization
- •Basic Fortran Support
- •Extended Fortran Support
- •The mpi Module
- •No Type Mismatch Problems for Subroutines with Choice Arguments
- •Additional Support for Fortran Numeric Intrinsic Types
- •Language Interoperability
- •Introduction
- •Assumptions
- •Initialization
- •Transfer of Handles
- •Status
- •MPI Opaque Objects
- •Datatypes
- •Callback Functions
- •Error Handlers
- •Reduce Operations
- •Addresses
- •Attributes
- •Extra State
- •Constants
- •Interlanguage Communication
- •Language Bindings Summary
- •Groups, Contexts, Communicators, and Caching Fortran Bindings
- •External Interfaces C++ Bindings
- •Change-Log
- •Bibliography
- •Examples Index
- •MPI Declarations Index
- •MPI Function Index
192 CHAPTER 6. GROUPS, CONTEXTS, COMMUNICATORS, AND CACHING
1MPI_COMM_GROUP (see below). MPI does not specify the correspondence between the
2process rank in MPI_COMM_WORLD and its (machine-dependent) absolute address. Neither
3does MPI specify the function of the host process, if any. Other implementation-dependent,
4
5
6
7
8
9
10
11
prede ned communicators may also be provided.
6.3 Group Management
This section describes the manipulation of process groups in MPI. These operations are local and their execution does not require interprocess communication.
12
13
14
15
16
17
18
19
6.3.1 Group Accessors
MPI_GROUP_SIZE(group, size)
IN |
group |
group (handle) |
OUT |
size |
number of processes in the group (integer) |
20int MPI_Group_size(MPI_Group group, int *size)
21MPI_GROUP_SIZE(GROUP, SIZE, IERROR)
22INTEGER GROUP, SIZE, IERROR
23 |
fint MPI::Group::Get_size() const (binding deprecated, see Section 15.2) g |
24 |
|
25 |
|
26 |
|
27 |
MPI_GROUP_RANK(group, rank) |
28
29
30
31
32
33
IN |
group |
group (handle) |
OUT |
rank |
rank of the calling process in group, or |
|
|
MPI_UNDEFINED if the process is not a member (in- |
|
|
teger) |
34 int MPI_Group_rank(MPI_Group group, int *rank)
35
MPI_GROUP_RANK(GROUP, RANK, IERROR)
36
INTEGER GROUP, RANK, IERROR
37
38 fint MPI::Group::Get_rank() const (binding deprecated, see Section 15.2) g
39
40
41
42
43
44
45
46
47
48
6.3. GROUP MANAGEMENT |
193 |
MPI_GROUP_TRANSLATE_RANKS (group1, n, ranks1, group2, ranks2)
IN |
group1 |
group1 (handle) |
IN |
n |
number of ranks in ranks1 and ranks2 arrays (integer) |
IN |
ranks1 |
array of zero or more valid ranks in group1 |
IN |
group2 |
group2 (handle) |
OUT |
ranks2 |
array of corresponding ranks in group2, |
|
|
MPI_UNDEFINED when no correspondence exists. |
int MPI_Group_translate_ranks (MPI_Group group1, int n, int *ranks1, MPI_Group group2, int *ranks2)
MPI_GROUP_TRANSLATE_RANKS(GROUP1, N, RANKS1, GROUP2, RANKS2, IERROR) INTEGER GROUP1, N, RANKS1(*), GROUP2, RANKS2(*), IERROR
fstatic void MPI::Group::Translate_ranks (const MPI::Group& group1, int n, const int ranks1[], const MPI::Group& group2, int ranks2[])
(binding deprecated, see Section 15.2) g
This function is important for determining the relative numbering of the same processes in two di erent groups. For instance, if one knows the ranks of certain processes in the group of MPI_COMM_WORLD, one might want to know their ranks in a subset of that group.
MPI_PROC_NULL is a valid rank for input to MPI_GROUP_TRANSLATE_RANKS, which returns MPI_PROC_NULL as the translated rank.
MPI_GROUP_COMPARE(group1, group2, result)
IN |
group1 |
rst group (handle) |
IN |
group2 |
second group (handle) |
OUT |
result |
result (integer) |
int MPI_Group_compare(MPI_Group group1,MPI_Group group2, int *result)
MPI_GROUP_COMPARE(GROUP1, GROUP2, RESULT, IERROR)
INTEGER GROUP1, GROUP2, RESULT, IERROR
fstatic int MPI::Group::Compare(const MPI::Group& group1,
const MPI::Group& group2) (binding deprecated, see Section 15.2) g
MPI_IDENT results if the group members and group order is exactly the same in both groups. This happens for instance if group1 and group2 are the same handle. MPI_SIMILAR results if the group members are the same but the order is di erent. MPI_UNEQUAL results otherwise.
6.3.2 Group Constructors
Group constructors are used to subset and superset existing groups. These constructors construct new groups from existing groups. These are local operations, and distinct groups may be de ned on di erent processes; a process may also de ne a group that does not include itself. Consistent de nitions are required when groups are used as arguments in
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
194 CHAPTER 6. GROUPS, CONTEXTS, COMMUNICATORS, AND CACHING
1communicator-building functions. MPI does not provide a mechanism to build a group
2from scratch, but only from other, previously de ned groups. The base group, upon which
3all other groups are de ned, is the group associated with the initial communicator
4MPI_COMM_WORLD (accessible through the function MPI_COMM_GROUP).
5
6 |
Rationale. |
In what follows, there is no group duplication function analogous to |
|
|
7MPI_COMM_DUP, de ned later in this chapter. There is no need for a group dupli-
8cator. A group, once created, can have several references to it by making copies of
9the handle. The following constructors address the need for subsets and supersets of
10
11
existing groups. (End of rationale.)
12Advice to implementors. Each group constructor behaves as if it returned a new
13group object. When this new group is a copy of an existing group, then one can
14avoid creating such new objects, using a reference-count mechanism. (End of advice
15to implementors.)
16
17
18
MPI_COMM_GROUP(comm, group)
19 |
|
|
|
20 |
IN |
comm |
communicator (handle) |
21 |
OUT |
group |
group corresponding to comm (handle) |
|
22
23
int MPI_Comm_group(MPI_Comm comm, MPI_Group *group)
24
25MPI_COMM_GROUP(COMM, GROUP, IERROR)
26INTEGER COMM, GROUP, IERROR
27
fMPI::Group MPI::Comm::Get_group() const (binding deprecated, see Section 15.2) g
28
29 MPI_COMM_GROUP returns in group a handle to the group of comm.
30
31
32
33
34
35
36
37
MPI_GROUP_UNION(group1, group2, newgroup)
IN |
group1 |
rst group (handle) |
IN |
group2 |
second group (handle) |
OUT |
newgroup |
union group (handle) |
38 |
int MPI_Group_union(MPI_Group group1, MPI_Group group2, |
39 |
MPI_Group *newgroup) |
40
MPI_GROUP_UNION(GROUP1, GROUP2, NEWGROUP, IERROR)
41
INTEGER GROUP1, GROUP2, NEWGROUP, IERROR
42
43 |
fstatic MPI::Group MPI::Group::Union(const MPI::Group& group1, |
44 |
const MPI::Group& group2) (binding deprecated, see Section 15.2) g |
|
|
45 |
|
46 |
|
47 |
|
48 |
|
6.3. GROUP MANAGEMENT |
195 |
|
MPI_GROUP_INTERSECTION(group1, group2, newgroup) |
||
IN |
group1 |
rst group (handle) |
IN |
group2 |
second group (handle) |
OUT |
newgroup |
intersection group (handle) |
int MPI_Group_intersection(MPI_Group group1, MPI_Group group2, MPI_Group *newgroup)
MPI_GROUP_INTERSECTION(GROUP1, GROUP2, NEWGROUP, IERROR)
INTEGER GROUP1, GROUP2, NEWGROUP, IERROR
fstatic MPI::Group MPI::Group::Intersect(const MPI::Group& group1,
const MPI::Group& group2) (binding deprecated, see Section 15.2) g
MPI_GROUP_DIFFERENCE(group1, group2, newgroup)
IN |
group1 |
rst group (handle) |
IN |
group2 |
second group (handle) |
OUT |
newgroup |
di erence group (handle) |
int MPI_Group_difference(MPI_Group group1, MPI_Group group2, MPI_Group *newgroup)
MPI_GROUP_DIFFERENCE(GROUP1, GROUP2, NEWGROUP, IERROR)
INTEGER GROUP1, GROUP2, NEWGROUP, IERROR
fstatic MPI::Group MPI::Group::Difference(const MPI::Group& group1, const MPI::Group& group2) (binding deprecated, see Section 15.2) g
The set-like operations are de ned as follows:
union All elements of the rst group (group1), followed by all elements of second group (group2) not in rst.
intersect all elements of the rst group that are also in the second group, ordered as inrst group.
di erence all elements of the rst group that are not in the second group, ordered as in the rst group.
Note that for these operations the order of processes in the output group is determined primarily by order in the rst group (if possible) and then, if necessary, by order in the second group. Neither union nor intersection are commutative, but both are associative.
The new group can be empty, that is, equal to MPI_GROUP_EMPTY.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
196 CHAPTER 6. GROUPS, CONTEXTS, COMMUNICATORS, AND CACHING
1
2
3
4
5
6
7
8
9
10
11
MPI_GROUP_INCL(group, n, ranks, newgroup)
IN |
group |
group (handle) |
IN |
n |
number of elements in array ranks (and size of |
|
|
newgroup) (integer) |
IN |
ranks |
ranks of processes in group to appear in |
|
|
newgroup (array of integers) |
OUT |
newgroup |
new group derived from above, in the order de ned by |
|
|
ranks (handle) |
12int MPI_Group_incl(MPI_Group group, int n, int *ranks, MPI_Group *newgroup)
13MPI_GROUP_INCL(GROUP, N, RANKS, NEWGROUP, IERROR)
14INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR
15 |
fMPI::Group MPI::Group::Incl(int n, const int ranks[]) const (binding |
16 |
|
17 |
deprecated, see Section 15.2) g |
18The function MPI_GROUP_INCL creates a group newgroup that consists of the
19n processes in group with ranks rank[0],: : :, rank[n-1]; the process with rank i in newgroup
20is the process with rank ranks[i] in group. Each of the n elements of ranks must be a valid
21rank in group and all elements must be distinct, or else the program is erroneous. If n = 0,
22then newgroup is MPI_GROUP_EMPTY. This function can, for instance, be used to reorder
23the elements of a group. See also MPI_GROUP_COMPARE.
24
25
26 MPI_GROUP_EXCL(group, n, ranks, newgroup)
27
28
29
30
31
32
33
34
35
IN |
group |
group (handle) |
IN |
n |
number of elements in array ranks (integer) |
IN |
ranks |
array of integer ranks in group not to appear in |
|
|
newgroup |
OUT |
newgroup |
new group derived from above, preserving the order |
|
|
de ned by group (handle) |
int MPI_Group_excl(MPI_Group group, int n, int *ranks, MPI_Group *newgroup)
36
37MPI_GROUP_EXCL(GROUP, N, RANKS, NEWGROUP, IERROR)
38INTEGER GROUP, N, RANKS(*), NEWGROUP, IERROR
39
fMPI::Group MPI::Group::Excl(int n, const int ranks[]) const (binding
40
deprecated, see Section 15.2) g
41
42The function MPI_GROUP_EXCL creates a group of processes newgroup that is obtained
43by deleting from group those processes with ranks ranks[0] ,: : : ranks[n-1]. The ordering of
44processes in newgroup is identical to the ordering in group. Each of the n elements of ranks
45must be a valid rank in group and all elements must be distinct; otherwise, the program is
46erroneous. If n = 0, then newgroup is identical to group.
47
48
6.3. GROUP MANAGEMENT |
197 |
|
MPI_GROUP_RANGE_INCL(group, n, ranges, newgroup) |
||
IN |
group |
group (handle) |
IN |
n |
number of triplets in array ranges (integer) |
IN |
ranges |
a one-dimensional array of integer triplets, of the form |
|
|
( rst rank, last rank, stride) indicating ranks in group |
|
|
of processes to be included in newgroup |
OUT |
newgroup |
new group derived from above, in the order de ned by |
|
|
ranges (handle) |
int MPI_Group_range_incl(MPI_Group group, int n, int ranges[][3], MPI_Group *newgroup)
MPI_GROUP_RANGE_INCL(GROUP, N, RANGES, NEWGROUP, IERROR)
INTEGER GROUP, N, RANGES(3,*), NEWGROUP, IERROR
fMPI::Group MPI::Group::Range_incl(int n, const int ranges[][3]) const
(binding deprecated, see Section 15.2) g
If ranges consist of the triplets
(first1; last1; stride1); :::; (firstn; lastn; striden)
then newgroup consists of the sequence of processes in group with ranks first1; first1 + stride1; :::; first1 + last1 first1 stride1; :::
firstn; firstn + striden; :::; firstn + lastn firstn striden: striden
Each computed rank must be a valid rank in group and all computed ranks must be distinct, or else the program is erroneous. Note that we may have firsti > lasti, and stridei may be negative, but cannot be zero.
The functionality of this routine is speci ed to be equivalent to expanding the array of ranges to an array of the included ranks and passing the resulting array of ranks and other arguments to MPI_GROUP_INCL. A call to MPI_GROUP_INCL is equivalent to a call to MPI_GROUP_RANGE_INCL with each rank i in ranks replaced by the triplet (i,i,1) in the argument ranges.
MPI_GROUP_RANGE_EXCL(group, n, ranges, newgroup)
IN |
group |
group (handle) |
IN |
n |
number of elements in array ranges (integer) |
IN |
ranges |
a one-dimensional array of integer triplets of the form |
|
|
( rst rank, last rank, stride), indicating the ranks in |
|
|
group of processes to be excluded from the output |
|
|
group newgroup. |
OUT |
newgroup |
new group derived from above, preserving the order |
|
|
in group (handle) |
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48