- •Contents
- •List of Figures
- •List of Tables
- •Acknowledgments
- •Introduction to MPI
- •Overview and Goals
- •Background of MPI-1.0
- •Background of MPI-1.1, MPI-1.2, and MPI-2.0
- •Background of MPI-1.3 and MPI-2.1
- •Background of MPI-2.2
- •Who Should Use This Standard?
- •What Platforms Are Targets For Implementation?
- •What Is Included In The Standard?
- •What Is Not Included In The Standard?
- •Organization of this Document
- •MPI Terms and Conventions
- •Document Notation
- •Naming Conventions
- •Semantic Terms
- •Data Types
- •Opaque Objects
- •Array Arguments
- •State
- •Named Constants
- •Choice
- •Addresses
- •Language Binding
- •Deprecated Names and Functions
- •Fortran Binding Issues
- •C Binding Issues
- •C++ Binding Issues
- •Functions and Macros
- •Processes
- •Error Handling
- •Implementation Issues
- •Independence of Basic Runtime Routines
- •Interaction with Signals
- •Examples
- •Point-to-Point Communication
- •Introduction
- •Blocking Send and Receive Operations
- •Blocking Send
- •Message Data
- •Message Envelope
- •Blocking Receive
- •Return Status
- •Passing MPI_STATUS_IGNORE for Status
- •Data Type Matching and Data Conversion
- •Type Matching Rules
- •Type MPI_CHARACTER
- •Data Conversion
- •Communication Modes
- •Semantics of Point-to-Point Communication
- •Buffer Allocation and Usage
- •Nonblocking Communication
- •Communication Request Objects
- •Communication Initiation
- •Communication Completion
- •Semantics of Nonblocking Communications
- •Multiple Completions
- •Non-destructive Test of status
- •Probe and Cancel
- •Persistent Communication Requests
- •Send-Receive
- •Null Processes
- •Datatypes
- •Derived Datatypes
- •Type Constructors with Explicit Addresses
- •Datatype Constructors
- •Subarray Datatype Constructor
- •Distributed Array Datatype Constructor
- •Address and Size Functions
- •Lower-Bound and Upper-Bound Markers
- •Extent and Bounds of Datatypes
- •True Extent of Datatypes
- •Commit and Free
- •Duplicating a Datatype
- •Use of General Datatypes in Communication
- •Correct Use of Addresses
- •Decoding a Datatype
- •Examples
- •Pack and Unpack
- •Canonical MPI_PACK and MPI_UNPACK
- •Collective Communication
- •Introduction and Overview
- •Communicator Argument
- •Applying Collective Operations to Intercommunicators
- •Barrier Synchronization
- •Broadcast
- •Example using MPI_BCAST
- •Gather
- •Examples using MPI_GATHER, MPI_GATHERV
- •Scatter
- •Examples using MPI_SCATTER, MPI_SCATTERV
- •Example using MPI_ALLGATHER
- •All-to-All Scatter/Gather
- •Global Reduction Operations
- •Reduce
- •Signed Characters and Reductions
- •MINLOC and MAXLOC
- •All-Reduce
- •Process-local reduction
- •Reduce-Scatter
- •MPI_REDUCE_SCATTER_BLOCK
- •MPI_REDUCE_SCATTER
- •Scan
- •Inclusive Scan
- •Exclusive Scan
- •Example using MPI_SCAN
- •Correctness
- •Introduction
- •Features Needed to Support Libraries
- •MPI's Support for Libraries
- •Basic Concepts
- •Groups
- •Contexts
- •Intra-Communicators
- •Group Management
- •Group Accessors
- •Group Constructors
- •Group Destructors
- •Communicator Management
- •Communicator Accessors
- •Communicator Constructors
- •Communicator Destructors
- •Motivating Examples
- •Current Practice #1
- •Current Practice #2
- •(Approximate) Current Practice #3
- •Example #4
- •Library Example #1
- •Library Example #2
- •Inter-Communication
- •Inter-communicator Accessors
- •Inter-communicator Operations
- •Inter-Communication Examples
- •Caching
- •Functionality
- •Communicators
- •Windows
- •Datatypes
- •Error Class for Invalid Keyval
- •Attributes Example
- •Naming Objects
- •Formalizing the Loosely Synchronous Model
- •Basic Statements
- •Models of Execution
- •Static communicator allocation
- •Dynamic communicator allocation
- •The General case
- •Process Topologies
- •Introduction
- •Virtual Topologies
- •Embedding in MPI
- •Overview of the Functions
- •Topology Constructors
- •Cartesian Constructor
- •Cartesian Convenience Function: MPI_DIMS_CREATE
- •General (Graph) Constructor
- •Distributed (Graph) Constructor
- •Topology Inquiry Functions
- •Cartesian Shift Coordinates
- •Partitioning of Cartesian structures
- •Low-Level Topology Functions
- •An Application Example
- •MPI Environmental Management
- •Implementation Information
- •Version Inquiries
- •Environmental Inquiries
- •Tag Values
- •Host Rank
- •IO Rank
- •Clock Synchronization
- •Memory Allocation
- •Error Handling
- •Error Handlers for Communicators
- •Error Handlers for Windows
- •Error Handlers for Files
- •Freeing Errorhandlers and Retrieving Error Strings
- •Error Codes and Classes
- •Error Classes, Error Codes, and Error Handlers
- •Timers and Synchronization
- •Startup
- •Allowing User Functions at Process Termination
- •Determining Whether MPI Has Finished
- •Portable MPI Process Startup
- •The Info Object
- •Process Creation and Management
- •Introduction
- •The Dynamic Process Model
- •Starting Processes
- •The Runtime Environment
- •Process Manager Interface
- •Processes in MPI
- •Starting Processes and Establishing Communication
- •Reserved Keys
- •Spawn Example
- •Manager-worker Example, Using MPI_COMM_SPAWN.
- •Establishing Communication
- •Names, Addresses, Ports, and All That
- •Server Routines
- •Client Routines
- •Name Publishing
- •Reserved Key Values
- •Client/Server Examples
- •Ocean/Atmosphere - Relies on Name Publishing
- •Simple Client-Server Example.
- •Other Functionality
- •Universe Size
- •Singleton MPI_INIT
- •MPI_APPNUM
- •Releasing Connections
- •Another Way to Establish MPI Communication
- •One-Sided Communications
- •Introduction
- •Initialization
- •Window Creation
- •Window Attributes
- •Communication Calls
- •Examples
- •Accumulate Functions
- •Synchronization Calls
- •Fence
- •General Active Target Synchronization
- •Lock
- •Assertions
- •Examples
- •Error Handling
- •Error Handlers
- •Error Classes
- •Semantics and Correctness
- •Atomicity
- •Progress
- •Registers and Compiler Optimizations
- •External Interfaces
- •Introduction
- •Generalized Requests
- •Examples
- •Associating Information with Status
- •MPI and Threads
- •General
- •Initialization
- •Introduction
- •File Manipulation
- •Opening a File
- •Closing a File
- •Deleting a File
- •Resizing a File
- •Preallocating Space for a File
- •Querying the Size of a File
- •Querying File Parameters
- •File Info
- •Reserved File Hints
- •File Views
- •Data Access
- •Data Access Routines
- •Positioning
- •Synchronism
- •Coordination
- •Data Access Conventions
- •Data Access with Individual File Pointers
- •Data Access with Shared File Pointers
- •Noncollective Operations
- •Collective Operations
- •Seek
- •Split Collective Data Access Routines
- •File Interoperability
- •Datatypes for File Interoperability
- •Extent Callback
- •Datarep Conversion Functions
- •Matching Data Representations
- •Consistency and Semantics
- •File Consistency
- •Random Access vs. Sequential Files
- •Progress
- •Collective File Operations
- •Type Matching
- •Logical vs. Physical File Layout
- •File Size
- •Examples
- •Asynchronous I/O
- •I/O Error Handling
- •I/O Error Classes
- •Examples
- •Subarray Filetype Constructor
- •Requirements
- •Discussion
- •Logic of the Design
- •Examples
- •MPI Library Implementation
- •Systems with Weak Symbols
- •Systems Without Weak Symbols
- •Complications
- •Multiple Counting
- •Linker Oddities
- •Multiple Levels of Interception
- •Deprecated Functions
- •Deprecated since MPI-2.0
- •Deprecated since MPI-2.2
- •Language Bindings
- •Overview
- •Design
- •C++ Classes for MPI
- •Class Member Functions for MPI
- •Semantics
- •C++ Datatypes
- •Communicators
- •Exceptions
- •Mixed-Language Operability
- •Problems With Fortran Bindings for MPI
- •Problems Due to Strong Typing
- •Problems Due to Data Copying and Sequence Association
- •Special Constants
- •Fortran 90 Derived Types
- •A Problem with Register Optimization
- •Basic Fortran Support
- •Extended Fortran Support
- •The mpi Module
- •No Type Mismatch Problems for Subroutines with Choice Arguments
- •Additional Support for Fortran Numeric Intrinsic Types
- •Language Interoperability
- •Introduction
- •Assumptions
- •Initialization
- •Transfer of Handles
- •Status
- •MPI Opaque Objects
- •Datatypes
- •Callback Functions
- •Error Handlers
- •Reduce Operations
- •Addresses
- •Attributes
- •Extra State
- •Constants
- •Interlanguage Communication
- •Language Bindings Summary
- •Groups, Contexts, Communicators, and Caching Fortran Bindings
- •External Interfaces C++ Bindings
- •Change-Log
- •Bibliography
- •Examples Index
- •MPI Declarations Index
- •MPI Function Index
1
2
3
4
5
248 |
CHAPTER 7. PROCESS TOPOLOGIES |
7.5 Topology Constructors
7.5.1 Cartesian Constructor
6MPI_CART_CREATE(comm_old, ndims, dims, periods, reorder, comm_cart)
7 |
IN |
comm_old |
input communicator (handle) |
|
8 |
||||
|
|
|
||
9 |
IN |
ndims |
number of dimensions of Cartesian grid (integer) |
|
10 |
IN |
dims |
integer array of size ndims specifying the number of |
|
|
||||
11 |
|
|
processes in each dimension |
|
|
|
|
||
12 |
IN |
periods |
logical array of size ndims specifying whether the grid |
|
13 |
||||
|
|
is periodic (true) or not (false) in each dimension |
||
14 |
|
|
||
|
|
|
||
15 |
IN |
reorder |
ranking may be reordered (true) or not (false) (logical) |
|
16 |
OUT |
comm_cart |
communicator with new Cartesian topology (handle) |
|
|
17
18
int MPI_Cart_create(MPI_Comm comm_old, int ndims, int *dims, int *periods,
19
int reorder, MPI_Comm *comm_cart)
20
21MPI_CART_CREATE(COMM_OLD, NDIMS, DIMS, PERIODS, REORDER, COMM_CART, IERROR)
22INTEGER COMM_OLD, NDIMS, DIMS(*), COMM_CART, IERROR
23LOGICAL PERIODS(*), REORDER
24
25
26
27
fMPI::Cartcomm MPI::Intracomm::Create_cart(int ndims, const int dims[], const bool periods[], bool reorder) const (binding deprecated, see Section 15.2) g
28MPI_CART_CREATE returns a handle to a new communicator to which the Cartesian
29topology information is attached. If reorder = false then the rank of each process in the
30new group is identical to its rank in the old group. Otherwise, the function may reorder
31the processes (possibly so as to choose a good embedding of the virtual topology onto
32the physical machine). If the total size of the Cartesian grid is smaller than the size of
33the group of comm, then some processes are returned MPI_COMM_NULL, in analogy to
34MPI_COMM_SPLIT. If ndims is zero then a zero-dimensional Cartesian topology is created.
35The call is erroneous if it speci es a grid that is larger than the group size or if ndims is
36negative.
37
38 |
7.5.2 Cartesian Convenience Function: MPI_DIMS_CREATE |
|
|
||
39 |
For Cartesian topologies, the function MPI_DIMS_CREATE helps the user select a balanced |
|
40 |
||
distribution of processes per coordinate direction, depending on the number of processes |
||
41 |
||
in the group to be balanced and optional constraints that can be speci ed by the user. |
||
42 |
||
One use is to partition all the processes (the size of MPI_COMM_WORLD's group) into an |
||
43 |
||
n-dimensional topology. |
||
|
7.5. TOPOLOGY CONSTRUCTORS |
249 |
MPI_DIMS_CREATE(nnodes, ndims, dims)
IN |
nnodes |
number of nodes in a grid (integer) |
IN |
ndims |
number of Cartesian dimensions (integer) |
INOUT |
dims |
integer array of size ndims specifying the number of |
|
|
nodes in each dimension |
int MPI_Dims_create(int nnodes, int ndims, int *dims)
MPI_DIMS_CREATE(NNODES, NDIMS, DIMS, IERROR)
INTEGER NNODES, NDIMS, DIMS(*), IERROR
fvoid MPI::Compute_dims(int nnodes, int ndims, int dims[]) (binding deprecated, see Section 15.2) g
The entries in the array dims are set to describe a Cartesian grid with ndims dimensions and a total of nnodes nodes. The dimensions are set to be as close to each other as possible, using an appropriate divisibility algorithm. The caller may further constrain the operation of this routine by specifying elements of array dims. If dims[i] is set to a positive number, the routine will not modify the number of nodes in dimension i; only those entries where
dims[i] = |
0 are modi ed by the call. |
|
Negative input values of dims[i] are erroneous. An error will occur if nnodes is not a |
||
multiple of |
Y |
dims[i]. |
i;dims[i]6=0
For dims[i] set by the call, dims[i] will be ordered in non-increasing order. Array dims is suitable for use as input to routine MPI_CART_CREATE. MPI_DIMS_CREATE is local.
|
dims |
function call |
dims |
|
before call |
|
on return |
|
|
|
|
Example 7.1 |
(0,0) |
MPI_DIMS_CREATE(6, 2, dims) |
(3,2) |
|
(0,0) |
MPI_DIMS_CREATE(7, 2, dims) |
(7,1) |
|
(0,3,0) |
MPI_DIMS_CREATE(6, 3, dims) |
(2,3,1) |
|
(0,3,0) |
MPI_DIMS_CREATE(7, 3, dims) |
erroneous call |
|
|
|
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
1
2
3
4
5
250 |
CHAPTER 7. PROCESS TOPOLOGIES |
7.5.3 General (Graph) Constructor
MPI_GRAPH_CREATE(comm_old, nnodes, index, edges, reorder, comm_graph)
6
7
8
9
10
11
12
13
14
IN |
comm_old |
input communicator (handle) |
IN |
nnodes |
number of nodes in graph (integer) |
IN |
index |
array of integers describing node degrees (see below) |
IN |
edges |
array of integers describing graph edges (see below) |
IN |
reorder |
ranking may be reordered (true) or not (false) (logical) |
OUT |
comm_graph |
communicator with graph topology added (handle) |
int MPI_Graph_create(MPI_Comm comm_old, int nnodes, int *index, int *edges,
15
int reorder, MPI_Comm *comm_graph)
16 |
|
17 |
MPI_GRAPH_CREATE(COMM_OLD, NNODES, INDEX, EDGES, REORDER, COMM_GRAPH, |
|
|
18 |
IERROR) |
|
19INTEGER COMM_OLD, NNODES, INDEX(*), EDGES(*), COMM_GRAPH, IERROR
20LOGICAL REORDER
21
22
23
24
fMPI::Graphcomm MPI::Intracomm::Create_graph(int nnodes, const int index[], const int edges[], bool reorder) const (binding deprecated, see Section 15.2) g
25MPI_GRAPH_CREATE returns a handle to a new communicator to which the graph
26topology information is attached. If reorder = false then the rank of each process in the
27new group is identical to its rank in the old group. Otherwise, the function may reorder the
28processes. If the size, nnodes, of the graph is smaller than the size of the group of comm,
29then some processes are returned MPI_COMM_NULL, in analogy to MPI_CART_CREATE
30and MPI_COMM_SPLIT. If the graph is empty, i.e., nnodes == 0, then MPI_COMM_NULL
31is returned in all processes. The call is erroneous if it speci es a graph that is larger than
32the group size of the input communicator.
33The three parameters nnodes, index and edges de ne the graph structure. nnodes is
34the number of nodes of the graph. The nodes are numbered from 0 to nnodes-1. The
35i-th entry of array index stores the total number of neighbors of the rst i graph nodes.
36The lists of neighbors of nodes 0, 1, ..., nnodes-1 are stored in consecutive locations
37in array edges. The array edges is a attened representation of the edge lists. The total
38number of entries in index is nnodes and the total number of entries in edges is equal to the
39number of graph edges.
40The de nitions of the arguments nnodes, index, and edges are illustrated with the
41following simple example.
42
43 |
Example 7.2 Assume there are four processes 0, 1, 2, 3 with the following adjacency |
|
|
44 |
matrix: |
|
|
45 |
|
46 |
|
47 |
|
48
7.5. TOPOLOGY CONSTRUCTORS |
251 |
||||
|
|
|
|
|
|
|
process |
|
neighbors |
|
|
|
0 |
|
1, 3 |
|
|
|
1 |
|
0 |
|
|
|
2 |
|
3 |
|
|
|
3 |
|
0, 2 |
|
|
|
|
|
|
|
|
Then, the input arguments are: |
|
||||
|
nnodes = |
4 |
|
|
|
|
index = |
2, 3, 4, 6 |
|
edges = 1, 3, 0, 3, 0, 2
Thus, in C, index[0] is the degree of node zero, and index[i] - index[i-1] is the degree of node i, i=1, ..., nnodes-1; the list of neighbors of node zero is stored in edges[j], for 0 j index[0] 1 and the list of neighbors of node i, i > 0, is stored in edges[j], index[i 1] j index[i] 1.
In Fortran, index(1) is the degree of node zero, and index(i+1) - index(i) is the degree of node i, i=1, ..., nnodes-1; the list of neighbors of node zero is stored in edges(j), for 1 j index(1) and the list of neighbors of node i, i > 0, is stored in edges(j), index(i) + 1 j index(i + 1).
A single process is allowed to be de ned multiple times in the list of neighbors of a process (i.e., there may be multiple edges between two processes). A process is also allowed to be a neighbor to itself (i.e., a self loop in the graph). The adjacency matrix is allowed to be non-symmetric.
Advice to users. Performance implications of using multiple edges or a non-symmetric adjacency matrix are not de ned. The de nition of a node-neighbor edge does not imply a direction of the communication. (End of advice to users.)
Advice to implementors. The following topology information is likely to be stored with a communicator:
Type of topology (Cartesian/graph),
For a Cartesian topology:
1.ndims (number of dimensions),
2.dims (numbers of processes per coordinate direction),
3.periods (periodicity information),
4.own_position (own position in grid, could also be computed from rank and dims)
For a graph topology:
1.index,
2.edges,
which are the vectors de ning the graph structure.
For a graph structure the number of nodes is equal to the number of processes in the group. Therefore, the number of nodes does not have to be stored explicitly. An additional zero entry at the start of array index simpli es access to the topology information. (End of advice to implementors.)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48