- •Contents
- •List of Figures
- •List of Tables
- •Acknowledgments
- •Introduction to MPI
- •Overview and Goals
- •Background of MPI-1.0
- •Background of MPI-1.1, MPI-1.2, and MPI-2.0
- •Background of MPI-1.3 and MPI-2.1
- •Background of MPI-2.2
- •Who Should Use This Standard?
- •What Platforms Are Targets For Implementation?
- •What Is Included In The Standard?
- •What Is Not Included In The Standard?
- •Organization of this Document
- •MPI Terms and Conventions
- •Document Notation
- •Naming Conventions
- •Semantic Terms
- •Data Types
- •Opaque Objects
- •Array Arguments
- •State
- •Named Constants
- •Choice
- •Addresses
- •Language Binding
- •Deprecated Names and Functions
- •Fortran Binding Issues
- •C Binding Issues
- •C++ Binding Issues
- •Functions and Macros
- •Processes
- •Error Handling
- •Implementation Issues
- •Independence of Basic Runtime Routines
- •Interaction with Signals
- •Examples
- •Point-to-Point Communication
- •Introduction
- •Blocking Send and Receive Operations
- •Blocking Send
- •Message Data
- •Message Envelope
- •Blocking Receive
- •Return Status
- •Passing MPI_STATUS_IGNORE for Status
- •Data Type Matching and Data Conversion
- •Type Matching Rules
- •Type MPI_CHARACTER
- •Data Conversion
- •Communication Modes
- •Semantics of Point-to-Point Communication
- •Buffer Allocation and Usage
- •Nonblocking Communication
- •Communication Request Objects
- •Communication Initiation
- •Communication Completion
- •Semantics of Nonblocking Communications
- •Multiple Completions
- •Non-destructive Test of status
- •Probe and Cancel
- •Persistent Communication Requests
- •Send-Receive
- •Null Processes
- •Datatypes
- •Derived Datatypes
- •Type Constructors with Explicit Addresses
- •Datatype Constructors
- •Subarray Datatype Constructor
- •Distributed Array Datatype Constructor
- •Address and Size Functions
- •Lower-Bound and Upper-Bound Markers
- •Extent and Bounds of Datatypes
- •True Extent of Datatypes
- •Commit and Free
- •Duplicating a Datatype
- •Use of General Datatypes in Communication
- •Correct Use of Addresses
- •Decoding a Datatype
- •Examples
- •Pack and Unpack
- •Canonical MPI_PACK and MPI_UNPACK
- •Collective Communication
- •Introduction and Overview
- •Communicator Argument
- •Applying Collective Operations to Intercommunicators
- •Barrier Synchronization
- •Broadcast
- •Example using MPI_BCAST
- •Gather
- •Examples using MPI_GATHER, MPI_GATHERV
- •Scatter
- •Examples using MPI_SCATTER, MPI_SCATTERV
- •Example using MPI_ALLGATHER
- •All-to-All Scatter/Gather
- •Global Reduction Operations
- •Reduce
- •Signed Characters and Reductions
- •MINLOC and MAXLOC
- •All-Reduce
- •Process-local reduction
- •Reduce-Scatter
- •MPI_REDUCE_SCATTER_BLOCK
- •MPI_REDUCE_SCATTER
- •Scan
- •Inclusive Scan
- •Exclusive Scan
- •Example using MPI_SCAN
- •Correctness
- •Introduction
- •Features Needed to Support Libraries
- •MPI's Support for Libraries
- •Basic Concepts
- •Groups
- •Contexts
- •Intra-Communicators
- •Group Management
- •Group Accessors
- •Group Constructors
- •Group Destructors
- •Communicator Management
- •Communicator Accessors
- •Communicator Constructors
- •Communicator Destructors
- •Motivating Examples
- •Current Practice #1
- •Current Practice #2
- •(Approximate) Current Practice #3
- •Example #4
- •Library Example #1
- •Library Example #2
- •Inter-Communication
- •Inter-communicator Accessors
- •Inter-communicator Operations
- •Inter-Communication Examples
- •Caching
- •Functionality
- •Communicators
- •Windows
- •Datatypes
- •Error Class for Invalid Keyval
- •Attributes Example
- •Naming Objects
- •Formalizing the Loosely Synchronous Model
- •Basic Statements
- •Models of Execution
- •Static communicator allocation
- •Dynamic communicator allocation
- •The General case
- •Process Topologies
- •Introduction
- •Virtual Topologies
- •Embedding in MPI
- •Overview of the Functions
- •Topology Constructors
- •Cartesian Constructor
- •Cartesian Convenience Function: MPI_DIMS_CREATE
- •General (Graph) Constructor
- •Distributed (Graph) Constructor
- •Topology Inquiry Functions
- •Cartesian Shift Coordinates
- •Partitioning of Cartesian structures
- •Low-Level Topology Functions
- •An Application Example
- •MPI Environmental Management
- •Implementation Information
- •Version Inquiries
- •Environmental Inquiries
- •Tag Values
- •Host Rank
- •IO Rank
- •Clock Synchronization
- •Memory Allocation
- •Error Handling
- •Error Handlers for Communicators
- •Error Handlers for Windows
- •Error Handlers for Files
- •Freeing Errorhandlers and Retrieving Error Strings
- •Error Codes and Classes
- •Error Classes, Error Codes, and Error Handlers
- •Timers and Synchronization
- •Startup
- •Allowing User Functions at Process Termination
- •Determining Whether MPI Has Finished
- •Portable MPI Process Startup
- •The Info Object
- •Process Creation and Management
- •Introduction
- •The Dynamic Process Model
- •Starting Processes
- •The Runtime Environment
- •Process Manager Interface
- •Processes in MPI
- •Starting Processes and Establishing Communication
- •Reserved Keys
- •Spawn Example
- •Manager-worker Example, Using MPI_COMM_SPAWN.
- •Establishing Communication
- •Names, Addresses, Ports, and All That
- •Server Routines
- •Client Routines
- •Name Publishing
- •Reserved Key Values
- •Client/Server Examples
- •Ocean/Atmosphere - Relies on Name Publishing
- •Simple Client-Server Example.
- •Other Functionality
- •Universe Size
- •Singleton MPI_INIT
- •MPI_APPNUM
- •Releasing Connections
- •Another Way to Establish MPI Communication
- •One-Sided Communications
- •Introduction
- •Initialization
- •Window Creation
- •Window Attributes
- •Communication Calls
- •Examples
- •Accumulate Functions
- •Synchronization Calls
- •Fence
- •General Active Target Synchronization
- •Lock
- •Assertions
- •Examples
- •Error Handling
- •Error Handlers
- •Error Classes
- •Semantics and Correctness
- •Atomicity
- •Progress
- •Registers and Compiler Optimizations
- •External Interfaces
- •Introduction
- •Generalized Requests
- •Examples
- •Associating Information with Status
- •MPI and Threads
- •General
- •Initialization
- •Introduction
- •File Manipulation
- •Opening a File
- •Closing a File
- •Deleting a File
- •Resizing a File
- •Preallocating Space for a File
- •Querying the Size of a File
- •Querying File Parameters
- •File Info
- •Reserved File Hints
- •File Views
- •Data Access
- •Data Access Routines
- •Positioning
- •Synchronism
- •Coordination
- •Data Access Conventions
- •Data Access with Individual File Pointers
- •Data Access with Shared File Pointers
- •Noncollective Operations
- •Collective Operations
- •Seek
- •Split Collective Data Access Routines
- •File Interoperability
- •Datatypes for File Interoperability
- •Extent Callback
- •Datarep Conversion Functions
- •Matching Data Representations
- •Consistency and Semantics
- •File Consistency
- •Random Access vs. Sequential Files
- •Progress
- •Collective File Operations
- •Type Matching
- •Logical vs. Physical File Layout
- •File Size
- •Examples
- •Asynchronous I/O
- •I/O Error Handling
- •I/O Error Classes
- •Examples
- •Subarray Filetype Constructor
- •Requirements
- •Discussion
- •Logic of the Design
- •Examples
- •MPI Library Implementation
- •Systems with Weak Symbols
- •Systems Without Weak Symbols
- •Complications
- •Multiple Counting
- •Linker Oddities
- •Multiple Levels of Interception
- •Deprecated Functions
- •Deprecated since MPI-2.0
- •Deprecated since MPI-2.2
- •Language Bindings
- •Overview
- •Design
- •C++ Classes for MPI
- •Class Member Functions for MPI
- •Semantics
- •C++ Datatypes
- •Communicators
- •Exceptions
- •Mixed-Language Operability
- •Problems With Fortran Bindings for MPI
- •Problems Due to Strong Typing
- •Problems Due to Data Copying and Sequence Association
- •Special Constants
- •Fortran 90 Derived Types
- •A Problem with Register Optimization
- •Basic Fortran Support
- •Extended Fortran Support
- •The mpi Module
- •No Type Mismatch Problems for Subroutines with Choice Arguments
- •Additional Support for Fortran Numeric Intrinsic Types
- •Language Interoperability
- •Introduction
- •Assumptions
- •Initialization
- •Transfer of Handles
- •Status
- •MPI Opaque Objects
- •Datatypes
- •Callback Functions
- •Error Handlers
- •Reduce Operations
- •Addresses
- •Attributes
- •Extra State
- •Constants
- •Interlanguage Communication
- •Language Bindings Summary
- •Groups, Contexts, Communicators, and Caching Fortran Bindings
- •External Interfaces C++ Bindings
- •Change-Log
- •Bibliography
- •Examples Index
- •MPI Declarations Index
- •MPI Function Index
1
2
3
4
5
6
7
8
9
94 |
CHAPTER 4. DATATYPES |
array_of_distribs(1) = MPI_DISTRIBUTE_CYCLIC array_of_dargs(1) = 10
array_of_gsizes(2) = 200 array_of_distribs(2) = MPI_DISTRIBUTE_NONE array_of_dargs(2) = 0
array_of_gsizes(3) = 300 array_of_distribs(3) = MPI_DISTRIBUTE_BLOCK array_of_dargs(3) = MPI_DISTRIBUTE_DFLT_DARG array_of_psizes(1) = 2
10array_of_psizes(2) = 1
11array_of_psizes(3) = 3
12call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)
13call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
14call MPI_TYPE_CREATE_DARRAY(size, rank, ndims, array_of_gsizes, &
15
16
17
array_of_distribs, array_of_dargs, array_of_psizes, |
& |
MPI_ORDER_FORTRAN, oldtype, newtype, ierr)
18 |
4.1.5 Address and Size Functions |
|
|
19 |
|
20The displacements in a general datatype are relative to some initial bu er address. Abso-
21lute addresses can be substituted for these displacements: we treat them as displacements
22relative to \address zero," the start of the address space. This initial address zero is indi-
23cated by the constant MPI_BOTTOM. Thus, a datatype can specify the absolute address of
24the entries in the communication bu er, in which case the buf argument is passed the value
25MPI_BOTTOM.
26 |
The address of a location in memory can be found by invoking the function |
27 |
MPI_GET_ADDRESS. |
28
29
MPI_GET_ADDRESS(location, address)
30
31
32
33
34
35
IN |
location |
location in caller memory (choice) |
OUT |
address |
address of location (integer) |
int MPI_Get_address(void *location, MPI_Aint *address)
36MPI_GET_ADDRESS(LOCATION, ADDRESS, IERROR)
37<type> LOCATION(*)
38INTEGER IERROR
39INTEGER(KIND=MPI_ADDRESS_KIND) ADDRESS
40
fMPI::Aint MPI::Get_address(void* location) (binding deprecated, see Section 15.2)
41
g
42
43This function replaces MPI_ADDRESS, whose use is deprecated. See also Chapter 15.
44Returns the (byte) address of location.
45
46
47
48
Advice to users. to any system. in the program.
Current Fortran MPI codes will run unmodi ed, and will port However, they may fail if addresses larger than 232 1 are used New codes should be written so that they use the new functions.
4.1. DERIVED DATATYPES |
95 |
This provides compatibility with C/C++ and avoids errors on 64 bit architectures. However, such newly written codes may need to be (slightly) rewritten to port to old Fortran 77 environments that do not support KIND declarations. (End of advice to users.)
Example 4.8 Using MPI_GET_ADDRESS for an array.
REAL A(100,100)
INTEGER(KIND=MPI_ADDRESS_KIND) I1, I2, DIFF
CALL MPI_GET_ADDRESS(A(1,1), I1, IERROR)
CALL MPI_GET_ADDRESS(A(10,10), I2, IERROR)
DIFF = I2 - I1
!The value of DIFF is 909*sizeofreal; the values of I1 and I2 are
!implementation dependent.
Advice to users. C users may be tempted to avoid the usage of MPI_GET_ADDRESS and rely on the availability of the address operator &. Note, however, that & cast-expression is a pointer, not an address. ISO C does not require that the value of a pointer (or the pointer cast to int) be the absolute address of the object pointed at | although this is commonly the case. Furthermore, referencing may not have a unique de nition on machines with a segmented address space. The use of MPI_GET_ADDRESS to \reference" C variables guarantees portability to such machines as well. (End of advice to users.)
Advice to users. To prevent problems with the argument copying and register optimization done by Fortran compilers, please note the hints in subsections \Problems Due to Data Copying and Sequence Association," and \A Problem with Register Optimization" in Section 16.2.2 on pages 482 and 485. (End of advice to users.)
The following auxiliary function provides useful information on derived datatypes.
MPI_TYPE_SIZE(datatype, size)
IN |
datatype |
datatype (handle) |
OUT |
size |
datatype size (integer) |
int MPI_Type_size(MPI_Datatype datatype, int *size)
MPI_TYPE_SIZE(DATATYPE, SIZE, IERROR)
INTEGER DATATYPE, SIZE, IERROR
fint MPI::Datatype::Get_size() const (binding deprecated, see Section 15.2) g
MPI_TYPE_SIZE returns the total size, in bytes, of the entries in the type signature associated with datatype; i.e., the total size of the data in a message that would be created with this datatype. Entries that occur multiple times in the datatype are counted with their multiplicity.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
1
2
96 |
CHAPTER 4. DATATYPES |
4.1.6 Lower-Bound and Upper-Bound Markers
It is often convenient to de ne explicitly the lower bound and upper bound of a type map,
3
and override the de nition given on page 96. This allows one to de ne a datatype that has
4
\holes" at its beginning or its end, or a datatype with entries that extend above the upper
5
bound or below the lower bound. Examples of such usage are provided in Section 4.1.14.
6
Also, the user may want to overide the alignment rules that are used to compute upper
7
bounds and extents. E.g., a C compiler may allow the user to overide default alignment
8
rules for some of the structures within a program. The user has to specify explicitly the
9
bounds of the datatypes that match these structures.
10
To achieve this, we add two additional \pseudo-datatypes," MPI_LB and MPI_UB, that can be used, respectively, to mark the lower bound or the upper bound of a datatype. These pseudo-datatypes occupy no space (extent(MPI_LB) = extent(MPI_UB) = 0). They do not a ect the size or count of a datatype, and do not a ect the content of a message created with this datatype. However, they do a ect the de nition of the extent of a datatype and, therefore, a ect the outcome of a replication of this datatype by a datatype constructor.
Example 4.9 Let D = (-3, 0, 6); T = (MPI_LB, MPI_INT, MPI_UB), and B = (1, 1, 1). Then a call to MPI_TYPE_STRUCT(3, B, D, T, type1) creates a new datatype that has an extent of 9 (from -3 to 5, 5 included), and contains an integer at displacement 0. This is the datatype de ned by the sequence f(lb, -3), (int, 0), (ub, 6)g . If this type is replicated twice by a call to MPI_TYPE_CONTIGUOUS(2, type1, type2) then the newly created type can be described by the sequence f(lb, -3), (int, 0), (int,9), (ub, 15)g . (An entry of type ub
can be deleted if there is another entry of type ub with a higher displacement; an entry of
24
type lb can be deleted if there is another entry of type lb with a lower displacement.)
25
26
In general, if
27
28 |
T ypemap = f(type0; disp0); :::; (typen 1; dispn 1)g; |
|
|
29 |
|
30
31
32
33
34
35
36
37
38
39
40
41
then the lower bound of T ypemap is de ned to be |
|
|
|
||||
lb(T ypemap) = |
minj dispj |
|
|
if no entry has basic type lb |
|||
( minj |
f |
dispj such that typej = lb |
g |
|
otherwise |
||
|
|
|
|
|
|
||
Similarly, the upper bound of T ypemap is de ned to be |
|||||||
ub(T ypemap) = |
maxj dispj + sizeof(typej) + |
|
|
if no entry has basic type ub |
|||
( maxj |
f |
dispj such that typej = ub |
g |
otherwise |
|||
|
|
|
|
|
|
||
Then
extent(T ypemap) = ub(T ypemap) lb(T ypemap)
42 If typei requires alignment to a byte address that is a multiple of ki, then is the least 43 non-negative increment needed to round extent(T ypemap) to the next multiple of maxi ki.
44The formal de nitions given for the various datatype constructors apply now, with the
45amended de nition of extent.
46
47
48
4.1. DERIVED DATATYPES |
97 |
4.1.7 Extent and Bounds of Datatypes
The following function replaces the three functions MPI_TYPE_UB, MPI_TYPE_LB and MPI_TYPE_EXTENT. It also returns address sized integers, in the Fortran binding. The use of MPI_TYPE_UB, MPI_TYPE_LB and MPI_TYPE_EXTENT is deprecated.
MPI_TYPE_GET_EXTENT(datatype, lb, extent)
IN |
datatype |
datatype to get information on (handle) |
OUT |
lb |
lower bound of datatype (integer) |
OUT |
extent |
extent of datatype (integer) |
int MPI_Type_get_extent(MPI_Datatype datatype, MPI_Aint *lb, MPI_Aint *extent)
MPI_TYPE_GET_EXTENT(DATATYPE, LB, EXTENT, IERROR)
INTEGER DATATYPE, IERROR
INTEGER(KIND = MPI_ADDRESS_KIND) LB, EXTENT
fvoid MPI::Datatype::Get_extent(MPI::Aint& lb, MPI::Aint& extent) const
(binding deprecated, see Section 15.2) g
Returns the lower bound and the extent of datatype (as de ned in Section 4.1.6 on page 96).
MPI allows one to change the extent of a datatype, using lower bound and upper bound markers (MPI_LB and MPI_UB). This is useful, as it allows to control the stride of successive datatypes that are replicated by datatype constructors, or are replicated by the count argument in a send or receive call. However, the current mechanism for achieving it is painful; also it is restrictive. MPI_LB and MPI_UB are \sticky": once present in a datatype, they cannot be overridden (e.g., the upper bound can be moved up, by adding a new MPI_UB marker, but cannot be moved down below an existing MPI_UB marker). A new type constructor is provided to facilitate these changes. The use of MPI_LB and MPI_UB is deprecated.
MPI_TYPE_CREATE_RESIZED(oldtype, lb, extent, newtype)
IN |
oldtype |
input datatype (handle) |
IN |
lb |
new lower bound of datatype (integer) |
IN |
extent |
new extent of datatype (integer) |
OUT |
newtype |
output datatype (handle) |
int MPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb, MPI_Aint extent, MPI_Datatype *newtype)
MPI_TYPE_CREATE_RESIZED(OLDTYPE, LB, EXTENT, NEWTYPE, IERROR) INTEGER OLDTYPE, NEWTYPE, IERROR INTEGER(KIND=MPI_ADDRESS_KIND) LB, EXTENT
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48