- •Contents
- •List of Figures
- •List of Tables
- •Acknowledgments
- •Introduction to MPI
- •Overview and Goals
- •Background of MPI-1.0
- •Background of MPI-1.1, MPI-1.2, and MPI-2.0
- •Background of MPI-1.3 and MPI-2.1
- •Background of MPI-2.2
- •Who Should Use This Standard?
- •What Platforms Are Targets For Implementation?
- •What Is Included In The Standard?
- •What Is Not Included In The Standard?
- •Organization of this Document
- •MPI Terms and Conventions
- •Document Notation
- •Naming Conventions
- •Semantic Terms
- •Data Types
- •Opaque Objects
- •Array Arguments
- •State
- •Named Constants
- •Choice
- •Addresses
- •Language Binding
- •Deprecated Names and Functions
- •Fortran Binding Issues
- •C Binding Issues
- •C++ Binding Issues
- •Functions and Macros
- •Processes
- •Error Handling
- •Implementation Issues
- •Independence of Basic Runtime Routines
- •Interaction with Signals
- •Examples
- •Point-to-Point Communication
- •Introduction
- •Blocking Send and Receive Operations
- •Blocking Send
- •Message Data
- •Message Envelope
- •Blocking Receive
- •Return Status
- •Passing MPI_STATUS_IGNORE for Status
- •Data Type Matching and Data Conversion
- •Type Matching Rules
- •Type MPI_CHARACTER
- •Data Conversion
- •Communication Modes
- •Semantics of Point-to-Point Communication
- •Buffer Allocation and Usage
- •Nonblocking Communication
- •Communication Request Objects
- •Communication Initiation
- •Communication Completion
- •Semantics of Nonblocking Communications
- •Multiple Completions
- •Non-destructive Test of status
- •Probe and Cancel
- •Persistent Communication Requests
- •Send-Receive
- •Null Processes
- •Datatypes
- •Derived Datatypes
- •Type Constructors with Explicit Addresses
- •Datatype Constructors
- •Subarray Datatype Constructor
- •Distributed Array Datatype Constructor
- •Address and Size Functions
- •Lower-Bound and Upper-Bound Markers
- •Extent and Bounds of Datatypes
- •True Extent of Datatypes
- •Commit and Free
- •Duplicating a Datatype
- •Use of General Datatypes in Communication
- •Correct Use of Addresses
- •Decoding a Datatype
- •Examples
- •Pack and Unpack
- •Canonical MPI_PACK and MPI_UNPACK
- •Collective Communication
- •Introduction and Overview
- •Communicator Argument
- •Applying Collective Operations to Intercommunicators
- •Barrier Synchronization
- •Broadcast
- •Example using MPI_BCAST
- •Gather
- •Examples using MPI_GATHER, MPI_GATHERV
- •Scatter
- •Examples using MPI_SCATTER, MPI_SCATTERV
- •Example using MPI_ALLGATHER
- •All-to-All Scatter/Gather
- •Global Reduction Operations
- •Reduce
- •Signed Characters and Reductions
- •MINLOC and MAXLOC
- •All-Reduce
- •Process-local reduction
- •Reduce-Scatter
- •MPI_REDUCE_SCATTER_BLOCK
- •MPI_REDUCE_SCATTER
- •Scan
- •Inclusive Scan
- •Exclusive Scan
- •Example using MPI_SCAN
- •Correctness
- •Introduction
- •Features Needed to Support Libraries
- •MPI's Support for Libraries
- •Basic Concepts
- •Groups
- •Contexts
- •Intra-Communicators
- •Group Management
- •Group Accessors
- •Group Constructors
- •Group Destructors
- •Communicator Management
- •Communicator Accessors
- •Communicator Constructors
- •Communicator Destructors
- •Motivating Examples
- •Current Practice #1
- •Current Practice #2
- •(Approximate) Current Practice #3
- •Example #4
- •Library Example #1
- •Library Example #2
- •Inter-Communication
- •Inter-communicator Accessors
- •Inter-communicator Operations
- •Inter-Communication Examples
- •Caching
- •Functionality
- •Communicators
- •Windows
- •Datatypes
- •Error Class for Invalid Keyval
- •Attributes Example
- •Naming Objects
- •Formalizing the Loosely Synchronous Model
- •Basic Statements
- •Models of Execution
- •Static communicator allocation
- •Dynamic communicator allocation
- •The General case
- •Process Topologies
- •Introduction
- •Virtual Topologies
- •Embedding in MPI
- •Overview of the Functions
- •Topology Constructors
- •Cartesian Constructor
- •Cartesian Convenience Function: MPI_DIMS_CREATE
- •General (Graph) Constructor
- •Distributed (Graph) Constructor
- •Topology Inquiry Functions
- •Cartesian Shift Coordinates
- •Partitioning of Cartesian structures
- •Low-Level Topology Functions
- •An Application Example
- •MPI Environmental Management
- •Implementation Information
- •Version Inquiries
- •Environmental Inquiries
- •Tag Values
- •Host Rank
- •IO Rank
- •Clock Synchronization
- •Memory Allocation
- •Error Handling
- •Error Handlers for Communicators
- •Error Handlers for Windows
- •Error Handlers for Files
- •Freeing Errorhandlers and Retrieving Error Strings
- •Error Codes and Classes
- •Error Classes, Error Codes, and Error Handlers
- •Timers and Synchronization
- •Startup
- •Allowing User Functions at Process Termination
- •Determining Whether MPI Has Finished
- •Portable MPI Process Startup
- •The Info Object
- •Process Creation and Management
- •Introduction
- •The Dynamic Process Model
- •Starting Processes
- •The Runtime Environment
- •Process Manager Interface
- •Processes in MPI
- •Starting Processes and Establishing Communication
- •Reserved Keys
- •Spawn Example
- •Manager-worker Example, Using MPI_COMM_SPAWN.
- •Establishing Communication
- •Names, Addresses, Ports, and All That
- •Server Routines
- •Client Routines
- •Name Publishing
- •Reserved Key Values
- •Client/Server Examples
- •Ocean/Atmosphere - Relies on Name Publishing
- •Simple Client-Server Example.
- •Other Functionality
- •Universe Size
- •Singleton MPI_INIT
- •MPI_APPNUM
- •Releasing Connections
- •Another Way to Establish MPI Communication
- •One-Sided Communications
- •Introduction
- •Initialization
- •Window Creation
- •Window Attributes
- •Communication Calls
- •Examples
- •Accumulate Functions
- •Synchronization Calls
- •Fence
- •General Active Target Synchronization
- •Lock
- •Assertions
- •Examples
- •Error Handling
- •Error Handlers
- •Error Classes
- •Semantics and Correctness
- •Atomicity
- •Progress
- •Registers and Compiler Optimizations
- •External Interfaces
- •Introduction
- •Generalized Requests
- •Examples
- •Associating Information with Status
- •MPI and Threads
- •General
- •Initialization
- •Introduction
- •File Manipulation
- •Opening a File
- •Closing a File
- •Deleting a File
- •Resizing a File
- •Preallocating Space for a File
- •Querying the Size of a File
- •Querying File Parameters
- •File Info
- •Reserved File Hints
- •File Views
- •Data Access
- •Data Access Routines
- •Positioning
- •Synchronism
- •Coordination
- •Data Access Conventions
- •Data Access with Individual File Pointers
- •Data Access with Shared File Pointers
- •Noncollective Operations
- •Collective Operations
- •Seek
- •Split Collective Data Access Routines
- •File Interoperability
- •Datatypes for File Interoperability
- •Extent Callback
- •Datarep Conversion Functions
- •Matching Data Representations
- •Consistency and Semantics
- •File Consistency
- •Random Access vs. Sequential Files
- •Progress
- •Collective File Operations
- •Type Matching
- •Logical vs. Physical File Layout
- •File Size
- •Examples
- •Asynchronous I/O
- •I/O Error Handling
- •I/O Error Classes
- •Examples
- •Subarray Filetype Constructor
- •Requirements
- •Discussion
- •Logic of the Design
- •Examples
- •MPI Library Implementation
- •Systems with Weak Symbols
- •Systems Without Weak Symbols
- •Complications
- •Multiple Counting
- •Linker Oddities
- •Multiple Levels of Interception
- •Deprecated Functions
- •Deprecated since MPI-2.0
- •Deprecated since MPI-2.2
- •Language Bindings
- •Overview
- •Design
- •C++ Classes for MPI
- •Class Member Functions for MPI
- •Semantics
- •C++ Datatypes
- •Communicators
- •Exceptions
- •Mixed-Language Operability
- •Problems With Fortran Bindings for MPI
- •Problems Due to Strong Typing
- •Problems Due to Data Copying and Sequence Association
- •Special Constants
- •Fortran 90 Derived Types
- •A Problem with Register Optimization
- •Basic Fortran Support
- •Extended Fortran Support
- •The mpi Module
- •No Type Mismatch Problems for Subroutines with Choice Arguments
- •Additional Support for Fortran Numeric Intrinsic Types
- •Language Interoperability
- •Introduction
- •Assumptions
- •Initialization
- •Transfer of Handles
- •Status
- •MPI Opaque Objects
- •Datatypes
- •Callback Functions
- •Error Handlers
- •Reduce Operations
- •Addresses
- •Attributes
- •Extra State
- •Constants
- •Interlanguage Communication
- •Language Bindings Summary
- •Groups, Contexts, Communicators, and Caching Fortran Bindings
- •External Interfaces C++ Bindings
- •Change-Log
- •Bibliography
- •Examples Index
- •MPI Declarations Index
- •MPI Function Index
34 |
CHAPTER 3. POINT-TO-POINT COMMUNICATION |
1status elds. In these cases, it is a waste for the user to allocate a status object, and it is
2particularly wasteful for the MPI implementation to ll in elds in this object.
3To cope with this problem, there are two prede ned constants, MPI_STATUS_IGNORE
4and MPI_STATUSES_IGNORE, which when passed to a receive, wait, or test function, inform
5the implementation that the status elds are not to be lled in. Note that
6MPI_STATUS_IGNORE is not a special type of MPI_STATUS object; rather, it is a special
7value for the argument. In C one would expect it to be NULL, not the address of a special
8MPI_STATUS.
9MPI_STATUS_IGNORE, and the array version MPI_STATUSES_IGNORE, can be used every-
10where a status argument is passed to a receive, wait, or test function. MPI_STATUS_IGNORE
11cannot be used when status is an IN argument. Note that in Fortran MPI_STATUS_IGNORE
12and MPI_STATUSES_IGNORE are objects like MPI_BOTTOM (not usable for initialization or
13assignment). See Section 2.5.4.
14In general, this optimization can apply to all functions for which status or an array of
15statuses is an OUT argument. Note that this converts status into an INOUT argument. The
16functions that can be passed MPI_STATUS_IGNORE are all the various forms of MPI_RECV,
17MPI_TEST, and MPI_WAIT, as well as MPI_REQUEST_GET_STATUS. When an array is
18passed, as in the MPI_fTESTjWAITgfALLjSOMEg functions, a separate constant,
19MPI_STATUSES_IGNORE, is passed for the array argument. It is possible for an MPI function
20to return MPI_ERR_IN_STATUS even when MPI_STATUS_IGNORE or MPI_STATUSES_IGNORE
21has been passed to that function.
22MPI_STATUS_IGNORE and MPI_STATUSES_IGNORE are not required to have the same
23values in C and Fortran.
24It is not allowed to have some of the statuses in an array of statuses for
25MPI_fTESTjWAITgfALLjSOMEg functions set to MPI_STATUS_IGNORE; one either speci es
26ignoring all of the statuses in such a call with MPI_STATUSES_IGNORE, or none of them by
27passing normal statuses in all positions in the array of statuses.
28There are no C++ bindings for MPI_STATUS_IGNORE or MPI_STATUSES_IGNORE. To
29allow an OUT or INOUT MPI::Status argument to be ignored, all MPI C++ bindings that
30have OUT or INOUT MPI::Status parameters are overloaded with a second version that
31omits the OUT or INOUT MPI::Status parameter.
32
33Example 3.1 The C++ bindings for MPI_PROBE are:
34void MPI::Comm::Probe(int source, int tag, MPI::Status& status) const
35void MPI::Comm::Probe(int source, int tag) const
36 |
|
|
|
37 |
3.3 Data Type Matching and Data Conversion |
||
38 |
|||
|
|
||
39 |
3.3.1 |
Type Matching Rules |
|
|
|||
40 |
One can think of message transfer as consisting of the following three phases. |
||
41 |
|||
42 |
1. Data is pulled out of the send bu er and a message is assembled. |
||
|
|||
43 |
|
|
|
44 |
2. |
A message is transferred from sender to receiver. |
|
45 |
|
|
|
3. Data is pulled from the incoming message and disassembled into the receive bu er.
46
47Type matching has to be observed at each of these three phases: The type of each
48variable in the sender bu er has to match the type speci ed for that entry by the send
3.3. DATA TYPE MATCHING AND DATA CONVERSION |
35 |
operation; the type speci ed by the send operation has to match the type speci ed by the receive operation; and the type of each variable in the receive bu er has to match the type speci ed for that entry by the receive operation. A program that fails to observe these three rules is erroneous.
To de ne type matching more precisely, we need to deal with two issues: matching of types of the host language with types speci ed in communication operations; and matching of types at sender and receiver.
The types of a send and receive match (phase two) if both operations use identical names. That is, MPI_INTEGER matches MPI_INTEGER, MPI_REAL matches MPI_REAL, and so on. There is one exception to this rule, discussed in Section 4.2, the type MPI_PACKED can match any other type.
The type of a variable in a host program matches the type speci ed in the communication operation if the datatype name used by that operation corresponds to the basic type of the host program variable. For example, an entry with type name MPI_INTEGER matches a Fortran variable of type INTEGER. A table giving this correspondence for Fortran and C appears in Section 3.2.2. There are two exceptions to this last rule: an entry with type name MPI_BYTE or MPI_PACKED can be used to match any byte of storage (on a byte-addressable machine), irrespective of the datatype of the variable that contains this byte. The type MPI_PACKED is used to send data that has been explicitly packed, or receive data that will be explicitly unpacked, see Section 4.2. The type MPI_BYTE allows one to transfer the binary value of a byte in memory unchanged.
To summarize, the type matching rules fall into the three categories below.
Communication of typed values (e.g., with datatype di erent from MPI_BYTE), where the datatypes of the corresponding entries in the sender program, in the send call, in the receive call and in the receiver program must all match.
Communication of untyped values (e.g., of datatype MPI_BYTE), where both sender and receiver use the datatype MPI_BYTE. In this case, there are no requirements on the types of the corresponding entries in the sender and the receiver programs, nor is it required that they be the same.
Communication involving packed data, where MPI_PACKED is used.
The following examples illustrate the rst two cases.
Example 3.2 Sender and receiver specify matching types.
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr) ELSE IF (rank.EQ.1) THEN
CALL MPI_RECV(b(1), 15, MPI_REAL, 0, tag, comm, status, ierr) END IF
This code is correct if both a and b are real arrays of size 10. (In Fortran, it might be correct to use this code even if a or b have size < 10: e.g., when a(1) can be equivalenced to an array with ten reals.)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Example 3.3 Sender and receiver do not specify matching types. |
48 |
36 |
CHAPTER 3. POINT-TO-POINT COMMUNICATION |
1
2
CALL MPI_COMM_RANK(comm, rank, ierr) IF (rank.EQ.0) THEN
3CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
4 |
ELSE IF (rank.EQ.1) THEN |
|
5CALL MPI_RECV(b(1), 40, MPI_BYTE, 0, tag, comm, status, ierr)
6
7
END IF
8This code is erroneous, since sender and receiver do not provide matching datatype
9arguments.
10 |
|
11 |
Example 3.4 Sender and receiver specify communication of untyped values. |
12
CALL MPI_COMM_RANK(comm, rank, ierr)
13
IF (rank.EQ.0) THEN
14
CALL MPI_SEND(a(1), 40, MPI_BYTE, 1, tag, comm, ierr)
15
ELSE IF (rank.EQ.1) THEN
16
CALL MPI_RECV(b(1), 60, MPI_BYTE, 0, tag, comm, status, ierr)
17
END IF
18
19This code is correct, irrespective of the type and size of a and b (unless this results in
20an out of bound memory access).
21
22Advice to users. If a bu er of type MPI_BYTE is passed as an argument to MPI_SEND,
23then MPI will send the data stored at contiguous locations, starting from the address
24indicated by the buf argument. This may have unexpected results when the data
25layout is not as a casual user would expect it to be. For example, some Fortran
26compilers implement variables of type CHARACTER as a structure that contains the
27character length and a pointer to the actual string. In such an environment, sending
28and receiving a Fortran CHARACTER variable using the MPI_BYTE type will not have
29the anticipated result of transferring the character string. For this reason, the user is
30advised to use typed communications whenever possible. (End of advice to users.)
31 |
|
32 |
Type MPI_CHARACTER |
33
The type MPI_CHARACTER matches one character of a Fortran variable of type CHARACTER,
34
rather then the entire character string stored in the variable. Fortran variables of type
35
CHARACTER or substrings are transferred as if they were arrays of characters. This is
36
illustrated in the example below.
37 |
|
38 |
Example 3.5 Transfer of Fortran CHARACTERs. |
|
|
39 |
|
40 |
CHARACTER*10 a |
|
|
41 |
CHARACTER*10 b |
|
|
42 |
|
43CALL MPI_COMM_RANK(comm, rank, ierr)
44IF (rank.EQ.0) THEN
45CALL MPI_SEND(a, 5, MPI_CHARACTER, 1, tag, comm, ierr)
46ELSE IF (rank.EQ.1) THEN
47CALL MPI_RECV(b(6:10), 5, MPI_CHARACTER, 0, tag, comm, status, ierr)
48END IF