Communicate halo data such as "ghost" atoms with neighboring tasks. More...

#include "haloExchange.h"
#include <assert.h>
#include "CoMDTypes.h"
#include "decomposition.h"
#include "parallel.h"
#include "linkCells.h"
#include "eam.h"
#include "memUtils.h"
#include "performanceTimers.h"

Include dependency graph for haloExchange.c:

Go to the source code of this file.

Data Structures
struct	AtomExchangeParmsSt
	Extra data members that are needed for the exchange of atom data. More...

struct	ForceExchangeParmsSt
	Extra data members that are needed for the exchange of force data. More...

struct	AtomMsgSt
	A structure to package data for a single atom to pack into a send/recv buffer. More...

struct	ForceMsgSt
	Package data for the force exchange. More...

Macros
#define	MAX(A, B) ((A) > (B) ? (A) : (B))

Typedefs
typedef struct AtomExchangeParmsSt	AtomExchangeParms
	Extra data members that are needed for the exchange of atom data. More...

typedef struct ForceExchangeParmsSt	ForceExchangeParms
	Extra data members that are needed for the exchange of force data. More...

typedef struct AtomMsgSt	AtomMsg
	A structure to package data for a single atom to pack into a send/recv buffer. More...

typedef struct ForceMsgSt	ForceMsg
	Package data for the force exchange. More...

Enumerations
enum	HaloFaceOrder { HALO_X_MINUS, HALO_X_PLUS, HALO_Y_MINUS, HALO_Y_PLUS, HALO_Z_MINUS, HALO_Z_PLUS }
	Don't change the order of the faces in this enum. More...

enum	HaloAxisOrder { HALO_X_AXIS, HALO_Y_AXIS, HALO_Z_AXIS }
	Don't change the order of the axes in this enum. More...

Functions
static HaloExchange *	initHaloExchange (Domain *domain)
	Base class constructor. More...

static void	exchangeData (HaloExchange haloExchange, void data, int iAxis)
	This is the function that does the heavy lifting for the communication of halo data. More...

static int *	mkAtomCellList (LinkCell *boxes, enum HaloFaceOrder iFace, const int nCells)
	Make a list of link cells that need to be sent across the specified face. More...

static int	loadAtomsBuffer (void vparms, void data, int face, char *charBuf)
	The loadBuffer function for a halo exchange of atom data. More...

static void	unloadAtomsBuffer (void vparms, void data, int face, int bufSize, char *charBuf)
	The unloadBuffer function for a halo exchange of atom data. More...

static void	destroyAtomsExchange (void *vparms)

static int *	mkForceSendCellList (LinkCell *boxes, int face, int nCells)
	Make a list of link cells that need to send data across the specified face. More...

static int *	mkForceRecvCellList (LinkCell *boxes, int face, int nCells)
	Make a list of link cells that need to receive data across the specified face. More...

static int	loadForceBuffer (void vparms, void vdata, int face, char *charBuf)
	The loadBuffer function for a force exchange. More...

static void	unloadForceBuffer (void vparms, void vdata, int face, int bufSize, char *charBuf)
	The unloadBuffer function for a force exchange. More...

static void	destroyForceExchange (void *vparms)

static int	sortAtomsById (const void a, const void b)
	A function suitable for passing to qsort to sort atoms by gid. More...

HaloExchange *	initAtomHaloExchange (Domain domain, LinkCell boxes)
	Create a HaloExchange for atom data. More...

HaloExchange *	initForceHaloExchange (Domain domain, LinkCell boxes)
	The force exchange is considerably simpler than the atom exchange. More...

void	destroyHaloExchange (HaloExchange **haloExchange)
	HaloExchange destructor. More...

void	haloExchange (HaloExchange haloExchange, void data)
	Execute a halo exchange. More...

void	sortAtomsInCell (Atoms atoms, LinkCell boxes, int iBox)
	Sort the atoms by gid in the specified link cell. More...

Detailed Description

Communicate halo data such as "ghost" atoms with neighboring tasks.

In addition to ghost atoms, the EAM potential also needs to exchange some force information. Hence this file implements both an atom exchange and a force exchange, each with slightly different properties due to their different roles.

The halo exchange in CoMD 1.1 takes advantage of the Cartesian domain decomposition as well as the link cell structure to quickly determine what data needs to be sent.

This halo exchange implementation is able to send data to all 26 neighboring tasks using only 6 messages. This is accomplished by sending data across the x-faces, then the y-faces, and finally across the z-faces. Some of the data that was received from the x-faces is included in the y-face sends and so on. This accumulation of data allows data to reach edge neighbors and corner neighbors by a two or three step process.

The advantage of this type of structured halo exchange is that it minimizes the number of MPI messages to send, and maximizes the size of those messages.

The disadvantage of this halo exchange is that it serializes message traffic. Only two messages can be in flight at once. The x-axis messages must be received and processed before the y-axis messages can begin. Architectures with low message latency and many off node network links would likely benefit from alternate halo exchange strategies that send independent messages to each neighbor task.

Definition in file haloExchange.c.

Macro Definition Documentation

#define MAX	(	A,
		B
	)	((A) > (B) ? (A) : (B))

Definition at line 43 of file haloExchange.c.

Typedef Documentation

typedef struct AtomExchangeParmsSt AtomExchangeParms

Extra data members that are needed for the exchange of atom data.

For an atom exchange, the HaloExchangeSt::parms will point to a structure of this type.

typedef struct AtomMsgSt AtomMsg

A structure to package data for a single atom to pack into a send/recv buffer.

Also used for sorting atoms within link cells.

typedef struct ForceExchangeParmsSt ForceExchangeParms

Extra data members that are needed for the exchange of force data.

For an force exchange, the HaloExchangeSt::parms will point to a structure of this type.

typedef struct ForceMsgSt ForceMsg

Package data for the force exchange.

Enumeration Type Documentation

enum HaloAxisOrder

Don't change the order of the axes in this enum.

Enumerator
HALO_X_AXIS
HALO_Y_AXIS
HALO_Z_AXIS

Definition at line 51 of file haloExchange.c.

enum HaloFaceOrder

Don't change the order of the faces in this enum.

Enumerator
HALO_X_MINUS
HALO_X_PLUS
HALO_Y_MINUS
HALO_Y_PLUS
HALO_Z_MINUS
HALO_Z_PLUS

Definition at line 46 of file haloExchange.c.

Function Documentation

void destroyAtomsExchange ( void * vparms )

static

Definition at line 427 of file haloExchange.c.

Here is the caller graph for this function:

void destroyForceExchange ( void * vparms )

static

Definition at line 593 of file haloExchange.c.

Here is the caller graph for this function:

void destroyHaloExchange ( HaloExchange ** haloExchange )

HaloExchange destructor.

Definition at line 239 of file haloExchange.c.

Here is the caller graph for this function:

void exchangeData	(	HaloExchange *	haloExchange,
		void *	data,
		int	iAxis
	)

static

This is the function that does the heavy lifting for the communication of halo data.

It is called once for each axis and sends and receives two message. Loading and unloading of the buffers is in the hands of the sub-class virtual functions.

Parameters

[in]	iAxis	Axis index.
[in,out]	data	Pointer to data that will be passed to the load and unload functions

Definition at line 277 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void haloExchange	(	HaloExchange *	haloExchange,
		void *	data
	)

Execute a halo exchange.

Definition at line 246 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

HaloExchange* initAtomHaloExchange	(	Domain *	domain,
		LinkCell *	boxes
	)

Create a HaloExchange for atom data.

When called in proper sequence by redistributeAtoms, the atom halo exchange helps serve three purposes:

Send ghost atom data to neighbor tasks.
Shift atom coordinates by the global simulation size when they cross periodic boundaries. This shift is performed in loadAtomsBuffer.
Transfer ownership of atoms between tasks as the atoms move across spatial domain boundaries. This transfer of ownership occurs in two places. The former owner gives up ownership when updateLinkCells moves a formerly local atom into a halo link cell. The new owner accepts ownership when unloadAtomsBuffer calls putAtomInBox to place a received atom into a local link cell.

This constructor does the following:

Sets the bufCapacity to hold the largest possible number of atoms that can be sent across a face.
Initialize function pointers to the atom-specific versions
Sets the number of link cells to send across each face.
Builds the list of link cells to send across each face. As explained in the comments for mkAtomCellList, this list must include any link cell, local or halo, that could possibly contain an atom that needs to be sent across the face. Atoms that need to be sent include "ghost atoms" that are located in local link cells that correspond to halo link cells on receiving tasks as well as formerly local atoms that have just moved into halo link cells and need to be sent to the rank that owns the spatial domain the atom has moved into.
Sets a coordinate shift factor for each face to account for periodic boundary conditions. For most faces the factor is zero. For faces on the +x, +y, or +z face of the simulation domain the factor is -1.0 (to shift the coordinates by -1 times the simulation domain size). For -x, -y, and -z faces of the simulation domain, the factor is +1.0.

See Also: redistributeAtoms

Definition at line 144 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

HaloExchange* initForceHaloExchange	(	Domain *	domain,
		LinkCell *	boxes
	)

The force exchange is considerably simpler than the atom exchange.

Create a HaloExchange for force data.

In the force case we only need to exchange data that is needed to complete the force calculation. Since the atoms have not moved we only need to send data from local link cells and we are guaranteed that the same atoms exist in the same order in corresponding halo cells on remote tasks. The only tricky part is the size of the plane of local cells that needs to be sent grows in each direction. This is because the y-axis send must send some of the data that was received from the x-axis send, and the z-axis must send some data from the y-axis send. This accumulation of data to send is responsible for data reaching neighbor cells that share only edges or corners.

See Also: eam.c for an explanation of the requirement to exchange force data.

Definition at line 205 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

HaloExchange * initHaloExchange ( Domain * domain )

static

Base class constructor.

Definition at line 253 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

int loadAtomsBuffer	(	void *	vparms,
		void *	data,
		int	face,
		char *	charBuf
	)

static

The loadBuffer function for a halo exchange of atom data.

Iterates link cells in the cellList and load any atoms into the send buffer. This function also shifts coordinates of the atoms by an appropriate factor if they are being sent across a periodic boundary.

See Also: HaloExchangeSt::loadBuffer for an explanation of the loadBuffer parameters.

Definition at line 360 of file haloExchange.c.

Here is the caller graph for this function:

int loadForceBuffer	(	void *	vparms,
		void *	vdata,
		int	face,
		char *	charBuf
	)

static

The loadBuffer function for a force exchange.

Iterate the send list and load the derivative of the embedding energy with respect to the local density into the send buffer.

See Also: HaloExchangeSt::loadBuffer for an explanation of the loadBuffer parameters.

Definition at line 542 of file haloExchange.c.

Here is the caller graph for this function:

int * mkAtomCellList	(	LinkCell *	boxes,
		enum HaloFaceOrder	iFace,
		const int	nCells
	)

static

Make a list of link cells that need to be sent across the specified face.

For each face, the list must include all cells, local and halo, in the first two planes of link cells. Halo cells must be included in the list of link cells to send since local atoms may have moved from local cells into halo cells on this time step. (Actual remote atoms should have been deleted, so the halo cells should contain only these few atoms that have just crossed.) Sending these atoms will allow them to be reassigned to the task that covers the spatial domain they have moved into.

Note that link cell grid coordinates range from -1 to gridSize[iAxis].

See Also: initLinkCells for an explanation link cell grid coordinates.

Parameters

[in]	boxes	Link cell information.
[in]	iFace	Index of the face data will be sent across.
[in]	nCells	Number of cells to send. This is used for a consistency check.

Returns: The list of cells to send. Caller is responsible to free the list.

Definition at line 327 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

int * mkForceRecvCellList	(	LinkCell *	boxes,
		int	face,
		int	nCells
	)

static

Make a list of link cells that need to receive data across the specified face.

Note that this list must be compatible with the corresponding send list to ensure that the data goes to the correct atoms.

See Also: initLinkCells for information about the conventions for grid coordinates of link cells.

Definition at line 494 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

int * mkForceSendCellList	(	LinkCell *	boxes,
		int	face,
		int	nCells
	)

static

Make a list of link cells that need to send data across the specified face.

Note that this list must be compatible with the corresponding recv list to ensure that the data goes to the correct atoms.

See Also: initLinkCells for information about the conventions for grid coordinates of link cells.

Definition at line 445 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

int sortAtomsById	(	const void *	a,
		const void *	b
	)

static

A function suitable for passing to qsort to sort atoms by gid.

Because every atom in the simulation is supposed to have a unique id, this function checks that the atoms have different gids. If that assertion ever fails it is a sign that something has gone wrong elsewhere in the code.

Definition at line 651 of file haloExchange.c.

Here is the caller graph for this function:

void sortAtomsInCell	(	Atoms *	atoms,
		LinkCell *	boxes,
		int	iBox
	)

Sort the atoms by gid in the specified link cell.

The force exchange assumes that the atoms are in the same order in both a given local link cell and the corresponding remote cell(s). However, the atom exchange does not guarantee this property, especially when atoms cross a domain decomposition boundary and move from one task to another. Trying to maintain the atom order during the atom exchange would immensely complicate that code. Instead, we just sort the atoms after the atom exchange.

Definition at line 612 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void unloadAtomsBuffer	(	void *	vparms,
		void *	data,
		int	face,
		int	bufSize,
		char *	charBuf
	)

static

The unloadBuffer function for a halo exchange of atom data.

Iterates the receive buffer and places each atom that was received into the link cell that corresponds to the atom coordinate. Note that this naturally accomplishes transfer of ownership of atoms that have moved from one spatial domain to another. Atoms with coordinates in local link cells automatically become local particles. Atoms that are owned by other ranks are automatically placed in halo kink cells.

See Also: HaloExchangeSt::unloadBuffer for an explanation of the unloadBuffer parameters.

Definition at line 405 of file haloExchange.c.

Here is the call graph for this function:

Here is the caller graph for this function:

void unloadForceBuffer	(	void *	vparms,
		void *	vdata,
		int	face,
		int	bufSize,
		char *	charBuf
	)

static

The unloadBuffer function for a force exchange.

Data is received in an order that naturally aligns with the atom storage so it is simple to put the data where it belongs.

See Also: HaloExchangeSt::unloadBuffer for an explanation of the unloadBuffer parameters.

Definition at line 570 of file haloExchange.c.

Here is the caller graph for this function:

Data Structures

Macros

Typedefs

Enumerations

Functions

Detailed Description

Macro Definition Documentation

Typedef Documentation

Enumeration Type Documentation

Function Documentation