A communication plan for scattering and gathering of ghosted data. More...

#include <Cabana_Halo.hpp>

Inheritance diagram for Cabana::Halo< MemorySpace, BuildType >:

Collaboration diagram for Cabana::Halo< MemorySpace, BuildType >:

Public Member Functions
template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Export >::value, int > = 0>
	Halo (MPI_Comm comm, const std::size_t num_local, const IdViewType &element_ids, const RankViewType &element_ranks, const std::vector< int > &neighbor_ranks)
	Neighbor and export rank constructor. Use this when you don't know who you will receiving from - only who you are sending to, but you already know which ranks neighbor each other (i.e. every rank already knows who they will be exporting to and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send and receive from the neighbors.

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Export >::value, int > = 0>
	Halo (MPI_Comm comm, const std::size_t num_local, const IdViewType &element_ids, const RankViewType &element_ranks)
	Export rank constructor. Use this when you don't know who you will receiving from - only who you are sending to. This is less efficient than if we already knew who our neighbors were because we have to determine the topology of the point-to-point communication first.

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Import >::value, int > = 0>
	Halo (MPI_Comm comm, const std::size_t num_local, const IdViewType &element_ids, const RankViewType &element_ranks, const std::vector< int > &neighbor_ranks)
	Neighbor and import rank constructor. Use this when you don't know who you will sending to - only who you are receiving from, but you already know which ranks neighbor each other (i.e. every rank already knows who they will be exporting to and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send and receive from the neighbors.

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Import >::value, int > = 0>
	Halo (MPI_Comm comm, const std::size_t num_local, const IdViewType &element_ids, const RankViewType &element_ranks)
	Import rank constructor. Use this when you don't know which ranks neighbor each other. (i.e. every rank does not already know who they will be exporting to and receiving from)

std::size_t	numLocal () const
	Get the number of elements locally owned by this rank.

std::size_t	numGhost () const
	Get the number of ghost elements this rank. Use this to resize a data structure for scatter/gather operations. For use with scatter gather, a data structure should be of size numLocal() + numGhost().

Public Member Functions inherited from Cabana::CommunicationPlan< MemorySpace >
	CommunicationPlan (MPI_Comm comm)
	Constructor.

MPI_Comm	comm () const
	Get the MPI communicator.

int	numNeighbor () const
	Get the number of neighbor ranks that this rank will communicate with.

int	neighborRank (const int neighbor) const
	Given a local neighbor id get its rank in the MPI communicator.

std::size_t	numExport (const int neighbor) const
	Get the number of elements this rank will export to a given neighbor.

std::size_t	totalNumExport () const
	Get the total number of exports this rank will do.

std::size_t	numImport (const int neighbor) const
	Get the number of elements this rank will import from a given neighbor.

std::size_t	totalNumImport () const
	Get the total number of imports this rank will do.

std::size_t	exportSize () const
	Get the number of export elements.

Kokkos::View< std::size_t *, memory_space >	getExportSteering () const
	Get the steering vector for the exports.

template<class ExecutionSpace, class RankViewType>
Kokkos::View< size_type *, memory_space >	createWithTopology (ExecutionSpace exec_space, Export, const RankViewType &element_export_ranks, const std::vector< int > &neighbor_ranks)
	Neighbor and export rank creator. Use this when you already know which ranks neighbor each other (i.e. every rank already knows who they will be sending and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send to and receive from the neighbors.

template<class RankViewType>
Kokkos::View< size_type *, memory_space >	createWithTopology (Export, const RankViewType &element_export_ranks, const std::vector< int > &neighbor_ranks)
	Neighbor and export rank creator. Use this when you already know which ranks neighbor each other (i.e. every rank already knows who they will be sending and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send to and receive from the neighbors.

template<class ExecutionSpace, class RankViewType>
Kokkos::View< size_type *, memory_space >	createWithoutTopology (ExecutionSpace exec_space, Export, const RankViewType &element_export_ranks)
	Export rank creator. Use this when you don't know who you will receiving from - only who you are sending to. This is less efficient than if we already knew who our neighbors were because we have to determine the topology of the point-to-point communication first.

template<class RankViewType>
Kokkos::View< size_type *, memory_space >	createWithoutTopology (Export, const RankViewType &element_export_ranks)
	Export rank creator. Use this when you don't know who you will receiving from - only who you are sending to. This is less efficient than if we already knew who our neighbors were because we have to determine the topology of the point-to-point communication first.

template<class ExecutionSpace, class RankViewType, class IdViewType>
auto	createWithTopology (ExecutionSpace exec_space, Import, const RankViewType &element_import_ranks, const IdViewType &element_import_ids, const std::vector< int > &neighbor_ranks) -> std::tuple< Kokkos::View< typename RankViewType::size_type , typename RankViewType::memory_space >, Kokkos::View< int , typename RankViewType::memory_space >, Kokkos::View< int *, typename IdViewType::memory_space > >
	Neighbor and import rank creator. Use this when you already know which ranks neighbor each other (i.e. every rank already knows who they will be sending and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send to and receive from the neighbors.

template<class RankViewType, class IdViewType>
auto	createWithTopology (Import, const RankViewType &element_import_ranks, const IdViewType &element_import_ids, const std::vector< int > &neighbor_ranks)
	Neighbor and import rank creator. Use this when you already know which ranks neighbor each other (i.e. every rank already knows who they will be sending and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send to and receive from the neighbors.

template<class ExecutionSpace, class RankViewType, class IdViewType>
auto	createWithoutTopology (ExecutionSpace exec_space, Import, const RankViewType &element_import_ranks, const IdViewType &element_import_ids) -> std::tuple< Kokkos::View< typename RankViewType::size_type , typename RankViewType::memory_space >, Kokkos::View< int , typename RankViewType::memory_space >, Kokkos::View< int *, typename IdViewType::memory_space > >
	Import rank creator. Use this when you don't know who you will be receiving from - only who you are importing from. This is less efficient than if we already knew who our neighbors were because we have to determine the topology of the point-to-point communication first.

template<class RankViewType, class IdViewType>
auto	createWithoutTopology (Import, const RankViewType &element_import_ranks, const IdViewType &element_import_ids)
	Import rank creator. Use this when you don't know who you will be receiving from - only who you are importing from. This is less efficient than if we already knew who our neighbors were because we have to determine the topology of the point-to-point communication first.

template<class PackViewType, class RankViewType>
void	createExportSteering (const PackViewType &neighbor_ids, const RankViewType &element_export_ranks)
	Create the export steering vector.

template<class PackViewType, class RankViewType, class IdViewType>
void	createExportSteering (const PackViewType &neighbor_ids, const RankViewType &element_export_ranks, const IdViewType &element_export_ids)
	Create the export steering vector.

Additional Inherited Members
Public Types inherited from Cabana::CommunicationPlan< MemorySpace >
using	memory_space = MemorySpace
	Kokkos memory space.

using	execution_space = typename memory_space::execution_space
	Default execution space.

using	size_type = typename memory_space::memory_space::size_type
	Size type.

Detailed Description

template<class MemorySpace, class BuildType = Export>
class Cabana::Halo< MemorySpace, BuildType >

A communication plan for scattering and gathering of ghosted data.

Template Parameters

MemorySpace Kokkos memory space in which data for this class will be allocated.

The halo allows for scatter and gather operations between locally-owned and ghosted data. All data in the Halo (e.g. export and import data) is from the point of view of the forward GATHER operation such that, for example, the number of exports is the number of exports in the gather and the number of imports is the number of imports in the gather. The reverse SCATTER operation sends the ghosted data back the the uniquely-owned decomposition and resolves collisions. Based on input for the forward communication plan (where local data will be sent) the local number of ghosts is computed. Some nomenclature:

Export - the local data we uniquely own that we will send to other ranks for those ranks to be used as ghosts. Export is used in the context of the forward communication plan (the gather).

Import - the ghost data that we get from other ranks. The rank we get a ghost from is the unique owner of that data. Import is used in the context of the forward communication plan (the gather).

Constructor & Destructor Documentation

◆ Halo() [1/4]

template<class MemorySpace, class BuildType = Export>

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Export >::value, int > = 0>

Cabana::Halo< MemorySpace, BuildType >::Halo	(	MPI_Comm	comm,
		const std::size_t	num_local,
		const IdViewType &	element_ids,
		const RankViewType &	element_ranks,
		const std::vector< int > &	neighbor_ranks )

inline

Neighbor and export rank constructor. Use this when you don't know who you will receiving from - only who you are sending to, but you already know which ranks neighbor each other (i.e. every rank already knows who they will be exporting to and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send and receive from the neighbors.

Template Parameters

IdViewType	The container type for the export element ids. This container type can be either a Kokkos View or a Cabana Slice.
RankViewType	The container type for the export element ranks. This container type can be either a Kokkos View or a Cabana Slice.
BuildType	must be Cabana::Export

Parameters

comm	The MPI communicator over which the halo is defined.
num_local	The number of locally-owned elements on this rank.
element_ids	The local ids of the elements that will be exported to other ranks to be used as ghosts. Element ids may be repeated in this list if they are sent to multiple destinations. Must be the same length as element_ranks. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.
element_ranks	The ranks to which we will send each element in element_ids. In this case each rank must be one of the neighbor ranks. Must be the same length as element_ids. A rank is allowed to send to itself. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.
neighbor_ranks	List of ranks this rank will send to and receive from. This list can include the calling rank. This is effectively a description of the topology of the point-to-point communication plan. The elements in this list must be unique.

◆ Halo() [2/4]

template<class MemorySpace, class BuildType = Export>

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Export >::value, int > = 0>

Cabana::Halo< MemorySpace, BuildType >::Halo	(	MPI_Comm	comm,
		const std::size_t	num_local,
		const IdViewType &	element_ids,
		const RankViewType &	element_ranks )

inline

Export rank constructor. Use this when you don't know who you will receiving from - only who you are sending to. This is less efficient than if we already knew who our neighbors were because we have to determine the topology of the point-to-point communication first.

Template Parameters

IdViewType	The container type for the export element ids. This container type can be either a Kokkos View or a Cabana Slice.
RankViewType	The container type for the export element ranks. This container type can be either a Kokkos View or a Cabana Slice.
BuildType	must be Cabana::Export

Parameters

comm	The MPI communicator over which the halo is defined.
num_local	The number of locally-owned elements on this rank.
element_ids	The local ids of the elements that will be sent to other ranks to be used as ghosts. Element ids may be repeated in this list if they are sent to multiple destinations. Must be the same length as element_ranks. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.
element_ranks	The ranks to which we will export each element in element_ids. Must be the same length as element_ids. The neighbor ranks will be determined from this list. A rank is allowed to send to itself. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.

◆ Halo() [3/4]

template<class MemorySpace, class BuildType = Export>

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Import >::value, int > = 0>

Cabana::Halo< MemorySpace, BuildType >::Halo	(	MPI_Comm	comm,
		const std::size_t	num_local,
		const IdViewType &	element_ids,
		const RankViewType &	element_ranks,
		const std::vector< int > &	neighbor_ranks )

inline

Neighbor and import rank constructor. Use this when you don't know who you will sending to - only who you are receiving from, but you already know which ranks neighbor each other (i.e. every rank already knows who they will be exporting to and receiving from) as it will be more efficient. In this case you already know the topology of the point-to-point communication but not how much data to send and receive from the neighbors.

Template Parameters

IdViewType	The container type for the export element ids. This container type can be either a Kokkos View or a Cabana Slice.
RankViewType	The container type for the export element ranks. This container type can be either a Kokkos View or a Cabana Slice.
BuildType	must be Cabana::Import

Parameters

comm	The MPI communicator over which the halo is defined.
num_local	The number of locally-owned elements on this rank.
element_ids	The local ids of the elements that will be imported from other ranks to be used as ghosts. Element ids may be repeated in this list if they are sent to multiple destinations. Must be the same length as element_ranks. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.
element_ranks	The ranks from which we will import each element in element_ids. In this case each rank must be one of the neighbor ranks. Must be the same length as element_ids. A rank is allowed to send to itself. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.
neighbor_ranks	List of ranks this rank will send to and receive from. This list can include the calling rank. This is effectively a description of the topology of the point-to-point communication plan. The elements in this list must be unique.

◆ Halo() [4/4]

template<class MemorySpace, class BuildType = Export>

template<class IdViewType, class RankViewType, typename T = BuildType, std::enable_if_t< std::is_same< T, Import >::value, int > = 0>

Cabana::Halo< MemorySpace, BuildType >::Halo	(	MPI_Comm	comm,
		const std::size_t	num_local,
		const IdViewType &	element_ids,
		const RankViewType &	element_ranks )

inline

Import rank constructor. Use this when you don't know which ranks neighbor each other. (i.e. every rank does not already know who they will be exporting to and receiving from)

Template Parameters

IdViewType	The container type for the export element ids. This container type can be either a Kokkos View or a Cabana Slice.
RankViewType	The container type for the export element ranks. This container type can be either a Kokkos View or a Cabana Slice.
BuildType	must be Cabana::Import

Parameters

comm	The MPI communicator over which the halo is defined.
num_local	The number of locally-owned elements on this rank.
element_ids	The local ids of the elements that will be imported from other ranks to be used as ghosts. Element ids may be repeated in this list if they are sent to multiple destinations. Must be the same length as element_ranks. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.
element_ranks	The ranks from which we will import each element in element_ids. In this case each rank must be one of the neighbor ranks. Must be the same length as element_ids. A rank is allowed to send to itself. The input is expected to be a Kokkos view or Cabana slice in the same memory space as the communication plan.

Member Function Documentation

◆ numGhost()

template<class MemorySpace, class BuildType = Export>

std::size_t Cabana::Halo< MemorySpace, BuildType >::numGhost ( ) const

inline

Get the number of ghost elements this rank. Use this to resize a data structure for scatter/gather operations. For use with scatter gather, a data structure should be of size numLocal() + numGhost().

Returns: The number of ghosted elements on this rank.

◆ numLocal()

template<class MemorySpace, class BuildType = Export>

std::size_t Cabana::Halo< MemorySpace, BuildType >::numLocal ( ) const

inline

Get the number of elements locally owned by this rank.

Returns: The number of elements locally owned by this rank.

The documentation for this class was generated from the following file:

core/src/Cabana_Halo.hpp

Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ Halo() [1/4]

◆ Halo() [2/4]

◆ Halo() [3/4]

◆ Halo() [4/4]

Member Function Documentation

◆ numGhost()

◆ numLocal()