MPI Application Analysis Configuration

To collect performance data for an MPI application, use the following command:

$ mpirun -n <n> -l amplxe-cl -result-dir my_result -quiet -collect <analysis type> my_app [my_app_ options]

where:

• <n> is the number of MPI processes to be run. As a result, the VTune Amplifier creates a number of result directories in the current directory, named as my_result.0, my_result.1, ... my_result.n, where the numeric suffix n corresponds to the MPI process rank. Using the suffix guarantees that multiple amplxe-cl instances launched in the same directory on different nodes do not overwrite the data of each other and can work in parallel. So, the VTune Amplifier creates a separate result directory for each analyzed process in the job.

  Note

  For hardware event-based sampling analysis types, the VTune Amplifier can collect single MPI ranks only. For example:

  $ mpirun -n 1 amplxe-cl -c advanced-hotspots -r ah -- ./test.x : -n 3 ./test.x

  To run event-based sampling analysis for several processes on one node, consider using -analyze-system option, for example:

  $ mpirun -host myhost -n 11 ./a.out : -host myhost -n 1 amplxe-cl -result-dir foo -c advanced-hotspots -analyze-system ./a.out 

  But this type of analysis configuration can collect ITT API (for example, Task and Frame analysis) and rank data only for the launched process.

• -l option of the mpiexec/mpirun tools marks stdout lines with an MPI rank.

• -quiet / -q option suppresses the diagnostic output like progress messages.

• <analysis type> is an analysis type you run with the VTune Amplifier. To view a list of available analysis types, use amplxe-cl -help collect command.

To collect data for a subset of MPI processes in the workload, use the per-host syntax of mpirun/mpiexec* and specify different command lines to execute for different processes.

If you use Intel MPI with version 5.0.2 or later you can use the -gtool option for easier selective rank profiling:

$ mpirun -genvall -gtool "amplxe-cl -r <my_result> -collect <analysis type>:<rank_set>[=exclusive]" -n <n> <my_app> [my_app_ options]

where <rank_set> specifies a ranks range no be involved to the tool execution. Separate ranks with a comma or use the “-” symbol for a set of contiguous ranks. Use the all value to configure profiling on all the ranks. exclusive launch mode helps prevent running more than one collection per node, which can be useful for PMU-based profiling. Starting with Intel MPI version 5.0.3 you can use the node-wide clause instead of exclusive to make collection on all ranks of the nodes where the <rank_set> resides on or for all nodes in the case of all ranks. In this case, the VTune Amplifier creates a result directory per node with a host name suffix for the result directory name. This is particularly convenient for PMU-based collections where there are limitations on simultaneous profiling by multiple amplxe-cl commands. For example:

$ mpirun -gtool "amplxe-cl -c advanced-hotspots -r my_dir:all=node-wide" -n 4 -ppn 2 my_mpi_app

Examples

1. This example runs the Hotspots analysis type recommended as a starting point:

   $ mpirun -n 4 amplxe-cl -result-dir my_result -collect hotspots -- my_app [my_app_ options]

2. This example collects Hotspots data for two out of 16 processes in the job distributed across the hosts:

   $ mpirun -host myhost -n 14 ./a.out : -host myhost -n 2 amplxe-cl -result-dir foo -c hotspots ./a.out

   As a result, the VTune Amplifier creates two directories in the current directory: foo.14 and foo.15 (given that process ranks 14 and 15 were assigned to the last two processes in the job).

3. As an alternative to the previous example, you can create a configuration file with the following content:

   # config.txt configuration file
   -host myhost -n 14 ./a.out
   -host myhost -n 2 amplxe-cl -quiet -collect hotspots -result-dir foo ./a.out

   and run the data collection as:

   $ mpirun -configfile ./config.txt

   to achieve the same result as in the previous example: foo.14 and foo.15 result directories are created.

4. Similarly, this example uses specific host names to control where the analyzed processes are executed:

   # config.txt configuration file
   -host myhost1 -n 14 ./a.out
   -host myhost2 -n 2 amplxe-cl -quiet -collect hotspots -result-dir foo ./a.out

   When you mention the host names, the VTune Amplifier allocates consecutive MPI ranks to the specified hosts. In this example, ranks 0 to 13, inclusive, will be assigned to myhost1. The remaining ranks 14 and 15 will be assigned to myhost2. You can omit specifying the exact hosts. In this case, the distribution of the processes between the hosts will be done in the round-robin fashion. That is, myhost1 will get MPI ranks 0, 2, and 4 thru 15, while myhost2 will get MPI ranks 1 and 3. The latter behavior may change in the future.

5. This example runs a system-wide Advanced Hotspot analysis on all nodes with Intel MPI 5.0.2, or later:

   $ mpirun –gtool “amplxe-cl -r my_result -collect advanced-hotspots –analyze-system:all=exclusive” –n 16 ppn -4 my_app [my_app_ options]

6. This example runs Advanced Hotspots analysis on 1, the first rank from 4-6 launched on node 2, rank 10:

   $ mpirun –gtool “amplxe-cl -r my_result -collect advanced-hotspots: 1,4-6,10” –n 16 ppn -4 my_app [my_app_ options]