Sparse Matrix Indexing Benchmark

Matrix indexing is a primitive operation common to most packages that perform matrix operations. However currently there no existing performance benchmarks that comprehensively measure a packages ability to manipulate matrices. The Sparse Matrix Indexing Benchmark is an effort to address this lack and to provide essential performance characteristic metrics to the community. The SMIB architecture diagram is shown below. The system consists of four basic components. Central to these is the SMIB Benchmark itself which consists of the three kernels described below. In addition to the Benchmark component there is a Matrix Generator component. This component can be interchanged to provide performance testing on particular class of problem or the default generator can be used. The third component of SMIB is the Indexing component which generates indices for the tests described below. Finally there is a visualization component. The visualization component produces the final output of the SMIB testing suite. Systems are measured against a metric defined as pointers passed per second per processor. This will provide the user with a measure of the relative performance of various packages as related to their class of problem on their own hardware.

The SMIB contains three testing kernels that can be used to measure the indexing ability of a package in various ways. We have identified three indexing primitives and provide benchmarking kernels for each. The first kernel measures the efficiency with which a package can instantiate a matrix with non-zero elements initialized to values equal to the elements in a block of another matrix. The second kernel measures a packages efficiency while assigning values to a block in a matrix that are equal to the values of another matrix. Finally the third kernel measures a packages efficiency while assigning the values from a block in one matrix to a block ability to a block in another matrix. This is summarized in the table below:

Kernel	Primitive	Purpose
1	A = B(i, j)	Measure ability to copy a block from a matrix.
2	A(i, j) = B	Measure ability to copy a matrix to a block from a matrix.
3	A(i, j) = B(k, c)	Measure ability to copy a block from into a block of another matrix.

The following tests are available to be applied to all three Kernels. These benchmarking tests are designed to mimic indexing situations that would be found in typical scientific computing problems. We have identified 3 common operations and have designed test to measure performance related to each operation. These tests are summarized below. In addition to this we have identified several non-standard tests to measure a packages performance under conditions that are not as common in scientific computing problems.

The first test is designed to measure a packages ability to manipulate rows of data. By default SMIB will vary the row size in this test including row sizes smaller/larger than m/p x n for an m x n matrix on p processors.

The column test is complementary to the row test above in that it measures a packages ability to manipulate columns of data. Similar to the row test SMIB will by default apply this test on column sizes smaller and larger than m x n/p for an m x n matrix with p processors.

The grid test is a combination of the row and column test and measures a packages ability to manipulate blocks of data. As with the row and the column test SMIB will by default choose block sizes that are smaller and larger than m/p x n/p for an m x n matrix on a system with p processors.

This section consists of permutations of the above three tests. These tests are intended to cover situations that are not as common in scientific computing problems as the first three test but may be important to a particular problem. These tests consist of the above test but with a modified form of i and j. For instance we may wish to manipulate only a portion of the column or row so we will specify a range for m or n. In addition to this we may want to test the packages ability to manipulate portions of a row or column in which case we will specify a strided range or a permutation.