INFINIBAND SPEED CHECK | INFINIBAND on Cluster

1)Whole cluster Network performance depends on the complete cluster Infiniband network cabling.
For single server Infiniband performance we can check by using ibstat,
But How can we check complete cluster server Infiniband Performance ???? Still unresolved.
2)While Running the parallel job in the cluster.if any one of the server cabling is fault.then it will affect all other server communication.it is like SERIAL COMMUNICAITON.

Example : Prasanth Hypre Job.
-------------------------------------------------------------------------------------------------------------------------
INTEL MPI Benchmark.
1)Test the CONNECTIVITY and PERFORMANCE.
LINK 1 ) https://dvinfo.ifh.de/Run_Intel_MPI_Benchmarks
LINK 2 ) https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/IBM%20Platform%20HPC%20Wiki/page/Use%20Intel%20MPI%20benchmarks
LINK 3 ) http://software.intel.com/en-us/articles/using-intel-cluster-checker-to-check-that-mpi-applications-will-properly-run-over
2)Is try to find the performance of the cluster network Also (MPI library Implementation and different compiler on communication performance).

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Welcome%20to%20High%20Performance%20Computing%20%28HPC%29%20Central/page/Mellanox%20Fabric%20Link%20Diagnosis
---------------------------------------------------------------------------------------
Intel MPI Benchmark Analysis Data : https://sites.google.com/site/hpchcl/others/Benchmark-Intel%20MPI%20Benchmark.doc?attredirects=0&d=1

To Check The Infiniband Diagnosis Utility command #ibdiagnet 

Here Infiniband speed/width check we are getting 3 errors. But it should be 0.
it is good Condition for infiniband cluster.