Reading Profile files in profile.* NODE 0;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 2 52,204 1 15 52204869 applu 96.6 1,210 50,454 3 37517 16818332 bcast_inputs 31.6 14,040 16,480 301 602 54752 rhs 19.4 8,238 10,122 9300 18600 1088 buts 16.3 7,776 8,513 9300 18600 915 blts 14.8 7,729 7,729 9300 0 831 jacld 12.2 6,369 6,369 9300 0 685 jacu 5.0 368 2,620 37200 37200 70 exchange_1 4.7 485 2,451 604 1812 4058 exchange_3 3.2 1,681 1,681 18600 0 90 MPI_Recv() 3.2 1,658 1,658 608 0 2728 MPI_Wait() 1.9 1,010 1,010 1 0 1010112 MPI_Finalize() 1.7 861 861 19204 0 45 MPI_Send() 0.7 234 351 1 47616 351657 setiv 0.3 148 148 57252 0 3 exact 0.2 116 117 1 2 117120 read_input 0.2 14 114 1 4 114542 init_comm 0.2 99 99 1 0 99827 MPI_Init() 0.1 62 74 1 2 74333 erhs 0.1 39 59 1 7937 59173 error 0.0 25 25 8 0 3242 MPI_Allreduce() 0.0 8 19 1 1700 19539 setbv 0.0 16 16 608 0 27 MPI_Irecv() 0.0 3 4 3 3 1632 l2norm 0.0 0.894 1 1 6 1502 pintgr 0.0 0.547 0.547 9 0 61 MPI_Bcast() 0.0 0.253 0.253 1 0 253 sethyper 0.0 0.0368 0.243 1 4 243 exchange_4 0.0 0.015 0.0978 1 2 98 exchange_6 0.0 0.0919 0.0919 1 0 92 MPI_Barrier() 0.0 0.0495 0.0495 2 0 25 MPI_Wtime() 0.0 0.0175 0.0487 1 2 49 exchange_5 0.0 0.0286 0.0286 2 0 14 MPI_Comm_size() 0.0 0.0186 0.0186 1 0 19 setcoeff 0.0 0.00953 0.00953 1 0 10 subdomain 0.0 0.00391 0.00391 1 0 4 MPI_Comm_rank() 0.0 0.00311 0.00311 1 0 3 proc_grid 0.0 0.00283 0.00283 1 0 3 neighbors 0.0 0.00166 0.00166 1 0 2 nodedim --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 1.92E+04 4.488E+04 640 2031 7721 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 9602 4.488E+04 640 2031 7721 Message size sent to node 1 9602 4.488E+04 640 2031 7721 Message size sent to node 2 0 0 0 0 0 Message size sent to node 3 --------------------------------------------------------------------------------------- NODE 1;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 0.648 52,204 1 14 52204862 applu 96.8 1,135 50,553 2 37517 25276626 bcast_inputs 31.2 13,047 16,307 301 602 54176 rhs 20.4 7,283 10,674 9300 18600 1148 blts 17.5 7,785 9,122 9300 18600 981 buts 13.8 7,187 7,187 9300 0 773 jacld 11.4 5,965 5,965 9300 0 641 jacu 9.1 365 4,727 37200 37200 127 exchange_1 7.4 3,850 3,850 18600 0 207 MPI_Recv() 6.3 476 3,289 604 1812 5446 exchange_3 4.7 2,455 2,455 606 0 4052 MPI_Wait() 2.0 1,035 1,035 1 0 1035340 MPI_Finalize() 1.6 842 842 19206 0 44 MPI_Send() 0.6 221 331 1 44640 331447 setiv 0.3 139 139 53713 0 3 exact 0.2 0.0249 117 1 1 117979 read_input 0.2 117 117 9 0 13098 MPI_Bcast() 0.2 13 114 1 4 114515 init_comm 0.2 100 100 1 0 100934 MPI_Init() 0.2 59 89 1 2 89473 erhs 0.1 37 59 1 7441 59175 error 0.1 43 43 8 0 5410 MPI_Allreduce() 0.1 26 26 606 0 44 MPI_Irecv() 0.0 7 19 1 1633 19237 setbv 0.0 3 18 3 3 6050 l2norm 0.0 0.901 1 1 6 1491 pintgr 0.0 0.23 0.23 1 0 230 sethyper 0.0 0.0381 0.15 1 3 150 exchange_4 0.0 0.0905 0.0905 1 0 90 MPI_Barrier() 0.0 0.0158 0.0654 1 2 65 exchange_6 0.0 0.0435 0.0435 2 0 22 MPI_Wtime() 0.0 0.00928 0.0292 1 1 29 exchange_5 0.0 0.0244 0.0244 1 0 24 MPI_Comm_size() 0.0 0.0146 0.0146 1 0 15 setcoeff 0.0 0.00861 0.00861 1 0 9 subdomain 0.0 0.00393 0.00393 1 0 4 proc_grid 0.0 0.00384 0.00384 1 0 4 MPI_Comm_rank() 0.0 0.00309 0.00309 1 0 3 neighbors 0.0 0.00164 0.00164 1 0 2 nodedim --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 1.921E+04 4.488E+04 264 1970 7498 Message size sent to all nodes 9604 4.488E+04 264 2031 7721 Message size sent to node 0 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 2 9602 4.224E+04 600 1910 7268 Message size sent to node 3 --------------------------------------------------------------------------------------- NODE 2;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 0.81 52,205 1 14 52205224 applu 96.8 1,142 50,552 2 37517 25276053 bcast_inputs 31.7 12,893 16,560 301 602 55018 rhs 19.6 7,175 10,230 9300 18600 1100 blts 17.8 7,951 9,271 9300 18600 997 buts 13.8 7,193 7,193 9300 0 774 jacld 11.5 5,996 5,996 9300 0 645 jacu 8.4 410 4,374 37200 37200 118 exchange_1 7.1 473 3,693 604 1812 6115 exchange_3 6.6 3,462 3,462 18600 0 186 MPI_Recv() 5.5 2,866 2,866 606 0 4730 MPI_Wait() 2.0 1,035 1,035 1 0 1035342 MPI_Finalize() 1.6 833 833 19206 0 43 MPI_Send() 0.6 225 336 1 44640 336506 setiv 0.3 140 140 53713 0 3 exact 0.2 0.0317 118 1 1 118215 read_input 0.2 118 118 9 0 13116 MPI_Bcast() 0.2 13 114 1 4 114905 init_comm 0.2 100 100 1 0 100945 MPI_Init() 0.2 60 86 1 2 86464 erhs 0.1 37 59 1 7441 59173 error 0.1 38 38 8 0 4854 MPI_Allreduce() 0.0 21 21 606 0 36 MPI_Irecv() 0.0 7 18 1 1633 18169 setbv 0.0 3 14 3 3 4796 l2norm 0.0 1 1 1 6 1479 pintgr 0.0 0.21 0.21 1 0 210 sethyper 0.0 0.0496 0.167 1 3 167 exchange_4 0.0 0.0895 0.0895 1 0 89 MPI_Barrier() 0.0 0.0249 0.0615 1 2 62 exchange_5 0.0 0.0139 0.0464 1 1 46 exchange_6 0.0 0.0431 0.0431 2 0 22 MPI_Wtime() 0.0 0.024 0.024 1 0 24 MPI_Comm_size() 0.0 0.0141 0.0141 1 0 14 setcoeff 0.0 0.00775 0.00775 1 0 8 subdomain 0.0 0.00393 0.00393 1 0 4 MPI_Comm_rank() 0.0 0.00275 0.00275 1 0 3 neighbors 0.0 0.00273 0.00273 1 0 3 proc_grid 0.0 0.00166 0.00166 1 0 2 nodedim --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 1.921E+04 4.488E+04 264 1970 7498 Message size sent to all nodes 9604 4.488E+04 264 2031 7721 Message size sent to node 0 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 2 9602 4.224E+04 600 1910 7268 Message size sent to node 3 --------------------------------------------------------------------------------------- NODE 3;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 0.618 52,204 1 14 52204833 applu 96.8 1,098 50,555 2 37517 25277989 bcast_inputs 32.0 12,156 16,690 301 602 55450 rhs 23.4 6,889 12,227 9300 18600 1315 blts 15.3 7,336 7,972 9300 18600 857 buts 12.9 6,743 6,743 9300 0 725 jacld 11.4 380 5,974 37200 37200 161 exchange_1 10.8 5,653 5,653 9300 0 608 jacu 9.8 5,112 5,112 18600 0 275 MPI_Recv() 8.8 438 4,582 604 1812 7587 exchange_3 7.5 3,892 3,892 604 0 6445 MPI_Wait() 2.0 1,035 1,035 1 0 1035336 MPI_Finalize() 1.3 687 687 19208 0 36 MPI_Send() 0.6 208 311 1 41850 311256 setiv 0.3 131 131 50393 0 3 exact 0.2 0.0253 118 1 1 118274 read_input 0.2 118 118 9 0 13131 MPI_Bcast() 0.2 13 114 1 4 114468 init_comm 0.2 58 107 1 2 107594 erhs 0.2 101 101 1 0 101065 MPI_Init() 0.1 34 59 1 6976 59169 error 0.1 54 54 8 0 6824 MPI_Allreduce() 0.1 44 44 604 0 74 MPI_Irecv() 0.1 4 26 3 3 8911 l2norm 0.0 7 18 1 1568 18628 setbv 0.0 0.859 1 1 6 1492 pintgr 0.0 0.24 0.24 1 0 240 sethyper 0.0 0.03 0.111 1 2 111 exchange_4 0.0 0.0773 0.0773 1 0 77 MPI_Barrier() 0.0 0.0414 0.0414 2 0 21 MPI_Wtime() 0.0 0.0327 0.0327 1 0 33 MPI_Comm_size() 0.0 0.00968 0.0305 1 1 31 exchange_6 0.0 0.00977 0.0287 1 1 29 exchange_5 0.0 0.014 0.014 1 0 14 setcoeff 0.0 0.00779 0.00779 1 0 8 subdomain 0.0 0.00416 0.00416 1 0 4 MPI_Comm_rank() 0.0 0.00319 0.00319 1 0 3 proc_grid 0.0 0.00302 0.00302 1 0 3 neighbors 0.0 0.00168 0.00168 1 0 2 nodedim --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 1.921E+04 4.224E+04 256 1909 7267 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 9604 4.224E+04 256 1909 7267 Message size sent to node 1 9604 4.224E+04 264 1909 7267 Message size sent to node 2 0 0 0 0 0 Message size sent to node 3 --------------------------------------------------------------------------------------- FUNCTION SUMMARY (total): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 4 3:28.819 4 57 52204947 applu 96.8 4,587 3:22.116 9 150068 22457370 bcast_inputs 31.6 52,138 1:06.038 1204 2408 54849 rhs 19.9 29,124 41,645 37200 74400 1120 blts 17.5 31,312 36,488 37200 74400 981 buts 13.8 28,853 28,853 37200 0 776 jacld 11.5 23,984 23,984 37200 0 645 jacu 8.5 1,525 17,696 148800 148800 119 exchange_1 6.8 14,105 14,105 74400 0 190 MPI_Recv() 6.7 1,875 14,016 2416 7248 5801 exchange_3 5.2 10,872 10,872 2424 0 4485 MPI_Wait() 2.0 4,116 4,116 4 0 1029033 MPI_Finalize() 1.5 3,225 3,225 76824 0 42 MPI_Send() 0.6 889 1,330 4 178746 332717 setiv 0.3 559 559 215071 0 3 exact 0.2 116 471 4 5 117897 read_input 0.2 55 458 4 16 114608 init_comm 0.2 402 402 4 0 100693 MPI_Init() 0.2 241 357 4 8 89466 erhs 0.2 354 354 36 0 9852 MPI_Bcast() 0.1 148 236 4 29795 59172 error 0.1 162 162 32 0 5083 MPI_Allreduce() 0.1 109 109 2424 0 45 MPI_Irecv() 0.0 31 75 4 6534 18893 setbv 0.0 15 64 12 12 5347 l2norm 0.0 3 5 4 24 1491 pintgr 0.0 0.933 0.933 4 0 233 sethyper 0.0 0.155 0.671 4 12 168 exchange_4 0.0 0.349 0.349 4 0 87 MPI_Barrier() 0.0 0.0544 0.24 4 6 60 exchange_6 0.0 0.177 0.177 8 0 22 MPI_Wtime() 0.0 0.0615 0.168 4 6 42 exchange_5 0.0 0.11 0.11 5 0 22 MPI_Comm_size() 0.0 0.0613 0.0613 4 0 15 setcoeff 0.0 0.0337 0.0337 4 0 8 subdomain 0.0 0.0158 0.0158 4 0 4 MPI_Comm_rank() 0.0 0.013 0.013 4 0 3 proc_grid 0.0 0.0117 0.0117 4 0 3 neighbors 0.0 0.00664 0.00664 4 0 2 nodedim FUNCTION SUMMARY (mean): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 1 52,204 1 14.25 52204947 applu 96.8 1,146 50,529 2.25 37517 22457370 bcast_inputs 31.6 13,034 16,509 301 602 54849 rhs 19.9 7,281 10,411 9300 18600 1120 blts 17.5 7,828 9,122 9300 18600 981 buts 13.8 7,213 7,213 9300 0 776 jacld 11.5 5,996 5,996 9300 0 645 jacu 8.5 381 4,424 37200 37200 119 exchange_1 6.8 3,526 3,526 18600 0 190 MPI_Recv() 6.7 468 3,504 604 1812 5801 exchange_3 5.2 2,718 2,718 606 0 4485 MPI_Wait() 2.0 1,029 1,029 1 0 1029033 MPI_Finalize() 1.5 806 806 19206 0 42 MPI_Send() 0.6 222 332 1 44686.5 332717 setiv 0.3 139 139 53767.8 0 3 exact 0.2 29 117 1 1.25 117897 read_input 0.2 13 114 1 4 114608 init_comm 0.2 100 100 1 0 100693 MPI_Init() 0.2 60 89 1 2 89466 erhs 0.2 88 88 9 0 9852 MPI_Bcast() 0.1 37 59 1 7448.75 59172 error 0.1 40 40 8 0 5083 MPI_Allreduce() 0.1 27 27 606 0 45 MPI_Irecv() 0.0 7 18 1 1633.5 18893 setbv 0.0 3 16 3 3 5347 l2norm 0.0 0.914 1 1 6 1491 pintgr 0.0 0.233 0.233 1 0 233 sethyper 0.0 0.0386 0.168 1 3 168 exchange_4 0.0 0.0873 0.0873 1 0 87 MPI_Barrier() 0.0 0.0136 0.06 1 1.5 60 exchange_6 0.0 0.0444 0.0444 2 0 22 MPI_Wtime() 0.0 0.0154 0.042 1 1.5 42 exchange_5 0.0 0.0274 0.0274 1.25 0 22 MPI_Comm_size() 0.0 0.0153 0.0153 1 0 15 setcoeff 0.0 0.00842 0.00842 1 0 8 subdomain 0.0 0.00396 0.00396 1 0 4 MPI_Comm_rank() 0.0 0.00324 0.00324 1 0 3 proc_grid 0.0 0.00292 0.00292 1 0 3 neighbors 0.0 0.00166 0.00166 1 0 2 nodedim