What: Move BTLs into separate layer Why: Several projects have expressed interest to use the BTLs. Use-cases such as the RTE using the BTLs for modex or tools collecting/distributing data in the fastest possible way may be possible.
Where: This would affect several components, that the BTLs depend on (namely allocator, mpool, rcache and the common part of the BTLs). Additionally some changes to classes were/are necessary. When: Preferably 1.5 (in case we use the Feature/Stable Release cycle ;-) Timeout: 23.03.2009 ------------------------------------------------------------------------ There has been much speculation about this project. This RFC should shed some light, if there is some more information required, please feel free to ask/comment. Of course, suggestions are welcome! The BTLs offer access to fast communication framework. Several projects have expressed interest to use them separate of other layers of Open MPI. Additionally (with further changes) BTLs maybe used within ORTE itself. COURSE OF WORK: The extraction is not easy (as was the extraction of ORTE and OMPI in the early stages of Open MPI?). In order to get as much input and be as visible as possible (e.g. in TRACS), the tmp-branch for this work has been set up on: https://svn.open-mpi.org/svn/ompi/tmp/koenig-btl We propose to have a separate ONET library living in onet, based on orte (see attached fig). In order to keep the diff between the trunk and the branch to a minimum several cleanup patches have already been applied to the trunk (e.g. unnecessary #include of ompi and orte header files, integration of ompi_bitmap_t into opal_bitmap_t, #include "*_config.h"). Additionally a script (attached below) has been kept up-to-date (contrib/move- btl-into-onet), that will perform this separation on a fresh checkout of trunk: svn list https://svn.open-mpi.org/svn/ompi/tmp/koenig-btl/contrib/move-btl- into-onet This script requires several patches (see attached TAR-ball). Please update the variable PATCH_DIR to match the location of patches. ./move-btl-into-onet ompi-clean/ # Lots of output deleted. cd ompi-clean/ rm -fr ompi/mca/common/ # No two mcas called common, too bad... ./autogen.sh OTHER RTEs: A preliminary header file is provided in onet/include/rte.h to accommodate the requirements of other RTEs (such as stci), that replaces selected functionality, as proposed by Jeff and Ralph in the Louisville meeting. Additionally, this header file is included before orte-header files (within onet)... By default, this does not change anything in the standard case (ORTE), otherwise -DHAVE_STCI, redefinitions for components orte-functionality required within onet is done. TESTS: First tests have been done locally on Linux/x86_64. The branch compiles without warnings. The wrappers have been updated. The Intel Testsuite runs without failures: ./run-tests.pl all_tests_no_perf PERFORMANCE: !!!Before any merge, do extensive performance tests on real machines!!! Initial tests on the cluster smoky, show no difference in comparison to ompi- trunk. Please see the enclosed output of NetPipe-3.7.1 run on a single node (--mca btl sm,self) on smoky. TODOS: There are still some todos, to finalize this: - Dependencies in the onet-layer into the ompi-layer (ompi_proc_t, ompi_converter) We are working on these, and have shortly talked about the latter with George. - Better abstraction from orte / cleanups, such as modex If these involve code-changes (and not just "save" and non-intrusive renames), such as a opal_keyval-change, we will continue to write RFCs. -- ------------------------------------------------------------------------ Rainer Keller, PhD Tel: +1 (865) 241-6293 Oak Ridge National Lab Fax: +1 (865) 241-4811 PO Box 2008 MS 6164 Email: kel...@ornl.gov Oak Ridge, TN 37831-2008 AIM/Skype: rusraink
ompi_onet-2009.02.27.pdf
Description: Adobe PDF document
move-btl-patches.tar
Description: Unix tar archive
move-btl-into-onet
Description: application/shellscript
1 9.144905 0.00000083
2 18.286412 0.00000083
3 27.254883 0.00000084
4 36.323561 0.00000084
6 54.519751 0.00000084
8 72.093067 0.00000085
12 107.840704 0.00000085
13 116.514993 0.00000085
16 142.217778 0.00000086
19 169.245287 0.00000086
21 186.789734 0.00000086
24 211.904469 0.00000086
27 220.993476 0.00000093
29 237.469187 0.00000093
32 260.154825 0.00000094
35 289.871208 0.00000092
45 366.996818 0.00000094
48 346.092758 0.00000106
51 363.059679 0.00000107
61 429.481902 0.00000108
64 452.480187 0.00000108
67 547.652267 0.00000093
93 646.006554 0.00000110
96 682.188374 0.00000107
99 806.632558 0.00000094
125 867.096486 0.00000110
128 902.332127 0.00000108
131 939.511465 0.00000106
189 1284.425449 0.00000112
192 1323.696534 0.00000111
195 1372.413004 0.00000108
253 1534.417789 0.00000126
256 1571.798374 0.00000124
259 1775.898337 0.00000111
381 2177.659049 0.00000133
384 2221.247747 0.00000132
387 2268.441197 0.00000130
509 2567.867235 0.00000151
512 2609.416613 0.00000150
515 2664.212041 0.00000147
765 3324.675259 0.00000176
768 3375.506720 0.00000174
771 3435.315216 0.00000171
1021 3729.527742 0.00000209
1024 3759.200618 0.00000208
1027 3998.257332 0.00000196
1533 4535.030453 0.00000258
1536 4629.047535 0.00000253
1539 4672.508641 0.00000251
2045 5016.953859 0.00000311
2048 5034.699304 0.00000310
2051 5089.816977 0.00000307
3069 5553.297287 0.00000422
3072 5559.106714 0.00000422
3075 5617.297672 0.00000418
4093 5787.492397 0.00000540
4096 5784.737016 0.00000540
4099 5760.403688 0.00000543
6141 6817.539197 0.00000687
6144 6814.555317 0.00000688
6147 6710.970954 0.00000699
8189 6961.980152 0.00000897
8192 6968.138567 0.00000897
8195 6904.197569 0.00000906
12285 7189.509304 0.00001304
12288 7184.998295 0.00001305
12291 7127.284686 0.00001316
16381 7278.311036 0.00001717
16384 7273.223779 0.00001719
16387 7260.405877 0.00001722
24573 7365.966991 0.00002545
24576 7357.734764 0.00002548
24579 7336.457697 0.00002556
32765 7411.620458 0.00003373
32768 7427.520879 0.00003366
32771 7419.442521 0.00003370
49149 8693.406725 0.00004313
49152 8700.247681 0.00004310
49155 8686.847753 0.00004317
65533 8424.519273 0.00005935
65536 8402.778957 0.00005950
65539 8419.341722 0.00005939
98301 8833.212613 0.00008490
98304 8828.101244 0.00008496
98307 8824.001317 0.00008500
131069 9226.911591 0.00010838
131072 9115.172251 0.00010971
131075 9112.970731 0.00010974
196605 9605.629351 0.00015616
196608 9601.731992 0.00015622
196611 9606.421310 0.00015615
262141 10061.764549 0.00019877
262144 10053.960425 0.00019893
262147 10051.171824 0.00019898
393213 10523.080257 0.00028509
393216 10419.464480 0.00028792
393219 10416.583996 0.00028800
524285 10429.133288 0.00038354
524288 10431.820467 0.00038344
524291 10432.204488 0.00038343
786429 10238.342610 0.00058603
786432 10240.453560 0.00058591
786435 10277.892485 0.00058378
1048573 10105.899971 0.00079161
1048576 10070.429704 0.00079441
1048579 10073.336623 0.00079418
1572861 9255.032191 0.00129659
1572864 9242.679956 0.00129832
1572867 9246.126712 0.00129784
2097149 8906.485888 0.00179644
2097152 8920.909589 0.00179354
2097155 8920.922351 0.00179354
3145725 8888.699059 0.00270006
3145728 8888.820183 0.00270002
3145731 8888.697844 0.00270006
4194301 8897.325099 0.00359658
4194304 8900.674956 0.00359523
4194307 8901.930104 0.00359473
6291453 8906.705943 0.00538920
6291456 8908.831152 0.00538791
6291459 8911.595807 0.00538624
8388605 8924.814354 0.00717101
8388608 8925.460503 0.00717050
8388611 8931.601181 0.00716557
1 9.100256 0.00000084
2 18.265129 0.00000084
3 27.340050 0.00000084
4 36.737509 0.00000083
6 54.545492 0.00000084
8 72.850961 0.00000084
12 108.671673 0.00000084
13 117.937351 0.00000084
16 144.037605 0.00000085
19 170.932517 0.00000085
21 187.762611 0.00000085
24 214.347530 0.00000085
27 225.181371 0.00000091
29 240.889442 0.00000092
32 263.654173 0.00000093
35 293.736834 0.00000091
45 369.675925 0.00000093
48 347.843346 0.00000105
51 366.038001 0.00000106
61 431.610748 0.00000108
64 457.023867 0.00000107
67 548.005106 0.00000093
93 650.423252 0.00000109
96 684.881916 0.00000107
99 808.216867 0.00000093
125 870.455194 0.00000110
128 904.505778 0.00000108
131 942.841428 0.00000106
189 1281.286415 0.00000113
192 1322.205692 0.00000111
195 1369.149837 0.00000109
253 1537.864543 0.00000126
256 1571.257815 0.00000124
259 1779.530641 0.00000111
381 2178.657608 0.00000133
384 2220.960134 0.00000132
387 2264.768136 0.00000130
509 2575.818416 0.00000151
512 2609.629297 0.00000150
515 2671.990949 0.00000147
765 3345.743255 0.00000174
768 3389.059186 0.00000173
771 3438.427966 0.00000171
1021 3741.535258 0.00000208
1024 3759.383786 0.00000208
1027 4010.389512 0.00000195
1533 4605.741444 0.00000254
1536 4633.529353 0.00000253
1539 4686.640169 0.00000251
2045 5017.722848 0.00000311
2048 5031.754206 0.00000311
2051 5096.080802 0.00000307
3069 5570.765962 0.00000420
3072 5580.249179 0.00000420
3075 5623.626117 0.00000417
4093 5751.389542 0.00000543
4096 5754.372865 0.00000543
4099 5738.985517 0.00000545
6141 6800.924663 0.00000689
6144 6792.232408 0.00000690
6147 6682.609002 0.00000702
8189 6984.735294 0.00000894
8192 6953.431651 0.00000899
8195 6914.051304 0.00000904
12285 7196.609545 0.00001302
12288 7189.557933 0.00001304
12291 7133.079920 0.00001315
16381 7248.974441 0.00001724
16384 7260.420327 0.00001722
16387 7257.702488 0.00001723
24573 7370.110946 0.00002544
24576 7367.610861 0.00002545
24579 7339.440869 0.00002555
32765 7410.635550 0.00003373
32768 7421.207086 0.00003369
32771 7404.832645 0.00003376
49149 8692.082157 0.00004314
49152 8691.239393 0.00004315
49155 8686.286578 0.00004317
65533 8420.203321 0.00005938
65536 8410.583396 0.00005945
65539 8416.192012 0.00005941
98301 8823.563884 0.00008500
98304 8826.763749 0.00008497
98307 8818.968703 0.00008505
131069 9039.091910 0.00011063
131072 9088.699870 0.00011003
131075 9092.660007 0.00010998
196605 9583.038864 0.00015652
196608 9583.226738 0.00015652
196611 9570.629986 0.00015673
262141 10020.931136 0.00019958
262144 10015.626492 0.00019969
262147 10016.841401 0.00019967
393213 10424.764625 0.00028777
393216 10409.323571 0.00028820
393219 10410.092681 0.00028818
524285 10494.911566 0.00038113
524288 10494.371794 0.00038116
524291 10494.206353 0.00038116
786429 10237.865318 0.00058606
786432 10235.496082 0.00058620
786435 10234.779945 0.00058624
1048573 10095.189266 0.00079245
1048576 10099.803234 0.00079209
1048579 10097.998796 0.00079224
1572861 9255.269651 0.00129656
1572864 9212.014592 0.00130265
1572867 9219.509716 0.00130159
2097149 8905.800356 0.00179658
2097152 8905.191566 0.00179670
2097155 8908.071945 0.00179613
3145725 8886.679168 0.00270067
3145728 8874.518714 0.00270437
3145731 8880.888588 0.00270243
4194301 8894.131462 0.00359788
4194304 8900.674956 0.00359523
4194307 8894.306646 0.00359781
6291453 8901.874076 0.00539212
6291456 8903.938923 0.00539087
6291459 8908.424772 0.00538816
8388605 8922.564733 0.00717282
8388608 8920.171224 0.00717475
8388611 8924.845465 0.00717100
