Colin, Thank you for clarifying (offlist) that EXTENDED COMMUNITY is encoded in bgpdump as UNKNOWN_ATTR(A, B, C) where A = attribute Flags octet B = attribute Type Code octet (B = 16 for EC) C = attribute length
So, yes, now we see and measure ECs in RIPE-RIS data (RIBs and Updates), and in Routeviews in the Updates data but not in RIBs (as you expected). Examples: ********************************************************************* RIPE-RIS: Community ANALYSIS (Collector : rrc03 From 2020-04-30 00:00 To 2020-04-30 00:55) ********************************************************************* # Updates = 1075583 (Total) # COMMUNITY = 859239 (79.89%) AS path length distribution = 1: 170 (0.02%) 2: 44803 (5.21%) 3: 141072 (16.42%) 4: 276271 (32.15%) 5: 238325 (27.74%) 6: 114158 (13.29%) 7: 31365 (3.65%) 8: 9018 (1.05%) 9: 2690 (0.31%) 10: 811 (0.09%) 11: 358 (0.04%) 12: 169 (0.02%) 13: 22 (0%) 14: 7 (0%) # LARGE_COMMUNITY = 152818 (14.21%) AS path length distribution = 2: 5655 (3.7%) 3: 17205 (11.26%) 4: 54372 (35.58%) 5: 45492 (29.77%) 6: 22065 (14.44%) 7: 6422 (4.2%) 8: 1068 (0.7%) 9: 397 (0.26%) 10: 71 (0.05%) 11: 35 (0.02%) 12: 26 (0.02%) 13: 6 (0%) 14: 4 (0%) # EXTENDED COMMUNITIES = 44606 (4.15%) AS path length distribution = 2: 2269 (5.09%) 3: 7435 (16.67%) 4: 17657 (39.58%) 5: 11600 (26.01%) 6: 3967 (8.89%) 7: 1221 (2.74%) 8: 371 (0.83%) 9: 57 (0.13%) 10: 19 (0.04%) 11: 8 (0.02%) 12: 1 (0%) 13: 1 (0%) ********************************************************************* Routeviews: Community ANALYSIS (Collector : saopaulo2 From 2020-04-30 00:00 To 2020-04-30 00:45) ********************************************************************* # Updates = 480962 (Total) # COMMUNITY = 332896 (69.21%) AS path length distribution = 1: 752 (0.23%) 2: 27870 (8.37%) 3: 93971 (28.23%) 4: 95553 (28.7%) 5: 64710 (19.44%) 6: 33943 (10.2%) 7: 11933 (3.58%) 8: 3157 (0.95%) 9: 686 (0.21%) 10: 165 (0.05%) 11: 82 (0.02%) 12: 45 (0.01%) 13: 21 (0.01%) 14: 8 (0%) # LARGE_COMMUNITY = 22910 (4.76%) AS path length distribution = 2: 3741 (16.33%) 3: 9986 (43.59%) 4: 5453 (23.8%) 5: 3190 (13.92%) 6: 411 (1.79%) 7: 122 (0.53%) 8: 6 (0.03%) 9: 1 (0%) # EXTENDED COMMUNITIES = 123035 (25.58%) AS path length distribution = 2: 8648 (7.03%) 3: 24466 (19.89%) 4: 30661 (24.92%) 5: 29805 (24.22%) 6: 18028 (14.65%) 7: 8345 (6.78%) 8: 2361 (1.92%) 9: 509 (0.41%) 10: 124 (0.1%) 11: 35 (0.03%) 12: 25 (0.02%) 13: 20 (0.02%) 14: 8 (0.01%) Thanks Lilia ________________________________ From: Colin Petrie <co...@spakka.net> Sent: Thursday, April 30, 2020 7:22 PM To: Hannachi, Lilia (IntlAssoc) <lilia.hanna...@nist.gov>; Sriram, Kotikalapudi (Fed) <kotikalapudi.sri...@nist.gov> Subject: Re: [GROW] Measurements on Regular, Extended, and Large Communities Sorry, I missed this part: > A = attribute Flags octet > B = attribute Type Code octet > C = attribute raw data blob 'C' is the attribute length. Apologies for the inaccurate response. The 'attribute raw data' is the bit after the '):' Compare: UNKNOWN_ATTR(192, 16, 8): 02 02 00 02 01 95 00 c9 UNKNOWN_ATTR(192, 16, 16): 02 02 00 02 01 95 00 c8 02 02 00 02 01 95 2a f9 'C' is how many octets to consume afterwards. The rest of the string is the attribute raw data blob. Thanks, Colin On 30-04-2020 19:00, Hannachi, Lilia (IntlAssoc) wrote: > Thanks Colin for the clarification > > Regards > > Lilia > ------------------------------------------------------------------------ > *From:* Colin Petrie <co...@spakka.net> > *Sent:* Thursday, April 30, 2020 12:54 PM > *To:* Hannachi, Lilia (IntlAssoc) <lilia.hanna...@nist.gov>; Sriram, > Kotikalapudi (Fed) <kotikalapudi.sri...@nist.gov> > *Subject:* Re: [GROW] Measurements on Regular, Extended, and Large > Communities > Hi Lilia, > >> In the shared results, we used updates generated on 2020-04-25 at 02:45.. >> That's why the number of updates is kind of low. After correcting the time >> and including the Extended communities we generated the following results: > > Ah, that makes more sense now. I see 365180 updates in RRC00 > 2020-04-25:02:45 > Thanks for clarifying. > >> We noticed that UNKNOWN_ATTR is associated with three different numbers >> "UNKNOWN_ATTR(A, B, C)". According to the generated results B is always 16 >> but A and C vary. >> I just would like to ask you if you have an idea about what does each number >> represent. > > Yes: > > A = attribute Flags octet > B = attribute Type Code octet > C = attribute raw data blob > > See: > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools..ietf.org%2Fhtml%2Frfc4271%23page-16&data=02%7C01%7Clilia.hannachi%40nist.gov%7Cebe14236d4d14c671abe08d7ed5d67d5%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C637238857741963449&sdata=sL9Ej53Gw1bqktXKNwqHWsE1oVv4ohFZYNIiMz6oApk%3D&reserved=0 > "Each path attribute is a triple <attribute type, attribute length, > attribute value> of variable length. Attribute Type is a two-octet field > that consists of the Attribute Flags octet, followed by the Attribute > Type Code octet". The document also describes the Flags available. > > Thanks, > Colin > > > > On 30/04/2020 18:32, Hannachi, Lilia (IntlAssoc) wrote: >> Colin, >> >> Thanks again for all the information. >> In the shared results, we used updates generated on 2020-04-25 at 02:45. >> That's why the number of updates is kind of low. After correcting the >> time and including the Extended communities we generated the following >> results: >> >> ********************************************************************* >> Community ANALYSIS (Collector : rrc00 2020-04-25:02:45) >> ********************************************************************* >> # Updates = 366516 (Total) >> # COMMUNITY = 293050 (79%) >> AS path length distribution = 2: 1480 (0%) 3: 7177 (2%) 4: >> 20212 (6%) 5: 23894 (8%) 6: 17841 (6%) 7: 8023 (2%) 8: 2317 >> (0%) 9: 539 (0%) 10: 197 (0%) 11: 150 (0%) 12: 102 (0%) >> 13: 32 (0%) 14: 12 (0%) >> # LARGE_COMMUNITY = 222778 (60%) >> AS path length distribution = 2: 534 (0%) 3: 5066 (2%) 4: 18288 >> (8%) 5: 75891 (34%) 6: 76389 (34%) 7: 33773 (15%) 8: 9108 >> (4%) 9: 2151 (0%) 10: 833 (0%) 11: 247 (0%) 12: 232 (0%) >> 13: 54 (0%) 14: 6 (0%) 16: 1 (0%) >> # EXTENDED COMMUNITIES = 15883 (4%) >> >> ********************************************************************* >> Community ANALYSIS (Collector : rrc15 2020-04-25:02:45) >> ********************************************************************* >> # Updates = 115386 (Total) >> # COMMUNITY = 92189 (79%) >> AS path length distribution = 1: 67 (0%) 2: 5996 (6%) 3: 23925 >> (25%) 4: 26179 (28%) 5: 16973 (18%) 6: 10503 (11%) 7: 4418 >> (4%) 8: 1153 (1%) 9: 305 (0%) 10: 79 (0%) 11: 55 (0%) 12: >> 38 (0%) 13: 19 (0%) 14: 6 (0%) >> # LARGE_COMMUNITY = 2555 (2%) >> AS path length distribution = 2: 89 (3%) 3: 1312 (51%) 4: 609 >> (23%) 5: 429 (16%) 6: 104 (4%) 7: 9 (0%) 8: 3 (0%) >> # EXTENDED COMMUNITIES = 11038 (9%) >> >> >> We noticed that UNKNOWN_ATTR is associated with three different numbers >> "UNKNOWN_ATTR(A, B, C)". According to the generated results B is always >> 16 but A and C vary. >> I just would like to ask you if you have an idea about what does each >> number represent. >> >> Regards >> >> Lilia >> >> ------------------------------------------------------------------------ >> *From:* Hannachi, Lilia (IntlAssoc) <lilia.hanna...@nist.gov> >> *Sent:* Thursday, April 30, 2020 9:45 AM >> *To:* Colin Petrie <co...@spakka.net>; Sriram, Kotikalapudi (Fed) >> <kotikalapudi.sri...@nist.gov> >> *Subject:* Re: [GROW] Measurements on Regular, Extended, and Large >> Communities >> Hi Colin, >> >> Thank you very much. You are right, currently we are using bgpdump. I >> noticed this UNKNOWN_ATTR in the results but I missed to link that with >> Extended Communities. >> >> We will work on that. >> >> Regards >> >> Lilia >> >> >> ------------------------------------------------------------------------ >> *From:* Colin Petrie <co...@spakka.net> >> *Sent:* Thursday, April 30, 2020 6:29 AM >> *To:* Sriram, Kotikalapudi (Fed) <kotikalapudi.sri...@nist.gov> >> *Cc:* Hannachi, Lilia (IntlAssoc) <lilia.hanna...@nist.gov> >> *Subject:* Re: [GROW] Measurements on Regular, Extended, and Large >> Communities >> Hi Sriram, >> >> (dropping the lists from this thread for the moment) >> >> Thank you for the clarification regarding your input sources. >> >> Please understand that the reason I am querying this, is because I know >> for a fact that there are lots of Extended Communities present in all >> of these sources! >> >> I used to work for RIPE RIS and we spent a lot of effort moving away >> from Quagga, partly to produce a new implementation that would preserve >> all attributes, to enable studying of new/unassigned/unknown attributes etc. >> >> Additionally, as I am the current maintainer of the 'bgpdump' MRT >> parsing software, I am aware of its attribute decode capabilities and >> have seen Extended Communities regularly while decoding MRT data (both >> in updates from RIS+route-view, and dumps from RIS). >> >> As you have clarified your input sources, I suggest your tooling does >> not handle Extended Communities correctly. What MRT parser are you using? >> >> bgpdump, for example, doesn't natively handle Extended Communities >> (although I have a feature request open to write a parser for ECs), but >> they are visible when enabling its 'dump unknown attributes' mode. >> >> bgpdump in 'long' output mode: >> ~/tmp/mrt/rrc00/2020.04$ bgpdump -q updates.20200401.0200.gz | grep >> UNKNOWN | grep 16, | head >> UNKNOWN_ATTR(192, 16, 16): 02 02 00 02 01 95 00 c8 02 02 00 02 01 95 >> 2a fa >> UNKNOWN_ATTR(192, 16, 16): 02 02 00 02 01 95 00 c8 02 02 00 02 01 95 >> 2a f9 >> UNKNOWN_ATTR(192, 16, 16): 02 02 00 02 01 95 00 c8 02 02 00 02 01 95 >> 2a f8 >> UNKNOWN_ATTR(192, 16, 16): 02 02 00 02 01 95 00 c8 02 02 00 02 01 95 >> 2a fa >> UNKNOWN_ATTR(192, 16, 8): 00 03 00 5a 00 00 00 5a >> UNKNOWN_ATTR(192, 16, 8): 00 03 00 62 00 00 00 62 >> UNKNOWN_ATTR(192, 16, 8): 02 02 00 02 01 95 00 c9 >> UNKNOWN_ATTR(192, 16, 16): 02 02 00 02 01 95 00 c8 02 02 00 02 01 95 >> 2a f9 >> UNKNOWN_ATTR(192, 16, 8): 00 03 00 5a 00 00 00 5a >> UNKNOWN_ATTR(192, 16, 8): 00 03 00 5a 00 00 00 5a >> (attribute 16 = Extended Communities) >> >> bgpdump in 'short' output mode, with the '-u' option: >> ~/tmp/mrt/rrc00/2020.04$ bgpdump -vmlpu updates.20200401.0200.gz | awk >> -F\| '{print $17}' | grep 10: | head >> 10:c0:02020002019500c80202000201952afa >> 10:c0:02020002019500c80202000201952af9 >> 10:c0:02020002019500c80202000201952af8 >> 10:c0:02020002019500c80202000201952afa >> 10:c0:0003005a0000005a >> 10:c0:0003006200000062 >> 10:c0:02020002019500c9 >> 10:c0:02020002019500c80202000201952af9 >> 10:c0:0003005a0000005a >> 10:c0:0003005a0000005a >> (attribute 10 hex = 16 dec = Extended Communities) >> >> So, to repeat a count on the same data window as your numbers: >> >> RIS rrc00, 2020-04-25.02*: >> ~/tmp/mrt/rrc00/2020.04$ for file in `ls updates*`; do echo -n ${file}:; >> bgpdump -q $file | grep "UNKNOWN_ATTR(" | grep ", 16," |wc -l ; done >> updates.20200425.0200.gz: 7716 >> updates.20200425.0205.gz: 6085 >> updates.20200425.0210.gz: 7719 >> updates.20200425.0215.gz: 7784 >> updates.20200425.0220.gz: 7989 >> updates.20200425.0225.gz: 7240 >> updates.20200425.0230.gz: 8154 >> updates.20200425.0235.gz: 16154 >> updates.20200425.0240.gz: 20336 >> updates.20200425.0245.gz: 15883 >> updates.20200425.0250.gz: 7366 >> updates.20200425.0255.gz: 8208 >> >> >> Anyway I hope I have demonstrated that there are many Extended >> Communities present in the MRT data from these collector projects, and >> hope that you can review your analysis method to see what is being >> missed here! >> >> Please let me know if I can help! >> >> Thanks, >> Colin >> >> P.S. Your update count seems unusually low. In the window of >> rrc00:2020-04-25.02.*, I count a total of 3,127,174 update messages. >> >> >> On 30/04/2020 01:04, Sriram, Kotikalapudi (Fed) wrote: >>> Thanks Nick and Colin. Insights you shared are helpful. >>> >>> Colin: >>> >>>> If you are analysing Route-Views table dumps (and not updates) then you >>>> won't see ECs because Quagga/FRR only dumps selected attributes into the >>>> MRT files [0]. ECs are not on the list of dumped attributes. LCs were >>>> added to the list as part of the LC implementation patch. >>> >>> Our analysis sees LCs in both Quagga and FRR RouteViews collectors in RIBs >>> and Updates. >>> We don't see ECs at all in any of these cases. >>> Some Quagga collectors show LCs in RIB (e.g., route-views3) and others >>> don't (e.g., route-views2). >>> >>>> RIPE's RIS implementation records all the received attributes opaquely >>>> so you will see ECs in there. They are also present within the >>>> route-views updates files (as these are dumped raw by Quagga). >>> >>> Our measurements show LCs in RIPE-RIS but no ECs. >>> Yep, we don't see ECs (in RIBs or Updates) in RIPE-RIS or Routeviews. >>> May be Nick is right about lack of compelling reason to use EC. >>> Examples (RIPE-RIS): >>> ********************************************************************* >>> Community ANALYSIS (Collector : rrc00 2020-04-25:02) >>> ********************************************************************* >>> # Updates = 366516 (Total) >>> # COMMUNITY = 293050 (79%) >>> AS path length distribution = 2: 1480 (0%) 3: 7177 (2%) 4: 20212 >>> (6%) 5: 23894 (8%) 6: 17841 (6%) 7: 8023 (2%) 8: 2317 (0%) >>> 9: 539 (0%) 10: 197 (0%) 11: 150 (0%) 12: 102 (0%) 13: 32 (0%) >>> 14: 12 (0%) >>> # LARGE_COMMUNITY = 222778 (60%) >>> AS path length distribution = 2: 534 (0%) 3: 5066 (2%) 4: 18288 >>> (8%) 5: 75891 (34%) 6: 76389 (34%) 7: 33773 (15%) 8: 9108 (4%) >>> 9: 2151 (0%) 10: 833 (0%) 11: 247 (0%) 12: 232 (0%) 13: 54 >>> (0%) 14: 6 (0%) 16: 1 (0%) >>> # EXTENDED COMMUNITIES = 0 (0%) >>> >>> ********************************************************************* >>> Community ANALYSIS (Collector : rrc15 2020-04-25:02) >>> ********************************************************************* >>> # Updates = 115386 (Total) >>> # COMMUNITY = 92189 (79%) >>> AS path length distribution = 1: 67 (0%) 2: 5996 (6%) 3: 23925 >>> (25%) 4: 26179 (28%) 5: 16973 (18%) 6: 10503 (11%) 7: 4418 (4%) >>> 8: 1153 (1%) 9: 305 (0%) 10: 79 (0%) 11: 55 (0%) 12: 38 (0%) >>> 13: 19 (0%) 14: 6 (0%) >>> # LARGE_COMMUNITY = 2555 (2%) >>> AS path length distribution = 2: 89 (3%) 3: 1312 (51%) 4: 609 >>> (23%) 5: 429 (16%) 6: 104 (4%) 7: 9 (0%) 8: 3 (0%) >>> # EXTENDED COMMUNITIES = 0 (0%) >>> >>> Sriram >>> >> >
_______________________________________________ GROW mailing list GROW@ietf.org https://www.ietf.org/mailman/listinfo/grow
