gianm commented on PR #20467: URL: https://github.com/apache/druid/pull/20467#issuecomment-5927908150
Some analysis of the impact, done by loading a few hundred segments and then measuring the size of objects in the heap. "Shared" column names means that all segments have the same names, "Unique" means that all segments have unique column names. You'd expect Shared to highly benefit from interning, and Unique to generally not be able to intern as much. "V10" is a V10 segment fully loaded. "V10 partial" is a V10 segment with partial loading enabled that hasn't had any parts loaded yet. Shared column names: | Schema | Segment type | Column names | Before | After | |---|---|---|---|---| | 10 columns | V9 | Shared | 8,555 | 6,578 (-23%) | | 10 columns | V10 | Shared | 11,861 | 7,831 (-34%) | | 10 columns | V10 partial | Shared | 9,218 | 4,548 (-51%) | | 50 non-null | V9 | Shared | 35,686 | 26,018 (-27%) | | 50 non-null | V10 | Shared | 50,087 | 29,062 (-42%) | | 50 non-null | V10 partial | Shared | 33,260 | 9,550 (-71%) | | 50 non-null + 50 null-only | V9 | Shared | 50,620 | 27,544 (-46%) | | 50 non-null + 50 null-only | V10 | Shared | 77,164 | 32,871 (-57%) | | 50 non-null + 50 null-only | V10 partial | Shared | 62,444 | 15,475 (-75%) | Unique column names: | Schema | Segment type | Column names | Before | After | |---|---|---|---|---| | 10 columns | V9 | Unique | 10,246 | 8,475 (-17%) | | 10 columns | V10 | Unique | 16,724 | 12,897 (-23%) | | 10 columns | V10 partial | Unique | 14,095 | 9,614 (-32%) | | 50 non-null | V9 | Unique | 45,282 | 36,131 (-20%) | | 50 non-null | V10 | Unique | 68,510 | 47,996 (-30%) | | 50 non-null | V10 partial | Unique | 51,812 | 28,489 (-45%) | | 50 non-null + 50 null-only | V9 | Unique | 69,196 | 47,034 (-32%) | | 50 non-null + 50 null-only | V10 | Unique | 108,391 | 65,020 (-40%) | | 50 non-null + 50 null-only | V10 partial | Unique | 94,058 | 47,611 (-49%) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
