alamb commented on PR #195: URL: https://github.com/apache/parquet-site/pull/195#issuecomment-5301817031
> I haven't really dug into the numbers yet or tried yet to reproduce the results (aka is gov26 really so much better than ALP, or is there something that ) I did a little testing of this this morning, and the reason is that the govt data is 99% zeros. This actually might be a case where encoding the entire thing as exceptions might be better than trying to encode the values. ```sql andrewlamb@Andrews-MacBook-Pro-3:~/Downloads/alp-parquet-output$ ~/Downloads/datafusion-cli-alamb_test_alp_encoding -c "select count(*) as cnt from 'gov26.alp.parquet'" DataFusion CLI v54.1.0 +-----------+ | cnt | +-----------+ | 141123827 | +-----------+ 1 row(s) fetched. Elapsed 0.003 seconds. ``` ```sql andrewlamb@Andrews-MacBook-Pro-3:~/Downloads/alp-parquet-output$ ~/Downloads/datafusion-cli-alamb_test_alp_encoding -c "select value, count(*) as cnt from 'gov26.alp.parquet' group by value order by cnt desc" DataFusion CLI v54.1.0 +----------+-----------+ | value | cnt | +----------+-----------+ | 0.0 | 140499627 | | 1.0 | 1963 | | 22234.0 | 1560 | | -3111.0 | 988 | | -2130.0 | 910 | | 667.0 | 728 | | 30000.0 | 663 | | 21677.0 | 572 | | 1049.0 | 559 | | 278.0 | 533 | | 50000.0 | 494 | | -1.0 | 481 | | 100000.0 | 468 | | -2660.0 | 455 | | 1050.0 | 442 | | -346.0 | 429 | | 1008.0 | 429 | | 21955.0 | 416 | | 1009.0 | 403 | | -2916.0 | 403 | | 10000.0 | 394 | | -413.0 | 390 | | 158.0 | 338 | | 3000.0 | 338 | | -1151.96 | 325 | | 20000.0 | 325 | | -4274.0 | 325 | | -3112.0 | 312 | | 575.0 | 312 | | 427.34 | 312 | | -5832.0 | 312 | | -2659.0 | 299 | | 1887.0 | 299 | | -248.0 | 299 | | 150000.0 | 286 | | -168.53 | 286 | | -4802.0 | 286 | | 60000.0 | 260 | | 15000.0 | 260 | | -4533.0 | 260 | | . | | . | | . | +----------+-----------+ 37921 row(s) fetched. (First 40 displayed. Use --maxrows to adjust) Elapsed 0.073 seconds. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
