Hi
For -- 4) Lazy decoding of the dictionary, just i tested 180 millions rows
data with the script:
"select province,sum(age),count(*) from presto_carbondata group by province
order by province"
Spark integration module has "dictionary lazy decode", presto doesn't have
"dictionary lazy decode", the performance is 4.5 times difference, so
"dictionary lazy decode" might much help to improve aggregation performance.
The detail test result as below :
*1. Presto+CarbonData is 9 second:*
presto:default> select province,sum(age),count(*) from presto_carbondata
group by province order by province;
province | _col1 | _col2
----------+----------+---------
AB | 57442740 | 1385010
BC | 57488826 | 1385580
MB | 57564702 | 1386510
NB | 57599520 | 1386960
NL | 57446592 | 1383774
NS | 57448734 | 1384272
NT | 57534228 | 1386936
NU | 57506844 | 1385346
ON | 57484956 | 1384470
PE | 57325164 | 1379802
QC | 57467886 | 1385076
SK | 57385152 | 1382364
YT | 57377556 | 1383900
(13 rows)
Query 20170720_022833_00004_c9ky2, FINISHED, 1 node
Splits: 55 total, 55 done (100.00%)
0:09 [18M rows, 34.3MB] [1.92M rows/s, 3.65MB/s]
*2.Spark+CarbonData is :2 seconds*
scala> benchmark { carbon.sql("select province,sum(age),count(*) from
presto_carbondata group by province order by province").show }
+--------+--------+--------+
|province|sum(age)|count(1)|
+--------+--------+--------+
| AB|57442740| 1385010|
| BC|57488826| 1385580|
| MB|57564702| 1386510|
| NB|57599520| 1386960|
| NL|57446592| 1383774|
| NS|57448734| 1384272|
| NT|57534228| 1386936|
| NU|57506844| 1385346|
| ON|57484956| 1384470|
| PE|57325164| 1379802|
| QC|57467886| 1385076|
| SK|57385152| 1382364|
| YT|57377556| 1383900|
+--------+--------+--------+
2109.346231ms
--
View this message in context:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Presto-CarbonData-optimization-work-discussion-tp18509p18522.html
Sent from the Apache CarbonData Dev Mailing List archive mailing list archive
at Nabble.com.
