Hi,
Using latest Carbondata and Parquet - comparing full scan, aggregation
queries on 5 node SPARK cluster each consists of 8cores, 32GB ram - we are
having 18 million records for a minute which has 40 columns and 20 are of
string type. We have a use case where in we want to do a full scan and
aggregate the data based on 32 keys which mostly integer or long type. Even
after changing / playing around carbon properties such
DICTIONARY_EXCLUDE/INCLUDE, INVERTED_INDEX, block and blocklet size - Carbon
query takes 2 secs for 1 aggregation (2 group by keys for SUM, MIN, MAX,
COUNT) where as Parquet takes 0.9 secs.
Carbon table creation is as follows -
carbon.sql("create table Mar18(flowSeqNum integer ,protocolId integer
,srcTos integer ,dstTos integer ,tcpBits integer ,srcPort
integer ,dstPort integer ,workerId integer ,epoch long ,rowNum long
,exportMs long ,pktSeqNum long ,inIfId long ,inIfEntityId long
,outIfId long ,outIfEntityId long ,inOctets long ,outOctets long ,inPackets
long ,outPackets long ,bgpSrcAsNum long ,bgpDstAsNum long
,endMs long ,startMs long ,policyQosQueueId long ,exporterIp string ,srcIp
string ,dstIp string ,srcMask string ,dstMask string
,direction string ,nextHopIp string ,bgpNextHopIp string ,appName string
,srcIpGroup string ,dstIpGroup string ,appId string
,policyQosClassificationHierarchy string ,inIfEnabled boolean ,outIfEnabled
boolean) stored as carbondata ")
Can u help here where we are going wrong?
BR,
Deepak
--
Sent from:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
