[ https://issues.apache.org/jira/browse/DRILL-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boaz Ben-Zvi updated DRILL-5728: -------------------------------- Description: When aggregating a non-nullable column (like *sum(l_partkey)* below), the code generation creates an extra value vector (in addition to the actual "sum" vector) which is used as a "nonNullCount". This is useless (as the underlying column is non-nullable), and wastes considerable memory ( 8 * 64K = 512K per each value in a batch !!) Example query: {{select sum(l_partkey) as slpk from cp.`tpch/lineitem.parquet` group by l_orderkry;}} And as can be seen in the generated code below, the bigint value vector *vv5* is only used to hold a *1* flag to note "not null": bq. public void updateAggrValuesInternal(int incomingRowIdx, int htRowIdx) bq. throws SchemaChangeException bq. { bq. { bq. IntHolder out11 = new IntHolder(); bq. { bq. out11 .value = vv8 .getAccessor().get((incomingRowIdx)); bq. } bq. IntHolder in = out11; bq. work0 .value = vv1 .getAccessor().get((htRowIdx)); bq. BigIntHolder value = work0; bq. work4 .value = vv5 .getAccessor().get((htRowIdx)); bq. BigIntHolder nonNullCount = work4; bq. bq. SumFunctions$IntSum_add: { bq. nonNullCount.value = 1; bq. value.value += in.value; bq. } bq. bq. work0 = value; bq. vv1 .getMutator().set((htRowIdx), work0 .value); bq. work4 = nonNullCount; bq. vv5 .getMutator().set((htRowIdx), work4 .value); bq. } bq. } was: When aggregating a non-nullable column (like *sum(l_partkey)* below), the code generation creates an extra value vector (in addition to the actual "sum" vector) which is used as a "nonNullCount". This is useless (as the underlying column is non-nullable), and wastes considerable memory ( 8 * 64K = 512K per each value in a batch !!) Example query: {{select sum(l_partkey) as slpk from cp.`tpch/lineitem.parquet` group by l_orderkry;}} And as can be seen in the generated code below, the bigint value vector *vv5* is only used to hold a *1* flag to note "not null": {quote}public void updateAggrValuesInternal(int incomingRowIdx, int htRowIdx) throws SchemaChangeException { { IntHolder out11 = new IntHolder(); { out11 .value = vv8 .getAccessor().get((incomingRowIdx)); } IntHolder in = out11; work0 .value = vv1 .getAccessor().get((htRowIdx)); BigIntHolder value = work0; work4 .value = vv5 .getAccessor().get((htRowIdx)); BigIntHolder nonNullCount = work4; SumFunctions$IntSum_add: { nonNullCount.value = 1; value.value += in.value; } work0 = value; vv1 .getMutator().set((htRowIdx), work0 .value); work4 = nonNullCount; vv5 .getMutator().set((htRowIdx), work4 .value); } }{quote} > Hash Aggregate: Useless bigint value vector in the values batch > --------------------------------------------------------------- > > Key: DRILL-5728 > URL: https://issues.apache.org/jira/browse/DRILL-5728 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Codegen > Affects Versions: 1.11.0 > Reporter: Boaz Ben-Zvi > Priority: Minor > > When aggregating a non-nullable column (like *sum(l_partkey)* below), the > code generation creates an extra value vector (in addition to the actual > "sum" vector) which is used as a "nonNullCount". > This is useless (as the underlying column is non-nullable), and wastes > considerable memory ( 8 * 64K = 512K per each value in a batch !!) > Example query: > {{select sum(l_partkey) as slpk from cp.`tpch/lineitem.parquet` group by > l_orderkry;}} > And as can be seen in the generated code below, the bigint value vector *vv5* > is only used to hold a *1* flag to note "not null": > bq. public void updateAggrValuesInternal(int incomingRowIdx, int htRowIdx) > bq. throws SchemaChangeException > bq. { > bq. { > bq. IntHolder out11 = new IntHolder(); > bq. { > bq. out11 .value = vv8 > .getAccessor().get((incomingRowIdx)); > bq. } > bq. IntHolder in = out11; > bq. work0 .value = vv1 .getAccessor().get((htRowIdx)); > bq. BigIntHolder value = work0; > bq. work4 .value = vv5 .getAccessor().get((htRowIdx)); > bq. BigIntHolder nonNullCount = work4; > bq. > bq. SumFunctions$IntSum_add: { > bq. nonNullCount.value = 1; > bq. value.value += in.value; > bq. } > bq. > bq. work0 = value; > bq. vv1 .getMutator().set((htRowIdx), work0 .value); > bq. work4 = nonNullCount; > bq. vv5 .getMutator().set((htRowIdx), work4 .value); > bq. } > bq. } > -- This message was sent by Atlassian JIRA (v6.4.14#64029)