baibaichen opened a new issue, #7726:
URL: https://github.com/apache/incubator-gluten/issues/7726
### Backend
VL (Velox)
### Bug description
Run test SQL in `GlutenClickHouseTPCHColumnarShuffleParquetAQESuite`
```sql
select LINEITEM.L_DISCOUNT,
PART.P_TYPE,
LINEITEM.L_COMMENT,
LINEITEM.L_SUPPKEY,
PART.P_PARTKEY,
PART.P_SIZE,
LINEITEM.L_RETURNFLAG,
LINEITEM.L_RECEIPTDATE,
PART.P_NAME,
SUPPLIER.S_COMMENT,
LINEITEM.L_ORDERKEY,
PART.P_MFGR,
SUPPLIER.S_ACCTBAL,
SUPPLIER.S_SUPPKEY,
LINEITEM.L_SHIPMODE,
SUPPLIER.S_NATIONKEY,
LINEITEM.L_SHIPDATE,
LINEITEM.L_COMMITDATE,
SUPPLIER.S_NAME,
PART.P_COMMENT,
LINEITEM.L_TAX,
LINEITEM.L_QUANTITY,
LINEITEM.L_PARTKEY,
PART.P_CONTAINER,
MIN(LINEITEM.L_EXTENDEDPRICE),
COUNT(LINEITEM.L_QUANTITY),
COUNT(DISTINCT LINEITEM.L_PARTKEY),
MIN(LINEITEM.L_TAX),
MIN(ORDERS.O_TOTALPRICE),
COUNT(LINEITEM.L_EXTENDEDPRICE),
COUNT(ORDERS.O_SHIPPRIORITY),
COUNT(1),
MAX(LINEITEM.L_DISCOUNT)
from LINEITEM
INNER JOIN SUPPLIER AS SUPPLIER ON LINEITEM.L_SUPPKEY =
SUPPLIER.S_SUPPKEY
INNER JOIN PART AS PART ON LINEITEM.L_PARTKEY = PART.P_PARTKEY
INNER JOIN ORDERS AS ORDERS ON LINEITEM.L_ORDERKEY =
ORDERS.O_ORDERKEY
where (not (((P_RETAILPRICE is not null or
((S_NATIONKEY is not null and P_MFGR like '%Manufacturer#1')
or P_BRAND not like 'Brand#11')) or
((S_SUPPKEY not in
(1206, 1496, 1191, 2445, 491, 1407, 1969, 261, 1418, 310,
2099, 1343, 327, 261, 707, 37, 753, 696, 1363,
628, 1158, 2239, 26, 1180, 2448, 1698, 2099, 1326, 1247,
1203, 161, 1698, 310, 692, 491, 1920, 28, 370,
370, 261, 2258, 1146, 983, 683, 24, 1611, 5161, 3141, 2258,
1287, 683, 1720, 1887, 310, 707, 1836, 1287,
2065, 1859, 1203, 1611, 1835, 2099, 701, 2314, 692, 1418,
2367, 425, 1720, 8285, 1969, 1804, 310, 2258,
1418, 463, 2048, 368, 1253, 549, 2258, 327, 1973, 817) and
1300 > S_SUPPKEY) or (S_PHONE not in
('10-246-381-9259',
'10-211-466-9198',
'10-509-209-3829',
'10-741-929-4244',
'10-393-500-3856',
'10-495-104-1252',
'10-983-665-2259',
'10-295-590-8708',
'10-983-665-2259',
'10-745-572-7198',
'10-384-209-1825',
'10-734-420-5738',
'10-845-970-4551',
'10-630-928-4130',
'10-325-193-7475',
'%10-475-868-5521',
'10-903-990-3612',
'10-352-443-2162%',
'10-842-403-7954',
'10-789-325-3069',
'10-996-906-4890',
'10-404-519-2270',
'10-848-716-8078',
'10-246-381-9259',
'10-262-377-2302',
'10-361-729-1693',
'10-745-572-7198',
'10-384-209-1825',
'10-262-132-6639',
'10-361-729-1693',
'10-746-144-5600',
'10-409-763-8909',
'10-123-465-1292',
'10-745-572-7198%',
'10-599-740-9848',
'10-453-843-1585',
'10-191-563-6127',
'10-848-716-8078',
'10-763-945-1271',
'10-393-500-3856') and
(not (P_NAME not like 'light dark lemon lace medium%' and
P_NAME is null))))) or
((((S_ADDRESS is null or P_CONTAINER in
('LG JAR', 'JUMBO CASE', 'JUMBO CASE',
'MED BOX', 'WRAP BAG', 'SM CASE',
'WRAP JAR', 'JUMBO PKG', 'SM CAN', 'SM
BOX', 'JUMBO CASE', 'MED BOX', 'LG JAR',
'JUMBO CASE', 'MED DRUM', 'JUMBO PKG',
'SM CAN', 'WRAP JAR', 'LG CASE', 'LG BAG',
'SM PACK', 'JUMBO DRUM', 'WRAP BOX',
'JUMBO CAN', 'LG PKG', 'WRAP CAN',
'MED PACK', 'SM BOX', 'SM DRUM', 'SM
PACK', 'MED DRUM', 'MED PACK', 'MED BOX',
'MED CAN%', 'SM JAR', 'SM CAN', 'JUMBO
BOX', 'JUMBO BAG', 'LG BAG', 'LG PKG',
'LG PACK', 'LG BAG', 'JUMBO BOX', 'SM
BOX', 'JUMBO CAN', 'JUMBO PKG', 'LG BAG',
'MED BOX', 'JUMBO CASE', 'MED BOX',
'LG BAG', 'LG PACK', 'MED BOX', 'LG PKG',
'SM BOX', 'WRAP BOX', 'LG CASE', 'MED
PACK', 'LG PKG', '%LG CASE', 'LG JAR',
'LG BAG', 'LG BOX', 'SM CAN', 'WRAP
CAN', 'WRAP PACK', 'JUMBO CASE', 'SM BOX',
'SM PACK', 'WRAP PKG', 'MED CAN', 'SM
BOX', 'LG CASE', 'JUMBO CAN', 'LG JAR',
'SM DRUM', 'MED PKG', 'JUMBO BAG', 'SM
CASE', 'MED BAG', 'SM PACK',
'SM PACK')) and S_SUPPKEY is not null)
and (P_PARTKEY not in
(1358682, 1592117, 1114403, 839396,
1114617, 959268, 1114713, 1358631,
806397, 959018, 1114926, 812800,
1568237, 959088, 839340, 959419,
1115053, 1358740, 1114282) and
(S_SUPPKEY between 463 and 1887 or
S_SUPPKEY not in
(1287, 1422, 1878, 1191, 1804, 476,
1097, 1326, 1597, 1158, 261, 1689,
1493, 2314, 817, 1097, 2239, 327,
1887, 118, 1547, 476, 2131, 1247,
1496, 1698, 1717, 454, 1692, 1920,
1973, 2010, 1804, 774, 1611, 425,
28, 1611, 183, 983, 800, 5915, 1311,
24, 2298, 118, 183, 784, 1592, 1549,
983, 1283, 1418, 291, 118, 1407,
2072, 291, 1180, 1404, 1097, 1724,
1611, 692, 491, 316, 161, 2314,
1404, 696, 2072, 2072, 491, 1692,
764, 742, 118, 425)))) and
(P_CONTAINER in
('SM PKG', 'LG PKG', 'LG CASE', 'MED PKG', 'WRAP JAR', 'LG
BAG', 'SM BOX', 'JUMBO BOX', 'SM PKG',
'SM PKG', 'JUMBO BOX', 'MED BOX', 'JUMBO PKG', 'WRAP CAN',
'MED DRUM', 'MED JAR', 'SM BAG', 'MED CAN',
'SM PACK', 'SM CASE', 'MED BAG', 'JUMBO PKG', 'LG CASE', 'SM
PKG', 'MED BOX', 'LG CASE', 'JUMBO DRUM',
'MED BAG', 'JUMBO CASE', 'SM BOX', 'JUMBO PACK', 'WRAP BOX',
'%JUMBO BOX', 'JUMBO BOX', 'JUMBO CASE',
'SM CAN', 'JUMBO BOX', 'SM CAN', 'LG CASE') and P_BRAND is
null))))
or (((P_BRAND is not null or (P_SIZE not in
(25, 11, 48, 15, 48, 16, 3, 45, 37, 42, 47,
42, 42, 16, 97, 16, 48, 12, 87, 13, 27, 22,
42, 37, 50, 9, 34) and S_NATIONKEY >= 0))
or 955.65 = S_ACCTBAL) or
(P_TYPE not like 'MEDIUM POLISHED STEEL%' or (not S_ACCTBAL is null)))
group by LINEITEM.L_DISCOUNT, PART.P_TYPE, LINEITEM.L_COMMENT,
LINEITEM.L_SUPPKEY, PART.P_PARTKEY, PART.P_SIZE,
LINEITEM.L_RETURNFLAG, LINEITEM.L_RECEIPTDATE, PART.P_NAME,
SUPPLIER.S_COMMENT, LINEITEM.L_ORDERKEY,
PART.P_MFGR, SUPPLIER.S_ACCTBAL, SUPPLIER.S_SUPPKEY,
LINEITEM.L_SHIPMODE, SUPPLIER.S_NATIONKEY,
LINEITEM.L_SHIPDATE, LINEITEM.L_COMMITDATE, SUPPLIER.S_NAME,
PART.P_COMMENT, LINEITEM.L_TAX,
LINEITEM.L_QUANTITY, LINEITEM.L_PARTKEY, PART.P_CONTAINER
```
Get the following error, it is ok in the previous codes!
```
org.apache.gluten.exception.GlutenException:
org.apache.gluten.exception.GlutenException: Memory limit exceeded: would use
512.29 MiB (attempt to allocate chunk of 4224948 bytes), current RSS 1.21 GiB,
maximum: 512.00 MiB.: While executing GraceAggregatingTransform
0.
..../SourceCode/rebase_ch/contrib/llvm-project/libcxx/include/exception:141:
Poco::Exception::Exception(String const&, int) @ 0x0000000014561359
1. ..../SourceCode/rebase_ch/src/Common/Exception.cpp:109:
DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @
0x000000000c2aa3d9
2. ..../SourceCode/rebase_ch/src/Common/Exception.h:110:
DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000006790a2c
3. ..../SourceCode/rebase_ch/src/Common/Exception.h:128:
DB::Exception::Exception<char const*, char const*, String, long&, String,
String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int,
FormatStringHelperImpl<std::type_identity<char const*>::type,
std::type_identity<char const*>::type, std::type_identity<String>::type,
std::type_identity<long&>::type, std::type_identity<String>::type,
std::type_identity<String>::type, std::type_identity<char const*>::type,
std::type_identity<std::basic_string_view<char,
std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, long&,
String&&, String&&, char const*&&, std::basic_string_view<char,
std::char_traits<char>>&&) @ 0x000000000c2d10c9
4. ..../SourceCode/rebase_ch/src/Common/MemoryTracker.cpp:316:
MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @
0x000000000c2d00d2
5. ..../SourceCode/rebase_ch/src/Common/MemoryTracker.cpp:373:
MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @
0x000000000c2cfc96
6. ..../SourceCode/rebase_ch/src/Common/CurrentMemoryTracker.cpp:64:
CurrentMemoryTracker::allocImpl(long, bool) @ 0x000000000c27899e
7. ..../SourceCode/rebase_ch/src/Common/Allocator.cpp:233: Allocator<false,
false>::realloc(void*, unsigned long, unsigned long, unsigned long) @
0x000000000c2772db
8. void DB::PODArrayBase<8ul, 4096ul, Allocator<false, false>, 63ul,
64ul>::realloc<>(unsigned long) @ 0x0000000006884976
9. DB::ColumnString::insertFrom(DB::IColumn const&, unsigned long) @
0x000000000f01fecd
10. ..../SourceCode/rebase_ch/src/Columns/ColumnNullable.cpp:273:
DB::ColumnNullable::insertFrom(DB::IColumn const&, unsigned long) @
0x0000000010c1bd1f
11. ..../SourceCode/rebase_ch/src/Columns/IColumn.cpp:149:
DB::IColumnHelper<DB::ColumnNullable, DB::IColumn>::scatter(unsigned long,
DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>
const&) const @ 0x0000000010e80b79
12. ..../SourceCode/rebase_ch/src/Interpreters/JoinUtils.cpp:569:
DB::JoinCommon::scatterBlockByHash(std::vector<String, std::allocator<String>>
const&, DB::Block const&, unsigned long) @ 0x0000000010369286
13.
..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:268:
local_engine::GraceAggregatingTransform::scatterBlock(DB::Block const&) @
0x000000000c805f63
14.
..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:495:
local_engine::GraceAggregatingTransform::mergeOneBlock(DB::Block const&, bool)
@ 0x000000000c80285f
15.
..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:383:
local_engine::GraceAggregatingTransform::prepareBucketOutputBlocks(unsigned
long) @ 0x000000000c803453
16.
..../SourceCode/rebase_ch/utils/extern-local-engine/Operator/GraceAggregatingTransform.cpp:158:
local_engine::GraceAggregatingTransform::work() @ 0x000000000c8021c8
17.
..../SourceCode/rebase_ch/src/Processors/Executors/ExecutionThreadContext.cpp:47:
DB::ExecutionThreadContext::executeTask() @ 0x00000000118b16a2
18.
..../SourceCode/rebase_ch/src/Processors/Executors/PipelineExecutor.cpp:289:
DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @
0x00000000118a695f
19.
..../SourceCode/rebase_ch/src/Processors/Executors/PipelineExecutor.cpp:163:
DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x00000000118a63c9
20.
..../SourceCode/rebase_ch/src/Processors/Executors/PullingPipelineExecutor.cpp:54:
DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x00000000118b8134
21.
..../SourceCode/rebase_ch/src/Processors/Executors/PullingPipelineExecutor.cpp:65:
DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x00000000118b8299
22.
..../SourceCode/rebase_ch/utils/extern-local-engine/Parser/LocalExecutor.cpp:68:
local_engine::LocalExecutor::hasNext() @ 0x000000000c672db1
23.
..../SourceCode/rebase_ch/utils/extern-local-engine/local_engine_jni.cpp:308:
Java_org_apache_gluten_vectorized_BatchIterator_nativeHasNext @
0x0000000006776237
at
org.apache.gluten.iterator.ClosableIterator.hasNext(ClosableIterator.java:41)
at
org.apache.gluten.backendsapi.clickhouse.CollectMetricIterator.hasNext(CHIteratorApi.scala:349)
at
org.apache.gluten.vectorized.CloseableCHColumnBatchIterator.$anonfun$hasNext$1(CloseableCHColumnBatchIterator.scala:42)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at
org.apache.gluten.metrics.GlutenTimeMetric$.withNanoTime(GlutenTimeMetric.scala:41)
at
org.apache.gluten.vectorized.CloseableCHColumnBatchIterator.hasNext(CloseableCHColumnBatchIterator.scala:42)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1931)
at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1274)
at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1274)
at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2268)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
