[jira] [Updated] (IMPALA-9957) Impalad crashes when serializing large rows in aggregation spilling
[
https://issues.apache.org/jira/browse/IMPALA-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-9957:
---
Component/s: Backend
> Impalad crashes when serializing large rows in aggregation spilling
> ---
>
> Key: IMPALA-9957
> URL: https://issues.apache.org/jira/browse/IMPALA-9957
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0, Impala 3.3.0, Impala 3.4.0
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
> Fix For: Impala 4.0
>
>
> Queries to reproduce the crash using the testdata:
> {code:sql}
> create table bigstrs stored as parquet as
> select *, repeat(uuid(), cast(random() * 10 as int)) as bigstr
> from functional.alltypes;
> set MAX_ROW_SIZE=3.5MB;
> set MEM_LIMIT=4GB;
> create table my_str_group stored as parquet as
> select group_concat(string_col) as ss, bigstr
> from bigstrs group by bigstr;
> {code}
> The last query 1) has large rows, 2) needs spilling in aggregation 3) has
> aggregation on functions needs serialize (e.g. group_concat, appx_median,
> min(string), etc). With these 3 conditions, it will trigger this bug.
> The crash stacktraces are different in different build modes. Crash
> stacktrace in RELEASE build with codegen enabled:
> {code:java}
> Thread 316 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 0x7f5fba83db3c
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 437 + 0x2]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 libstdc++.so.6.0.24 + 0x120b28
> 7
> impalad!apache::hive::service::cli::thrift::TColumnValue::printTo(std::ostream&)
> const [converter_lexical_streams.hpp : 161 + 0x8]
> 8 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 9 impalad!tc_newarray + 0x171
> {code}
> Crash stacktrace in RELEASE build with codegen disabled (set
> DISABLE_CODEGEN=true):
> {code:java}
> Thread 320 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 impalad!impala::Status
> impala::GroupingAggregator::AddBatchImpl(impala::RowBatch*,
> impala::TPrefetchMode::type, impala::HashTableCtx*)
> [grouping-aggregator-ir.cc : 148 + 0x11]
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 439 + 0x5]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc
> : 97 + 0x12]
> 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 815 + 0x19]
> 9 impalad!impala::Thread::SuperviseThread(std::__cxx11::basic_string std::char_traits, std::allocator > const&,
> std::__cxx11::basic_string, std::allocator
> > const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) [function_template.hpp : 770
> + 0x7]
> 10 impalad!boost::detail::thread_data (*)(std::__cxx11::basic_string,
> std::allocator > const&, std::__cxx11::basic_string std::char_traits, std::allocator > const&, boost::function ()>, impala::ThreadDebugInfo const*, impala::Promise (impala::PromiseMode)0>*),
> boost::_bi::list5 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >
> >::run() [bind.hpp : 531 + 0xc]
> 11 impalad!thread_proxy + 0x72
> 12 libpthread-2.23.so + 0x76ba
> 13 libc-2.23.so + 0x1074dd
> {code}
> Crash stacktrace in DEBUG build with codegen disabled is a bit ealier -
> crashed at a DCHECK:
> {code:java}
> F0715 20:29:24.389505 16868 grouping-aggregator-partition.cc:125]
> 1d4b40df02e6ad76:433ed5740003] Check failed: !status.ok() Stream was
> unpinned - AddRow() only fails on error
> *** Check failure stack trace: ***
> @ 0x513f31c google::LogMessage::Fail()
> @ 0x5140c0c google::LogMessage::SendToLog()
> @ 0x513ec7a google::LogMessage::Flush()
> @ 0x5142878 google::LogMessageFata
[jira] [Updated] (IMPALA-9957) Impalad crashes when serializing large rows in aggregation spilling
[
https://issues.apache.org/jira/browse/IMPALA-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-9957:
---
Affects Version/s: Impala 3.2.0
Impala 3.3.0
Impala 3.4.0
> Impalad crashes when serializing large rows in aggregation spilling
> ---
>
> Key: IMPALA-9957
> URL: https://issues.apache.org/jira/browse/IMPALA-9957
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.2.0, Impala 3.3.0, Impala 3.4.0
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
> Fix For: Impala 4.0
>
>
> Queries to reproduce the crash using the testdata:
> {code:sql}
> create table bigstrs stored as parquet as
> select *, repeat(uuid(), cast(random() * 10 as int)) as bigstr
> from functional.alltypes;
> set MAX_ROW_SIZE=3.5MB;
> set MEM_LIMIT=4GB;
> create table my_str_group stored as parquet as
> select group_concat(string_col) as ss, bigstr
> from bigstrs group by bigstr;
> {code}
> The last query 1) has large rows, 2) needs spilling in aggregation 3) has
> aggregation on functions needs serialize (e.g. group_concat, appx_median,
> min(string), etc). With these 3 conditions, it will trigger this bug.
> The crash stacktraces are different in different build modes. Crash
> stacktrace in RELEASE build with codegen enabled:
> {code:java}
> Thread 316 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 0x7f5fba83db3c
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 437 + 0x2]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 libstdc++.so.6.0.24 + 0x120b28
> 7
> impalad!apache::hive::service::cli::thrift::TColumnValue::printTo(std::ostream&)
> const [converter_lexical_streams.hpp : 161 + 0x8]
> 8 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 9 impalad!tc_newarray + 0x171
> {code}
> Crash stacktrace in RELEASE build with codegen disabled (set
> DISABLE_CODEGEN=true):
> {code:java}
> Thread 320 (crashed)
> 0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
> 1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
> [grouping-aggregator-partition.cc : 180 + 0x9]
> 2 impalad!impala::GroupingAggregator::SpillPartition(bool)
> [grouping-aggregator.cc : 904 + 0x10]
> 3 impalad!impala::Status
> impala::GroupingAggregator::AddBatchImpl(impala::RowBatch*,
> impala::TPrefetchMode::type, impala::HashTableCtx*)
> [grouping-aggregator-ir.cc : 148 + 0x11]
> 4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
> impala::RowBatch*) [grouping-aggregator.cc : 439 + 0x5]
> 5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
> [aggregation-node.cc : 70 + 0x6]
> 6 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc
> : 396 + 0x11]
> 7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc
> : 97 + 0x12]
> 8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 815 + 0x19]
> 9 impalad!impala::Thread::SuperviseThread(std::__cxx11::basic_string std::char_traits, std::allocator > const&,
> std::__cxx11::basic_string, std::allocator
> > const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*) [function_template.hpp : 770
> + 0x7]
> 10 impalad!boost::detail::thread_data (*)(std::__cxx11::basic_string,
> std::allocator > const&, std::__cxx11::basic_string std::char_traits, std::allocator > const&, boost::function ()>, impala::ThreadDebugInfo const*, impala::Promise (impala::PromiseMode)0>*),
> boost::_bi::list5 std::char_traits, std::allocator > >,
> boost::_bi::value,
> std::allocator > >, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> > >
> >::run() [bind.hpp : 531 + 0xc]
> 11 impalad!thread_proxy + 0x72
> 12 libpthread-2.23.so + 0x76ba
> 13 libc-2.23.so + 0x1074dd
> {code}
> Crash stacktrace in DEBUG build with codegen disabled is a bit ealier -
> crashed at a DCHECK:
> {code:java}
> F0715 20:29:24.389505 16868 grouping-aggregator-partition.cc:125]
> 1d4b40df02e6ad76:433ed5740003] Check failed: !status.ok() Stream was
> unpinned - AddRow() only fails on error
> *** Check failure stack trace: ***
> @ 0x513f31c google::LogMessage::Fail()
> @ 0x5140c0c google::LogMessage::SendToLog()
> @ 0x513ec7a google::LogMessage::Flush(
[jira] [Updated] (IMPALA-9957) Impalad crashes when serializing large rows in aggregation spilling
[
https://issues.apache.org/jira/browse/IMPALA-9957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-9957:
---
Description:
Queries to reproduce the crash using the testdata:
{code:sql}
create table bigstrs stored as parquet as
select *, repeat(uuid(), cast(random() * 10 as int)) as bigstr
from functional.alltypes;
set MAX_ROW_SIZE=3.5MB;
set MEM_LIMIT=4GB;
create table my_str_group stored as parquet as
select group_concat(string_col) as ss, bigstr
from bigstrs group by bigstr;
{code}
The last query 1) has large rows, 2) needs spilling in aggregation 3) has
aggregation on functions needs serialize (e.g. group_concat, appx_median,
min(string), etc). With these 3 conditions, it will trigger this bug.
The crash stacktraces are different in different build modes. Crash stacktrace
in RELEASE build with codegen enabled:
{code:java}
Thread 316 (crashed)
0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
[grouping-aggregator-partition.cc : 180 + 0x9]
2 impalad!impala::GroupingAggregator::SpillPartition(bool)
[grouping-aggregator.cc : 904 + 0x10]
3 0x7f5fba83db3c
4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
impala::RowBatch*) [grouping-aggregator.cc : 437 + 0x2]
5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
[aggregation-node.cc : 70 + 0x6]
6 libstdc++.so.6.0.24 + 0x120b28
7
impalad!apache::hive::service::cli::thrift::TColumnValue::printTo(std::ostream&)
const [converter_lexical_streams.hpp : 161 + 0x8]
8 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc :
396 + 0x11]
9 impalad!tc_newarray + 0x171
{code}
Crash stacktrace in RELEASE build with codegen disabled (set
DISABLE_CODEGEN=true):
{code:java}
Thread 320 (crashed)
0 impalad!impala::HashTable::Close() [hash-table.cc : 512 + 0x0]
1 impalad!impala::GroupingAggregator::Partition::Spill(bool)
[grouping-aggregator-partition.cc : 180 + 0x9]
2 impalad!impala::GroupingAggregator::SpillPartition(bool)
[grouping-aggregator.cc : 904 + 0x10]
3 impalad!impala::Status
impala::GroupingAggregator::AddBatchImpl(impala::RowBatch*,
impala::TPrefetchMode::type, impala::HashTableCtx*) [grouping-aggregator-ir.cc
: 148 + 0x11]
4 impalad!impala::GroupingAggregator::AddBatch(impala::RuntimeState*,
impala::RowBatch*) [grouping-aggregator.cc : 439 + 0x5]
5 impalad!impala::AggregationNode::Open(impala::RuntimeState*)
[aggregation-node.cc : 70 + 0x6]
6 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc :
396 + 0x11]
7 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc :
97 + 0x12]
8 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
[query-state.cc : 815 + 0x19]
9 impalad!impala::Thread::SuperviseThread(std::__cxx11::basic_string, std::allocator > const&,
std::__cxx11::basic_string, std::allocator >
const&, boost::function, impala::ThreadDebugInfo const*,
impala::Promise*) [function_template.hpp : 770 +
0x7]
10 impalad!boost::detail::thread_data,
std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, boost::function, impala::ThreadDebugInfo const*, impala::Promise*),
boost::_bi::list5, std::allocator > >,
boost::_bi::value,
std::allocator > >, boost::_bi::value >,
boost::_bi::value,
boost::_bi::value*> > > >::run()
[bind.hpp : 531 + 0xc]
11 impalad!thread_proxy + 0x72
12 libpthread-2.23.so + 0x76ba
13 libc-2.23.so + 0x1074dd
{code}
Crash stacktrace in DEBUG build with codegen disabled is a bit ealier - crashed
at a DCHECK:
{code:java}
F0715 20:29:24.389505 16868 grouping-aggregator-partition.cc:125]
1d4b40df02e6ad76:433ed5740003] Check failed: !status.ok() Stream was
unpinned - AddRow() only fails on error
*** Check failure stack trace: ***
@ 0x513f31c google::LogMessage::Fail()
@ 0x5140c0c google::LogMessage::SendToLog()
@ 0x513ec7a google::LogMessage::Flush()
@ 0x5142878 google::LogMessageFatal::~LogMessageFatal()
@ 0x28b2ca7
impala::GroupingAggregator::Partition::SerializeStreamForSpilling()
@ 0x28b360f impala::GroupingAggregator::Partition::Spill()
@ 0x28a4122 impala::GroupingAggregator::SpillPartition()
@ 0x28b169c impala::GroupingAggregator::AddIntermediateTuple<>()
@ 0x28b09d9 impala::GroupingAggregator::ProcessRow<>()
@ 0x28af535 impala::GroupingAggregator::AddBatchImpl<>()
@ 0x289f6ad impala::GroupingAggregator::AddBatch()
@ 0x28db463 impala::AggregationNode::Open()
@ 0x22598bd impala::FragmentInstanceState::Open()
@ 0x22562f4 impala::FragmentInstanceState::Exec()
@ 0x22801ed impala::QueryState::ExecFInstance()
@ 0x227e5ef _ZZN6impala10QuerySt
