[jira] [Resolved] (IMPALA-8833) Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32) in BatchedBitReader::UnpackBatch()
[
https://issues.apache.org/jira/browse/IMPALA-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-8833.
---
Resolution: Fixed
> Check failed: bit_width <= sizeof(T) * 8 (40 vs. 32) in
> BatchedBitReader::UnpackBatch()
>
>
> Key: IMPALA-8833
> URL: https://issues.apache.org/jira/browse/IMPALA-8833
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Daniel Becker
>Priority: Blocker
> Labels: broken-build, crash, flaky
>
> {noformat}
> F0801 21:24:10.571285 15993 bit-stream-utils.inline.h:126]
> d04ba69d5da8ffd1:a9045b820001] Check failed: bit_width <= sizeof(T) * 8
> (40 vs. 32)
> *** Check failure stack trace: ***
> @ 0x52f63ac google::LogMessage::Fail()
> @ 0x52f7c51 google::LogMessage::SendToLog()
> @ 0x52f5d86 google::LogMessage::Flush()
> @ 0x52f934d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2b265b5 impala::BatchedBitReader::UnpackBatch<>()
> @ 0x2ae8623 impala::RleBatchDecoder<>::FillLiteralBuffer()
> @ 0x2b2cadb impala::RleBatchDecoder<>::DecodeLiteralValues<>()
> @ 0x2b27bfb impala::DictDecoder<>::DecodeNextValue()
> @ 0x2b16fed
> impala::ScalarColumnReader<>::ReadSlotsNoConversion()
> @ 0x2ac7252 impala::ScalarColumnReader<>::ReadSlots()
> @ 0x2a76cef
> impala::ScalarColumnReader<>::MaterializeValueBatchRepeatedDefLevel()
> @ 0x2a58faa impala::ScalarColumnReader<>::ReadValueBatch<>()
> @ 0x2a20e8e
> impala::ScalarColumnReader<>::ReadNonRepeatedValueBatch()
> @ 0x29b189c impala::HdfsParquetScanner::AssembleRows()
> @ 0x29ac6de impala::HdfsParquetScanner::GetNextInternal()
> @ 0x29aa656 impala::HdfsParquetScanner::ProcessSplit()
> @ 0x249172d impala::HdfsScanNode::ProcessSplit()
> @ 0x2490902 impala::HdfsScanNode::ScannerThread()
> @ 0x248fc8b
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x2492253
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6915
> Log lines around the failure:
> {noformat}
> [gw5] PASSED
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: parquet/none]
> query_test/test_nested_types.py::TestMaxNestingDepth::test_load_hive_table[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_scanners.py::TestParquet::test_bad_compression_codec[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q7[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw1] PASSED
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q8[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> query_test/test_tpcds_queries.py::TestTpcdsQuery::test_tpcds_q10a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw10] PASSED
> query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_decimal_tbl[protocol:
> bee
[jira] [Created] (IMPALA-8841) Dataload flakiness: hive-exec-*.jar changed on src filesystem
Csaba Ringhofer created IMPALA-8841:
---
Summary: Dataload flakiness: hive-exec-*.jar changed on src
filesystem
Key: IMPALA-8841
URL: https://issues.apache.org/jira/browse/IMPALA-8841
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.3.0
Reporter: Csaba Ringhofer
The following exception appears time to time in Hive 3 +Tez builds:
{code}
Failing this attempt.Diagnostics: [2019-07-25 11:26:46.931]Resource
hdfs://localhost:20500/user/ubuntu/.hiveJars/hive-exec-3.1.0.7.0.0.0-280-f198e31861337336ae45a4c9779fc5db5af5d097a4b2000f8a96876573d03c8e.jar
changed on src filesystem (expected 1564054001302, was 1564054001832
java.io.IOException: Resource
hdfs://localhost:20500/user/ubuntu/.hiveJars/hive-exec-3.1.0.7.0.0.0-280-f198e31861337336ae45a4c9779fc5db5af5d097a4b2000f8a96876573d03c8e.jar
changed on src filesystem (expected 1564054001302, was 1564054001832
at
org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:273)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:242)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:235)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:223)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IMPALA-8842) Impala should set correct value for 'engine' column in TAB_COL_STATS HMS table and read stats accordingly
Attila Jeges created IMPALA-8842: Summary: Impala should set correct value for 'engine' column in TAB_COL_STATS HMS table and read stats accordingly Key: IMPALA-8842 URL: https://issues.apache.org/jira/browse/IMPALA-8842 Project: IMPALA Issue Type: New Feature Components: Catalog Affects Versions: Impala 3.3.0 Reporter: Attila Jeges Assignee: Attila Jeges Note that Impala doesn't use PART_COL_STATS HMS table at all, therefore the 'engine' value needs to be set for TAB_COL_STATS HMS table only. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (IMPALA-8771) Missing stats warning for complex type columns
[
https://issues.apache.org/jira/browse/IMPALA-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamas Mate resolved IMPALA-8771.
Resolution: Fixed
> Missing stats warning for complex type columns
> --
>
> Key: IMPALA-8771
> URL: https://issues.apache.org/jira/browse/IMPALA-8771
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 2.13.0, Impala 3.3.0
>Reporter: bharath v
>Assignee: Tamas Mate
>Priority: Minor
> Labels: observability
> Fix For: Impala 3.3.0
>
>
> We currently don't support column stats for complex typed columns (ingored in
> `compute stats` statements). However running queries against those columns
> throws the missing col stats warning which is confusing.
>
> {noformat}
> select count(*) from
> customers c,
> c.orders o;{noformat}
> {noformat}
> Max Per-Host Resource Reservation: Memory=16.00KB Threads=3
> Per-Host Resource Estimates: Memory=36MB
> WARNING: The following tables are missing relevant table and/or column
> statistics.
> default.customers
> Analyzed query: SELECT count(*) FROM `default`.customers c, c.orders
> o{noformat}
>
> We could probably skip the warnings if we detect the missing stats are for
> complex typed columns, until we support them.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IMPALA-8843) Restrict bit unpacking to unsigned integer types
Daniel Becker created IMPALA-8843: - Summary: Restrict bit unpacking to unsigned integer types Key: IMPALA-8843 URL: https://issues.apache.org/jira/browse/IMPALA-8843 Project: IMPALA Issue Type: Improvement Reporter: Daniel Becker Assignee: Daniel Becker Restrict bit unpacking to the unsigned integer types uint8_t, uint16_t, uint32_t and uint64_t. It is straightforward how to unpack to these types and less so with signed types. Instead of bool, we can use uint8_t and possibly cast it to bool. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IMPALA-8844) Decouple kinit renewal thread from Keberos configuration.
bharath v created IMPALA-8844:
-
Summary: Decouple kinit renewal thread from Keberos configuration.
Key: IMPALA-8844
URL: https://issues.apache.org/jira/browse/IMPALA-8844
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: bharath v
Currently, Impala starts a kinit renewal thread only when kerberos is enabled,
{noformat}
Status SecureAuthProvider::Start() {
// True for kerberos internal use
if (needs_kinit_) {
DCHECK(is_internal_);
DCHECK(!principal_.empty());
// IMPALA-8154: Disable any Kerberos auth_to_local mappings.
FLAGS_use_system_auth_to_local = false;
// Starts a thread that periodically does a 'kinit'. The thread lives
as long as the
// process does.
KUDU_RETURN_IF_ERROR(kudu::security::InitKerberosForServer(principal_,
keytab_file_,
FLAGS_krb5_ccname, false), "Could not init kerberos"); < starts
the thread
LOG(INFO) << "Kerberos ticket granted to " << principal_;
}
{noformat}
There can be cases where Impala's internal connections are *not* kerberized but
communication with external components like HMS/Ranger/Atlas could be
kerberized. In such setups, Impala process doesn't have a tgt initialized
resulting in failing connections to these components.
We could start with decoupling the kinit thread from the kerberos configuration
and have it run in cases where any communication is kerberized (use a flag?).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IMPALA-8845) Close ExecNode tree prior to calling FlushFinal in FragmentInstanceState
Sahil Takiar created IMPALA-8845:
Summary: Close ExecNode tree prior to calling FlushFinal in
FragmentInstanceState
Key: IMPALA-8845
URL: https://issues.apache.org/jira/browse/IMPALA-8845
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
While testing IMPALA-8818, I found that IMPALA-8780 does not always cause all
non-coordinator fragments to shutdown. In certain setups, TopN queries
({{select * from [table] order by [col] limit [limit]}}) where all results are
successfully spooled, still keep non-coordinator fragments alive.
The issue is that sometimes the {{DATASTREAM SINK}} for the TopN <-- Scan Node
fragment ends up blocking waiting for a response to a {{TransmitData()}} RPC.
This prevents the fragment from shutting down.
I haven't traced the issue exactly, but what I *think* is happening is that the
{{MERGING-EXCHANGE}} operator in the coordinator fragment hits {{eos}} whenever
it has received enough rows to reach the limit defined in the query, which
could occur before the {{DATASTREAM SINK}} sends all the rows from the TopN /
Scan Node fragment.
So the TopN / Scan Node fragments end up hanging until they are explicitly
closed.
The fix is to close the {{ExecNode}} tree in {{FragmentInstanceState}} as
eagerly as possible. Moving the close call to before the call to
{{DataSink::FlushFinal}} fixes the issue. It has the added benefit that it
shuts down and releases all {{ExecNode}} resources as soon as it can. When
result spooling is enabled, this is particularly important because
{{FlushFinal}} might block until the consumer reads all rows.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
