[jira] [Updated] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated IMPALA-7834: - Attachment: stacktrace_125057_05Nov.txt stacktrace_117834_06Nov2018.txt stacktrace_74831_28Oct.txt stacktrace_72492_01Nov2018.txt > Impala different types of crashes > - > > Key: IMPALA-7834 > URL: https://issues.apache.org/jira/browse/IMPALA-7834 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Manikandan R >Priority: Major > Labels: CDH5, crash > Attachments: stacktrace_117834_06Nov2018.txt, > stacktrace_125057_05Nov.txt, stacktrace_72492_01Nov2018.txt, > stacktrace_74831_28Oct.txt > > > Off late, We had witnessed different types of crashes in cluster. I don't see > any similarities among crashes stack traces and also not able to reproduce. > Below are the stack traces occurred in different daemons at different > timings. I do have complete stack traces and let me know if it helps for > further debugging. > 1) > 10-1-33-172 > Nov 6 18:04 > Thread 1 (Thread 0x7f018b423700 (LWP 96325)): > #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7f0aad280185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7f0aad422593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7f0aad28568f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7f0a44ca3000 in ?? () > No symbol table info available. > #8 0x00dba5c1 in > impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () > No symbol table info available. > #9 0x00dba924 in > impala::HdfsParquetScanner::AssembleRows(std::vector std::allocator > const&, impala::RowBatch*, > bool*) () > No symbol table info available. > #10 0x00dbf5f6 in > impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () > No symbol table info available. > #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () > No symbol table info available. > #12 0x00d835e6 in > impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*, > impala::io::ScanRange*) () > No symbol table info available. > #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () > No symbol table info available. > #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function, impala::Promise*) () > No symbol table info available. > #15 0x00d173c4 in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function, > impala::Promise*), boost::_bi::list4, > boost::_bi::value, boost::_bi::value >, > boost::_bi::value*> > > >::run() () > No symbol table info available. > #16 0x0128fada in thread_proxy () > No symbol table info available. > #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 > No symbol table info available. > Nov 1 11:21 > Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): > #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7fc297d12185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fc297eb4593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7fc297d1768f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7fc1dd9f in ?? () > No symbol table info available. > #8 0x00fdd9f9 in > impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () > No symbol table info available. > #9 0x00b74d6d in impala::FragmentInstanceState::Open() () > No symbol table info
[jira] [Commented] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680993#comment-16680993 ] Manikandan R commented on IMPALA-7834: -- We are using CDH 5.14.4. Attached stack traces for each incident as well. > Impala different types of crashes > - > > Key: IMPALA-7834 > URL: https://issues.apache.org/jira/browse/IMPALA-7834 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Manikandan R >Priority: Major > Labels: CDH5, crash > > Off late, We had witnessed different types of crashes in cluster. I don't see > any similarities among crashes stack traces and also not able to reproduce. > Below are the stack traces occurred in different daemons at different > timings. I do have complete stack traces and let me know if it helps for > further debugging. > 1) > 10-1-33-172 > Nov 6 18:04 > Thread 1 (Thread 0x7f018b423700 (LWP 96325)): > #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7f0aad280185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7f0aad422593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7f0aad28568f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7f0a44ca3000 in ?? () > No symbol table info available. > #8 0x00dba5c1 in > impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () > No symbol table info available. > #9 0x00dba924 in > impala::HdfsParquetScanner::AssembleRows(std::vector std::allocator > const&, impala::RowBatch*, > bool*) () > No symbol table info available. > #10 0x00dbf5f6 in > impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () > No symbol table info available. > #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () > No symbol table info available. > #12 0x00d835e6 in > impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*, > impala::io::ScanRange*) () > No symbol table info available. > #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () > No symbol table info available. > #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function, impala::Promise*) () > No symbol table info available. > #15 0x00d173c4 in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function, > impala::Promise*), boost::_bi::list4, > boost::_bi::value, boost::_bi::value >, > boost::_bi::value*> > > >::run() () > No symbol table info available. > #16 0x0128fada in thread_proxy () > No symbol table info available. > #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 > No symbol table info available. > Nov 1 11:21 > Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): > #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7fc297d12185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fc297eb4593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7fc297d1768f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7fc1dd9f in ?? () > No symbol table info available. > #8 0x00fdd9f9 in > impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () > No symbol table info available. > #9 0x00b74d6d in impala::FragmentInstanceState::Open() () > No symbol table info available. > #10 0x00b763ab in impala::FragmentInstanceState::Exec() () > No symbol table info available. > #11 0x00b65b38 in > impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () > No
[jira] [Resolved] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
[
https://issues.apache.org/jira/browse/IMPALA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-7835.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Creating a role with the same name as the user name with object ownership
> enabled can cause INVALIDATE METADATA to hang
> ---
>
> Key: IMPALA-7835
> URL: https://issues.apache.org/jira/browse/IMPALA-7835
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Critical
> Fix For: Impala 3.1.0
>
>
> Start Impala with object ownership enabled.
> {noformat}
> [localhost:21000] default> create database foo;
> [localhost:21000] default> create role impdev;
> [localhost:21000] default> grant all on server to role impdev;
> [localhost:21000] default> grant role impdev to group impdev;
> [localhost:21000] default> invalidate metadata; -- this will hang
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
[
https://issues.apache.org/jira/browse/IMPALA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya resolved IMPALA-7835.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Creating a role with the same name as the user name with object ownership
> enabled can cause INVALIDATE METADATA to hang
> ---
>
> Key: IMPALA-7835
> URL: https://issues.apache.org/jira/browse/IMPALA-7835
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Critical
> Fix For: Impala 3.1.0
>
>
> Start Impala with object ownership enabled.
> {noformat}
> [localhost:21000] default> create database foo;
> [localhost:21000] default> create role impdev;
> [localhost:21000] default> grant all on server to role impdev;
> [localhost:21000] default> grant role impdev to group impdev;
> [localhost:21000] default> invalidate metadata; -- this will hang
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-5904) Enable ThreadSanitizer for Impala
[ https://issues.apache.org/jira/browse/IMPALA-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-5904: -- Assignee: (was: Michal Ostrowski) > Enable ThreadSanitizer for Impala > - > > Key: IMPALA-5904 > URL: https://issues.apache.org/jira/browse/IMPALA-5904 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Priority: Major > Attachments: impalad.ERROR > > > It would be great to be able to automatically detect data races in Impala > using ThreadSanitizer to avoid tricky-to-reproduce bugs. This issue tracks > enabling ThreadSanitizer, fixing bugs and adding suppressions to get to the > point where Impala runs cleanly with the sanitizer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6852) Add a section to the query profile covering number of fragments and total bytes scanned per host
[ https://issues.apache.org/jira/browse/IMPALA-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-6852: -- Assignee: (was: Michal Ostrowski) > Add a section to the query profile covering number of fragments and total > bytes scanned per host > > > Key: IMPALA-6852 > URL: https://issues.apache.org/jira/browse/IMPALA-6852 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.13.0 >Reporter: Mostafa Mokhtar >Priority: Major > Labels: supportability > > Root causing performance issues usually comes down to skew in terms of work > done across hosts. Un-even assignment of fragments or bytes scanned per host > is a common. > Making this information readily available in the query profile will help > speedup RCA. > Proposal is to add two tables to the query profile that cover > * Number of fragments per host > * Number of bytes scanned per host -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7183) We should print the sender name when logging a report for an unknown status report on the coordinatior
[ https://issues.apache.org/jira/browse/IMPALA-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-7183: -- Assignee: (was: Michal Ostrowski) > We should print the sender name when logging a report for an unknown status > report on the coordinatior > -- > > Key: IMPALA-7183 > URL: https://issues.apache.org/jira/browse/IMPALA-7183 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Distributed Exec >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Priority: Critical > Labels: ramp-up > > We should print the sender name when logging a report for an unknown status > report on the coordinatior in > [impala-server.cc:1229|https://github.com/apache/impala/blob/e7d5a25a4516337ef651983b1d945abf06c3a831/be/src/service/impala-server.cc#L1229]. > That will help identify backends with stuck fragment instances who fail to > get cancelled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6590) Disable expr rewrites and codegen for VALUES() statements
[
https://issues.apache.org/jira/browse/IMPALA-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-6590:
--
Assignee: (was: Michal Ostrowski)
> Disable expr rewrites and codegen for VALUES() statements
> -
>
> Key: IMPALA-6590
> URL: https://issues.apache.org/jira/browse/IMPALA-6590
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0
>Reporter: Alexander Behm
>Priority: Major
> Labels: perf, planner, regression
>
> The analysis of statements with big VALUES clauses like INSERT INTO
> VALUES is slow due to expression rewrites like constant folding. The
> performance of such statements has regressed since the introduction of expr
> rewrites and constant folding in IMPALA-1788.
> We should skip expr rewrites for VALUES altogether since it mostly provides
> no benefit but can have a large overhead due to evaluation of expressions in
> the backend (constant folding). These expressions are ultimately evaluated
> and materialized in the backend anyway, so there's no point in folding them
> during analysis.
> Similarly, there is no point in doing codegen for these exprs in the backend
> union node.
> *Workaround*
> {code}
> SET ENABLE_EXPR_REWRITES=FALSE;
> SET DISABLE_CODEGEN=TRUE;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-2515) Impala is unable to read a Parquet decimal column if size is larger than needed
[
https://issues.apache.org/jira/browse/IMPALA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-2515:
--
Assignee: (was: Michal Ostrowski)
> Impala is unable to read a Parquet decimal column if size is larger than
> needed
> ---
>
> Key: IMPALA-2515
> URL: https://issues.apache.org/jira/browse/IMPALA-2515
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Affects Versions: Impala 2.3.0
>Reporter: Taras Bobrovytsky
>Priority: Minor
> Labels: ramp-up
>
> Impala cannot read this:
> {code}
> {"name": "tmp_1",
> "type": "fixed",
> "size": 8,
> "logicalType": "decimal",
> "precision": 10,
> "scale": 5}
> {code}
> However, this can be read:
> {code}
> {"name": "tmp_1",
> "type": "fixed",
> "size": 5,
> "logicalType": "decimal",
> "precision": 10,
> "scale": 5}
> {code}
> Size must be precisely set to this, or Impala is unable to read the decimal
> column:
> {code}
> size = int(math.ceil((math.log(2, 10) + precision) / math.log(256, 10)))
> {code}
> There is nothing in the Parquet spec that says that Decimal columns must be
> sized precisely. Arguably it's a bug in the writer if it's doing it, because
> it's just wasting space.
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7482) Deadlock with unknown lock holder in JVM in java.security.Provider.getService()
[
https://issues.apache.org/jira/browse/IMPALA-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680682#comment-16680682
]
Joe McDonnell commented on IMPALA-7482:
---
[~dgarg] I'm not actively looking at this beyond IMPALA-7738
> Deadlock with unknown lock holder in JVM in
> java.security.Provider.getService()
> ---
>
> Key: IMPALA-7482
> URL: https://issues.apache.org/jira/browse/IMPALA-7482
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: hang
> Attachments: vb1220-jstack2.out
>
>
> We've seen several instances of these mystery deadlocks in impalad's embedded
> JVM. The signature is a deadlock stemming from sun.security.provider.Sun
> being locked by an unknown owner.
> {noformat}
> Found one Java-level deadlock:
> =
> "Thread-24":
> waiting to lock monitor 0x12364688 (object 0x8027ef30, a
> sun.security.provider.Sun),
> which is held by UNKNOWN_owner_addr=0x14120800
> {noformat}
> If this happens in HDFS, it causes HDFS I/O to hang and queries to get stuck.
> If it happens in the Kudu client it also causes hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-2343) Capture operator timing information covering open/close & first/last batch close
[ https://issues.apache.org/jira/browse/IMPALA-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680661#comment-16680661 ] Tim Armstrong commented on IMPALA-2343: --- Attached some initial work I did in figuring out the best way to do this. The key things are: * It needs to be low overhead - getting the current time on every GetNext() call may add measurable overhead for some queries. * It needs to work correctly for subplans. > Capture operator timing information covering open/close & first/last batch > close > > > Key: IMPALA-2343 > URL: https://issues.apache.org/jira/browse/IMPALA-2343 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.2.4 >Reporter: Mostafa Mokhtar >Priority: Minor > Labels: performance, supportability > Attachments: > 0001-Add-start-and-end-time-to-a-bunch-of-plan-nodes.patch > > > Currently Impala query profile doesn't cover operator level timeline, which > makes it difficult to understand the query timeline and fragment dependencies. > Such information will allow us to provide query swim lanes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-2343) Capture operator timing information covering open/close & first/last batch close
[ https://issues.apache.org/jira/browse/IMPALA-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-2343: -- Attachment: 0001-Add-start-and-end-time-to-a-bunch-of-plan-nodes.patch > Capture operator timing information covering open/close & first/last batch > close > > > Key: IMPALA-2343 > URL: https://issues.apache.org/jira/browse/IMPALA-2343 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.2.4 >Reporter: Mostafa Mokhtar >Priority: Minor > Labels: performance, supportability > Attachments: > 0001-Add-start-and-end-time-to-a-bunch-of-plan-nodes.patch > > > Currently Impala query profile doesn't cover operator level timeline, which > makes it difficult to understand the query timeline and fragment dependencies. > Such information will allow us to provide query swim lanes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Issue Comment Deleted] (IMPALA-6189) Guard blocking function calls with a kernel watchdog
[
https://issues.apache.org/jira/browse/IMPALA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Garg updated IMPALA-6189:
Comment: was deleted
(was: Triage logs : [~bharathv] had a patch a while back. Todd suggested a
diffrent approach, [~bharathv] to further look into this.)
> Guard blocking function calls with a kernel watchdog
>
>
> Key: IMPALA-6189
> URL: https://issues.apache.org/jira/browse/IMPALA-6189
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Lars Volker
>Assignee: bharath v
>Priority: Major
> Labels: observability, supportability
>
> Kudu uses a [watchdog
> thread|https://github.com/apache/kudu/blob/master/src/kudu/util/kernel_stack_watchdog.h]
> to log warning about threads that seem stuck.
> {quote}
> Before performing some operation which may stall (eg IO) or which we expect
> should be short (e.g. a callback on a critical thread that should not block),
> threads may mark themselves as "watched", with a threshold beyond which they
> would like warnings to be emitted including their stack trace at that time.
> {quote}
> We already pulled kudu/util into our codebase and it seems after starting a
> watchdog thread we should be able to use this, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7233) Impala 3.1 Doc: Doc the support for IANA timezone database
[ https://issues.apache.org/jira/browse/IMPALA-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7233: Summary: Impala 3.1 Doc: Doc the support for IANA timezone database (was: Impala 3.1 Doc: Doc the support for IANA time zone database) > Impala 3.1 Doc: Doc the support for IANA timezone database > -- > > Key: IMPALA-7233 > URL: https://issues.apache.org/jira/browse/IMPALA-7233 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7227) Swimlanes visualisation for debugging
[ https://issues.apache.org/jira/browse/IMPALA-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Garg reassigned IMPALA-7227: --- Assignee: Philip Zeyliger > Swimlanes visualisation for debugging > - > > Key: IMPALA-7227 > URL: https://issues.apache.org/jira/browse/IMPALA-7227 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Tim Armstrong >Assignee: Philip Zeyliger >Priority: Major > Labels: observability > Attachments: image-2018-06-29-15-54-13-596.png > > > !image-2018-06-29-15-54-13-596.png|thumbnail! > We implemented a hack version of this timeline which shows when different > plan nodes are active. The nodes are grouped by pipeline. > cc [~joemcdonnell][~janulatha][~bikramjeet.vig] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6195) Provide a timeline view of fragment instance execution events (swimlane)
[ https://issues.apache.org/jira/browse/IMPALA-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680599#comment-16680599 ] Dinesh Garg commented on IMPALA-6195: - [~tarmstr...@cloudera.com] and [~lv] , Is this and IMPALA-7227 same? > Provide a timeline view of fragment instance execution events (swimlane) > > > Key: IMPALA-6195 > URL: https://issues.apache.org/jira/browse/IMPALA-6195 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Distributed Exec >Reporter: Lars Volker >Priority: Major > Labels: observability > > It would help debug distributed performance issues to have a swimlane view of > fragment instance execution events, such as calls to Open(), GetNext(), > Close, first and last batch being sent, etc. This example could be a good > starting point: http://bl.ocks.org/bunkat/1962173 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7227) Swimlanes visualisation for debugging
[ https://issues.apache.org/jira/browse/IMPALA-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Garg updated IMPALA-7227: Labels: observability (was: ) > Swimlanes visualisation for debugging > - > > Key: IMPALA-7227 > URL: https://issues.apache.org/jira/browse/IMPALA-7227 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Tim Armstrong >Priority: Major > Labels: observability > Attachments: image-2018-06-29-15-54-13-596.png > > > !image-2018-06-29-15-54-13-596.png|thumbnail! > We implemented a hack version of this timeline which shows when different > plan nodes are active. The nodes are grouped by pipeline. > cc [~joemcdonnell][~janulatha][~bikramjeet.vig] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6754) Exchange node includes FirstBatchArrivalWaitTime in summary
[
https://issues.apache.org/jira/browse/IMPALA-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-6754:
-
Assignee: Michael Ho
> Exchange node includes FirstBatchArrivalWaitTime in summary
> ---
>
> Key: IMPALA-6754
> URL: https://issues.apache.org/jira/browse/IMPALA-6754
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Balazs Jeszenszky
>Assignee: Michael Ho
>Priority: Minor
>
> In the following execution summary:
> {code:java}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> ---
> 15:AGGREGATE 1 141.556us 141.556us 1 1 20.00 KB
> 10.00 MB FINALIZE
> 14:EXCHANGE1 2h46m 2h46m 1 1 0
> 0 UNPARTITIONED
> 09:AGGREGATE 1 16s442ms 16s442ms 1 1 768.00 KB
> 10.00 MB
> 08:HASH JOIN 1 1h38m 1h38m 2.63B -1 122.64 MB
> 2.00 GB LEFT OUTER JOIN, BROADCAST
> [...]
> {code}
> the timer for the EXCHANGE node is misleading. It's unlikely that sending a
> single row across network took so long, and individual counters confirm:
> {code:java}
> EXCHANGE_NODE (id=14):(Total: 2h46m, non-child: 2h46m, % non-child:
> 100.00%)
> - ConvertRowBatchTime: 901.000ns
> - PeakMemoryUsage: 0
> - RowsReturned: 1 (1)
> - RowsReturnedRate: 0
> DataStreamReceiver:
>- BytesReceived: 16.00 B (16)
>- DeserializeRowBatchTimer: 4.965us
>- FirstBatchArrivalWaitTime: 2h46m
>- PeakMemoryUsage: 4.01 KB (4104)
>- SendersBlockedTimer: 0.000ns
>- SendersBlockedTotalTimer(*): 0.000ns
> {code}
> In this case, the underlying joins took long (which is correctly reported in
> the rest of the profile).
> Exchange timers should reflect the time it took to transfer rows across
> network.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6189) Guard blocking function calls with a kernel watchdog
[
https://issues.apache.org/jira/browse/IMPALA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680577#comment-16680577
]
Dinesh Garg commented on IMPALA-6189:
-
Triage logs : [~bharathv] had a patch a while back. Todd suggested a diffrent
approach, [~bharathv] to further look into this.
> Guard blocking function calls with a kernel watchdog
>
>
> Key: IMPALA-6189
> URL: https://issues.apache.org/jira/browse/IMPALA-6189
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Lars Volker
>Assignee: bharath v
>Priority: Major
> Labels: observability, supportability
>
> Kudu uses a [watchdog
> thread|https://github.com/apache/kudu/blob/master/src/kudu/util/kernel_stack_watchdog.h]
> to log warning about threads that seem stuck.
> {quote}
> Before performing some operation which may stall (eg IO) or which we expect
> should be short (e.g. a callback on a critical thread that should not block),
> threads may mark themselves as "watched", with a threshold beyond which they
> would like warnings to be emitted including their stack trace at that time.
> {quote}
> We already pulled kudu/util into our codebase and it seems after starting a
> watchdog thread we should be able to use this, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6189) Guard blocking function calls with a kernel watchdog
[
https://issues.apache.org/jira/browse/IMPALA-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Garg updated IMPALA-6189:
Labels: observability supportability (was: observability)
> Guard blocking function calls with a kernel watchdog
>
>
> Key: IMPALA-6189
> URL: https://issues.apache.org/jira/browse/IMPALA-6189
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Lars Volker
>Assignee: bharath v
>Priority: Major
> Labels: observability, supportability
>
> Kudu uses a [watchdog
> thread|https://github.com/apache/kudu/blob/master/src/kudu/util/kernel_stack_watchdog.h]
> to log warning about threads that seem stuck.
> {quote}
> Before performing some operation which may stall (eg IO) or which we expect
> should be short (e.g. a callback on a critical thread that should not block),
> threads may mark themselves as "watched", with a threshold beyond which they
> would like warnings to be emitted including their stack trace at that time.
> {quote}
> We already pulled kudu/util into our codebase and it seems after starting a
> watchdog thread we should be able to use this, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7482) Deadlock with unknown lock holder in JVM in java.security.Provider.getService()
[
https://issues.apache.org/jira/browse/IMPALA-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680574#comment-16680574
]
Dinesh Garg commented on IMPALA-7482:
-
[~joemcdonnell] , are you currently looking into this?
> Deadlock with unknown lock holder in JVM in
> java.security.Provider.getService()
> ---
>
> Key: IMPALA-7482
> URL: https://issues.apache.org/jira/browse/IMPALA-7482
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.0
>Reporter: Tim Armstrong
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: hang
> Attachments: vb1220-jstack2.out
>
>
> We've seen several instances of these mystery deadlocks in impalad's embedded
> JVM. The signature is a deadlock stemming from sun.security.provider.Sun
> being locked by an unknown owner.
> {noformat}
> Found one Java-level deadlock:
> =
> "Thread-24":
> waiting to lock monitor 0x12364688 (object 0x8027ef30, a
> sun.security.provider.Sun),
> which is held by UNKNOWN_owner_addr=0x14120800
> {noformat}
> If this happens in HDFS, it causes HDFS I/O to hang and queries to get stuck.
> If it happens in the Kudu client it also causes hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
[ https://issues.apache.org/jira/browse/IMPALA-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680560#comment-16680560 ] Alex Rodoni commented on IMPALA-7836: - https://gerrit.cloudera.org/#/c/11914/ > Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort > conversion > --- > > Key: IMPALA-7836 > URL: https://issues.apache.org/jira/browse/IMPALA-7836 > Project: IMPALA > Issue Type: Sub-task > Components: Docs, Frontend >Affects Versions: Impala 2.9.0 >Reporter: Sahil Takiar >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we > should document. The changes in IMPALA-5004 work by estimating the amount of > memory required to run a TopN operator. The memory estimate is based on the > size of the individual tuples that need to be processed by the TopN operator, > as well as the sum of the limit and offset in the query. TopN operators don't > spill to disk so they have to keep all rows they process in memory. > If the estimated size of the working set of the TopN operator exceeds the > threshold of 'topn_bytes_limit' the TopN operator will be replaced with a > Sort operator. The Sort operator can spill to disk, but it processes all the > data (the limit and offset have no affect). So switching to Sort might incur > performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7841) Refactor QueryStmt, other analysis code for easier debugging
Paul Rogers created IMPALA-7841:
---
Summary: Refactor QueryStmt, other analysis code for easier
debugging
Key: IMPALA-7841
URL: https://issues.apache.org/jira/browse/IMPALA-7841
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Assignee: Paul Rogers
IMPALA-7808 started the process of refactoring the Analyzer code for easier
debugging. It did so by grouping {{SelectStmt}} code into a nested class, which
then allowed breaking up a large function into smaller chunks.
This ticket continues that process with two changes:
* Follow-on refactoring of {{SelectStmt}} to make some of the newly-created
functions simpler. (The first change tried to keep code unchanged as much as
possible.)
* Apply the same technique to the {{QueryStmt}} base class and to its other
subclasses.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7841) Refactor QueryStmt, other analysis code for easier debugging
Paul Rogers created IMPALA-7841:
---
Summary: Refactor QueryStmt, other analysis code for easier
debugging
Key: IMPALA-7841
URL: https://issues.apache.org/jira/browse/IMPALA-7841
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Assignee: Paul Rogers
IMPALA-7808 started the process of refactoring the Analyzer code for easier
debugging. It did so by grouping {{SelectStmt}} code into a nested class, which
then allowed breaking up a large function into smaller chunks.
This ticket continues that process with two changes:
* Follow-on refactoring of {{SelectStmt}} to make some of the newly-created
functions simpler. (The first change tried to keep code unchanged as much as
possible.)
* Apply the same technique to the {{QueryStmt}} base class and to its other
subclasses.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7840) test_concurrent_schema_change is missing a possible error message
Thomas Tauber-Marshall created IMPALA-7840: -- Summary: test_concurrent_schema_change is missing a possible error message Key: IMPALA-7840 URL: https://issues.apache.org/jira/browse/IMPALA-7840 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.1.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall test_concurrent_schema_change runs a series of alters and inserts on the same Kudu table concurrently to ensure that Impala can handle this without crashing. There is a list of expected error messages in the test. One possible legitimate error is missing, causing the test to sometimes be flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7840) test_concurrent_schema_change is missing a possible error message
Thomas Tauber-Marshall created IMPALA-7840: -- Summary: test_concurrent_schema_change is missing a possible error message Key: IMPALA-7840 URL: https://issues.apache.org/jira/browse/IMPALA-7840 Project: IMPALA Issue Type: Bug Affects Versions: Impala 3.1.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall test_concurrent_schema_change runs a series of alters and inserts on the same Kudu table concurrently to ensure that Impala can handle this without crashing. There is a list of expected error messages in the test. One possible legitimate error is missing, causing the test to sometimes be flaky. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-7839) Refactor Catalog::toCatalogObjectKey and CatalogObject::getUniqueName to reduce code repetition
Fredy Wijaya created IMPALA-7839: Summary: Refactor Catalog::toCatalogObjectKey and CatalogObject::getUniqueName to reduce code repetition Key: IMPALA-7839 URL: https://issues.apache.org/jira/browse/IMPALA-7839 Project: IMPALA Issue Type: Improvement Components: Catalog Reporter: Fredy Wijaya Some examples of code repetition. https://github.com/apache/impala/blob/7299a6aedbbb7563e465dcfc6a20a981e9f1b804/fe/src/main/java/org/apache/impala/catalog/Catalog.java#L544-L582 https://github.com/apache/impala/blob/7299a6aedbbb7563e465dcfc6a20a981e9f1b804/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java#L146-L150 https://github.com/apache/impala/blob/7299a6aedbbb7563e465dcfc6a20a981e9f1b804/fe/src/main/java/org/apache/impala/catalog/Table.java#L457-L458 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7245) test_kudu.py::test_kudu_insert() fails with "Error adding columns to Kudu table tbl_with_defaults"
[
https://issues.apache.org/jira/browse/IMPALA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7245:
--
Target Version: Impala 3.2.0 (was: Impala 3.1.0)
> test_kudu.py::test_kudu_insert() fails with "Error adding columns to Kudu
> table tbl_with_defaults"
> --
>
> Key: IMPALA-7245
> URL: https://issues.apache.org/jira/browse/IMPALA-7245
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Tianyi Wang
>Priority: Critical
> Labels: broken-build, flaky
>
> query_test.test_kudu.TestKuduOperations.test_kudu_insert hit the following
> error:
> {noformat}
> query_test/test_kudu.py:93: in test_kudu_insert
> self.run_test_case('QueryTest/kudu_insert', vector,
> use_db=unique_database)
> common/impala_test_suite.py:405: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:620: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:173: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:343: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:339: in execute_query_async
> return self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:483: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: ImpalaRuntimeException: Error adding columns to Kudu table
> tbl_with_defaults
> E CAUSED BY: NonRecoverableException: The column already exists: j{noformat}
> This is very similar to what we saw in IMPALA-6107, which we couldn't
> reproduce.
> It seems to be related to our interaction with the Kudu client. The statement
> starts executing at 23:05:11.17. From impalad.INFO:
> {noformat}
> I0629 23:05:11.170009 20602 impala-beeswax-server.cc:54] query(): query=alter
> table tbl_with_defaults add columns (j int null, k int not null default
> 1){noformat}
> This results in actions in catalogd, but catalogd encounters a timeout in the
> AsyncKuduClient while processing this and then immediately hits the error:
> {noformat}
> I0629 23:05:11.212054 28109 AsyncKuduClient.java:1756] Invalidating location
> master-127.0.0.1:7051(127.0.0.1:7051) for tablet Kudu Master: [peer
> master-127.0.0.1:7051(127.0.0.1:7051)] encountered a read timeout; closing
> the channel
> I0629 23:05:11.288261 11518 jni-util.cc:230]
> org.apache.impala.common.ImpalaRuntimeException: Error adding columns to Kudu
> table tbl_with_defaults
> at
> org.apache.impala.service.KuduCatalogOpExecutor.alterKuduTable(KuduCatalogOpExecutor.java:499)
> at
> org.apache.impala.service.KuduCatalogOpExecutor.addColumn(KuduCatalogOpExecutor.java:412)
> at
> org.apache.impala.service.CatalogOpExecutor.alterKuduTable(CatalogOpExecutor.java:600)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:420)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:270)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
> Caused by: org.apache.kudu.client.NonRecoverableException: The column already
> exists: j
> at
> org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
> at
> org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:351)
> at org.apache.kudu.client.KuduClient.alterTable(KuduClient.java:141)
> at
> org.apache.impala.service.KuduCatalogOpExecutor.alterKuduTable(KuduCatalogOpExecutor.java:494)
> ... 5 more
> {noformat}
> The Kudu master sees the alter table request, then the connection is torn
> down, then a duplicate alter table request:
> {noformat}
> I0629 23:05:11.197782 16157 catalog_manager.cc:2086] Servicing AlterTable
> request from {username='jenkins'} at 127.0.0.1:34332:
> table { table_name: "impala::test_kudu_insert_ca9324f5.tbl_with_defaults" }
> alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "j" type:
> INT32 is_key: false is_nullable: true cfile_block_size: 0 } } }
> alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "k" type:
> INT32 is_key: false is_nullable: false read_default_value: "\020\'\000\000"
> cfile_block_size: 0 } } }
> W0629 23:05:11.218549 15925 connection.cc:420] Connection torn down before
> Call kudu.master.MasterService.AlterTable from 127.0.0.1:34332 (request call
> id 24) could
[jira] [Resolved] (IMPALA-7528) Division by zero when computing cardinalities of many to many joins on NULL columns
[
https://issues.apache.org/jira/browse/IMPALA-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig resolved IMPALA-7528.
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Division by zero when computing cardinalities of many to many joins on NULL
> columns
> ---
>
> Key: IMPALA-7528
> URL: https://issues.apache.org/jira/browse/IMPALA-7528
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Balazs Jeszenszky
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: planner
> Fix For: Impala 3.1.0
>
>
> The following:
> {code:java}
> | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | Per-Host Resources: mem-estimate=33.94MB mem-reservation=1.94MB|
> | 02:HASH JOIN [INNER JOIN, BROADCAST] |
> | | hash predicates: b.code = a.code|
> | | fk/pk conjuncts: none |
> | | runtime filters: RF000 <- a.code|
> | | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB |
> | | tuple-ids=1,0 row-size=163B cardinality=9223372036854775807 |
> < Estimation due to overflow.
> | | |
> | |--03:EXCHANGE [BROADCAST] |
> | | | mem-estimate=0B mem-reservation=0B |
> | | | tuple-ids=0 row-size=82B cardinality=823 |
> | | | |
> | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | | Per-Host Resources: mem-estimate=32.00MB mem-reservation=0B |
> | | 00:SCAN HDFS [default.sample_07 a, RANDOM] |
> | | partitions=1/1 files=1 size=44.98KB |
> | | stats-rows=823 extrapolated-rows=disabled|
> | | table stats: rows=823 size=44.98KB |
> | | column stats: all|
> | | mem-estimate=32.00MB mem-reservation=0B |
> | | tuple-ids=0 row-size=82B cardinality=823 |
> | | |
> | 01:SCAN HDFS [default.sample_08 b, RANDOM] |
> |partitions=1/1 files=1 size=44.99KB |
> |runtime filters: RF000 -> b.code|
> |stats-rows=823 extrapolated-rows=disabled |
> |table stats: rows=823 size=44.99KB |
> |column stats: all |
> |mem-estimate=32.00MB mem-reservation=0B |
> |tuple-ids=1 row-size=82B cardinality=823|
> ++
> {code}
> is the result of both join columns having 0 as NDV.
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/org/apache/impala/planner/JoinNode.java#L368
> should handle this more gracefully.
> IMPALA-7310 makes it a bit more likely that someone will run into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IMPALA-7824) Running INVALIDATE METADATA with authorization enabled can cause a hang if Sentry is unavailable
[ https://issues.apache.org/jira/browse/IMPALA-7824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya updated IMPALA-7824: - Priority: Critical (was: Major) > Running INVALIDATE METADATA with authorization enabled can cause a hang if > Sentry is unavailable > > > Key: IMPALA-7824 > URL: https://issues.apache.org/jira/browse/IMPALA-7824 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.1.0 > > > Steps to reproduce: > 1. Start Impala with authorization > 2. Kill Sentry > 3. Run INVALIDATE METADATA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
[
https://issues.apache.org/jira/browse/IMPALA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya updated IMPALA-7835:
-
Priority: Critical (was: Major)
> Creating a role with the same name as the user name with object ownership
> enabled can cause INVALIDATE METADATA to hang
> ---
>
> Key: IMPALA-7835
> URL: https://issues.apache.org/jira/browse/IMPALA-7835
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Critical
>
> Start Impala with object ownership enabled.
> {noformat}
> [localhost:21000] default> create database foo;
> [localhost:21000] default> create role impdev;
> [localhost:21000] default> grant all on server to role impdev;
> [localhost:21000] default> grant role impdev to group impdev;
> [localhost:21000] default> invalidate metadata; -- this will hang
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7839) Refactor Catalog::toCatalogObjectKey and CatalogObject::getUniqueName to reduce code repetition
Fredy Wijaya created IMPALA-7839: Summary: Refactor Catalog::toCatalogObjectKey and CatalogObject::getUniqueName to reduce code repetition Key: IMPALA-7839 URL: https://issues.apache.org/jira/browse/IMPALA-7839 Project: IMPALA Issue Type: Improvement Components: Catalog Reporter: Fredy Wijaya Some examples of code repetition. https://github.com/apache/impala/blob/7299a6aedbbb7563e465dcfc6a20a981e9f1b804/fe/src/main/java/org/apache/impala/catalog/Catalog.java#L544-L582 https://github.com/apache/impala/blob/7299a6aedbbb7563e465dcfc6a20a981e9f1b804/fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java#L146-L150 https://github.com/apache/impala/blob/7299a6aedbbb7563e465dcfc6a20a981e9f1b804/fe/src/main/java/org/apache/impala/catalog/Table.java#L457-L458 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7528) Division by zero when computing cardinalities of many to many joins on NULL columns
[
https://issues.apache.org/jira/browse/IMPALA-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig resolved IMPALA-7528.
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Division by zero when computing cardinalities of many to many joins on NULL
> columns
> ---
>
> Key: IMPALA-7528
> URL: https://issues.apache.org/jira/browse/IMPALA-7528
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Balazs Jeszenszky
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: planner
> Fix For: Impala 3.1.0
>
>
> The following:
> {code:java}
> | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | Per-Host Resources: mem-estimate=33.94MB mem-reservation=1.94MB|
> | 02:HASH JOIN [INNER JOIN, BROADCAST] |
> | | hash predicates: b.code = a.code|
> | | fk/pk conjuncts: none |
> | | runtime filters: RF000 <- a.code|
> | | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB |
> | | tuple-ids=1,0 row-size=163B cardinality=9223372036854775807 |
> < Estimation due to overflow.
> | | |
> | |--03:EXCHANGE [BROADCAST] |
> | | | mem-estimate=0B mem-reservation=0B |
> | | | tuple-ids=0 row-size=82B cardinality=823 |
> | | | |
> | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | | Per-Host Resources: mem-estimate=32.00MB mem-reservation=0B |
> | | 00:SCAN HDFS [default.sample_07 a, RANDOM] |
> | | partitions=1/1 files=1 size=44.98KB |
> | | stats-rows=823 extrapolated-rows=disabled|
> | | table stats: rows=823 size=44.98KB |
> | | column stats: all|
> | | mem-estimate=32.00MB mem-reservation=0B |
> | | tuple-ids=0 row-size=82B cardinality=823 |
> | | |
> | 01:SCAN HDFS [default.sample_08 b, RANDOM] |
> |partitions=1/1 files=1 size=44.99KB |
> |runtime filters: RF000 -> b.code|
> |stats-rows=823 extrapolated-rows=disabled |
> |table stats: rows=823 size=44.99KB |
> |column stats: all |
> |mem-estimate=32.00MB mem-reservation=0B |
> |tuple-ids=1 row-size=82B cardinality=823|
> ++
> {code}
> is the result of both join columns having 0 as NDV.
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/org/apache/impala/planner/JoinNode.java#L368
> should handle this more gracefully.
> IMPALA-7310 makes it a bit more likely that someone will run into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
[ https://issues.apache.org/jira/browse/IMPALA-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7836 started by Alex Rodoni. --- > Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort > conversion > --- > > Key: IMPALA-7836 > URL: https://issues.apache.org/jira/browse/IMPALA-7836 > Project: IMPALA > Issue Type: Sub-task > Components: Docs, Frontend >Affects Versions: Impala 2.9.0 >Reporter: Sahil Takiar >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we > should document. The changes in IMPALA-5004 work by estimating the amount of > memory required to run a TopN operator. The memory estimate is based on the > size of the individual tuples that need to be processed by the TopN operator, > as well as the sum of the limit and offset in the query. TopN operators don't > spill to disk so they have to keep all rows they process in memory. > If the estimated size of the working set of the TopN operator exceeds the > threshold of 'topn_bytes_limit' the TopN operator will be replaced with a > Sort operator. The Sort operator can spill to disk, but it processes all the > data (the limit and offset have no affect). So switching to Sort might incur > performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7824) Running INVALIDATE METADATA with authorization enabled can cause a hang if Sentry is unavailable
[ https://issues.apache.org/jira/browse/IMPALA-7824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-7824. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Running INVALIDATE METADATA with authorization enabled can cause a hang if > Sentry is unavailable > > > Key: IMPALA-7824 > URL: https://issues.apache.org/jira/browse/IMPALA-7824 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > Fix For: Impala 3.1.0 > > > Steps to reproduce: > 1. Start Impala with authorization > 2. Kill Sentry > 3. Run INVALIDATE METADATA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7791) Aggregation Node memory estimates don't account for number of fragment instances
[
https://issues.apache.org/jira/browse/IMPALA-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nilangekar resolved IMPALA-7791.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Aggregation Node memory estimates don't account for number of fragment
> instances
>
>
> Key: IMPALA-7791
> URL: https://issues.apache.org/jira/browse/IMPALA-7791
> Project: IMPALA
> Issue Type: Sub-task
>Affects Versions: Impala 3.1.0
>Reporter: Pooja Nilangekar
>Assignee: Pooja Nilangekar
>Priority: Blocker
> Fix For: Impala 3.1.0
>
>
> AggregationNode's memory estimates are calculated based on the input
> cardinality of the node, without accounting for the division of input data
> across fragment instances. This results in very high memory estimates. In
> reality, the nodes often use only a part of this memory.
> Example query:
> {code:java}
> [localhost:21000] default> select distinct * from tpch.lineitem limit 5;
> {code}
> Summary:
> {code:java}
> +--++--+--+---++---+---+---+
> | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem
> | Est. Peak Mem | Detail
>
>
>
>
>|
> +--++--+--+---++---+---+---+
> | 04:EXCHANGE | 1 | 21.24us | 21.24us | 5 | 5 | 48.00 KB
> | 16.00 KB | UNPARTITIONED
>
>
>
>
>|
> | 03:AGGREGATE | 3 | 5.11s| 5.15s| 15| 5 | 576.21
> MB | 1.62 GB | FINALIZE
>
>
>
>
> |
> | 02:EXCHANGE | 3 | 709.75ms | 728.91ms | 6.00M | 6.00M | 5.46 MB
> | 10.78 MB |
> HASH(tpch.lineitem.l_orderkey,tpch.lineitem.l_partkey,tpch.lineitem.l_suppkey,tpch.lineitem.l_linenumber,tpch.lineitem.l_quantity,tpch.lineitem.l_extendedprice,tpch.lineitem.l_discount,tpch.lineitem.l_tax,tpch.lineitem.l_returnflag,tpch.lineitem.l_linestatus,tpch.lineitem.l_shipdate,tpch.lineitem.l_commitdate,tpch.lineitem.l_receiptdate,tpch.lineitem.l_shipinstruct,tpch.lineitem.l_shipmode,tpch.lineitem.l_comment)
> |
> | 01:AGGREGATE | 3 | 4.37s| 4.70s| 6.00M | 6.00M | 36.77 MB
> | 1.62 GB | STREAMING
>
>
>
>
>|
> | 00:SCAN HDFS | 3 | 437.14ms | 480.60ms | 6.00M |
[jira] [Updated] (IMPALA-7804) Various scanner tests intermittently failing on S3 on different runs
[
https://issues.apache.org/jira/browse/IMPALA-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-7804:
--
Target Version: Impala 3.2.0
> Various scanner tests intermittently failing on S3 on different runs
>
>
> Key: IMPALA-7804
> URL: https://issues.apache.org/jira/browse/IMPALA-7804
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Priority: Critical
> Labels: S3
>
> The failures have to do with getting AWS client credentials.
> *query_test/test_scanners.py:696: in test_decimal_encodings*
> _Stacktrace_
> {noformat}
> query_test/test_scanners.py:696: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:438: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:260: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E -255.00,-255.00,-255.00 == -255.00,-255.00,-255.00
> E -255.00,-255.00,-255.00 != -65535.00,-65535.00,-65535.00
> E -65535.00,-65535.00,-65535.00 != -999.99,-999.99,-999.99
> E -65535.00,-65535.00,-65535.00 !=
> 0.00,-.99,-.99
> E -999.99,-999.99,-999.99 != 0.00,0.00,0.00
> E -999.99,-999.99,-999.99 !=
> 0.00,.99,.99
> E 0.00,-.99,-.99 !=
> 255.00,255.00,255.00
> E 0.00,-.99,-.99 !=
> 65535.00,65535.00,65535.00
> E 0.00,0.00,0.00 != 999.99,999.99,999.99
> E 0.00,0.00,0.00 != None
> E 0.00,.99,.99 != None
> E 0.00,.99,.99 != None
> E 255.00,255.00,255.00 != None
> E 255.00,255.00,255.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 999.99,999.99,999.99 != None
> E 999.99,999.99,999.99 != None
> E Number of rows returned (expected vs actual): 18 != 9
> {noformat}
> _Standard Error_
> {noformat}
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_huge_num_rows_76a09ef1` CASCADE;
> -- 2018-11-01 09:42:41,140 INFO MainThread: Started query
> 4c4bc0e7b69d7641:130ffe73
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_huge_num_rows_76a09ef1`;
> -- 2018-11-01 09:42:42,402 INFO MainThread: Started query
> e34d714d6a62cba1:2a8544d0
> -- 2018-11-01 09:42:42,405 INFO MainThread: Created database
> "test_huge_num_rows_76a09ef1" for test ID
> "query_test/test_scanners.py::TestParquet::()::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
> 18/11/01 09:42:43 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for
> impala-test-uswest2-1
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Propagating entries under
> fs.s3a.bucket.impala-test-uswest2-1.
> 18/11/01 09:42:43 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
> 18/11/01 09:42:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 18/11/01 09:42:43 INFO impl.MetricsSystemImpl: s3a-file-system metrics system
> started
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: For URI s3a://impala-test-uswest2-1/,
> using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider
> com.amazonaws.auth.InstanceProfileCredentialsProvider@15bbf42f
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.maximum is
> 1500
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.attempts.maximum is 20
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of
> fs.s3a.connection.establish.timeout is 5000
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.timeout is
> 20
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of
[jira] [Updated] (IMPALA-7804) Various scanner tests intermittently failing on S3 on different runs
[
https://issues.apache.org/jira/browse/IMPALA-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-7804:
--
Affects Version/s: (was: Impala 3.1.0)
Impala 3.2.0
> Various scanner tests intermittently failing on S3 on different runs
>
>
> Key: IMPALA-7804
> URL: https://issues.apache.org/jira/browse/IMPALA-7804
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Priority: Critical
> Labels: S3
>
> The failures have to do with getting AWS client credentials.
> *query_test/test_scanners.py:696: in test_decimal_encodings*
> _Stacktrace_
> {noformat}
> query_test/test_scanners.py:696: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:438: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:260: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E -255.00,-255.00,-255.00 == -255.00,-255.00,-255.00
> E -255.00,-255.00,-255.00 != -65535.00,-65535.00,-65535.00
> E -65535.00,-65535.00,-65535.00 != -999.99,-999.99,-999.99
> E -65535.00,-65535.00,-65535.00 !=
> 0.00,-.99,-.99
> E -999.99,-999.99,-999.99 != 0.00,0.00,0.00
> E -999.99,-999.99,-999.99 !=
> 0.00,.99,.99
> E 0.00,-.99,-.99 !=
> 255.00,255.00,255.00
> E 0.00,-.99,-.99 !=
> 65535.00,65535.00,65535.00
> E 0.00,0.00,0.00 != 999.99,999.99,999.99
> E 0.00,0.00,0.00 != None
> E 0.00,.99,.99 != None
> E 0.00,.99,.99 != None
> E 255.00,255.00,255.00 != None
> E 255.00,255.00,255.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 999.99,999.99,999.99 != None
> E 999.99,999.99,999.99 != None
> E Number of rows returned (expected vs actual): 18 != 9
> {noformat}
> _Standard Error_
> {noformat}
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_huge_num_rows_76a09ef1` CASCADE;
> -- 2018-11-01 09:42:41,140 INFO MainThread: Started query
> 4c4bc0e7b69d7641:130ffe73
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_huge_num_rows_76a09ef1`;
> -- 2018-11-01 09:42:42,402 INFO MainThread: Started query
> e34d714d6a62cba1:2a8544d0
> -- 2018-11-01 09:42:42,405 INFO MainThread: Created database
> "test_huge_num_rows_76a09ef1" for test ID
> "query_test/test_scanners.py::TestParquet::()::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
> 18/11/01 09:42:43 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for
> impala-test-uswest2-1
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Propagating entries under
> fs.s3a.bucket.impala-test-uswest2-1.
> 18/11/01 09:42:43 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
> 18/11/01 09:42:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 18/11/01 09:42:43 INFO impl.MetricsSystemImpl: s3a-file-system metrics system
> started
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: For URI s3a://impala-test-uswest2-1/,
> using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider
> com.amazonaws.auth.InstanceProfileCredentialsProvider@15bbf42f
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.maximum is
> 1500
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.attempts.maximum is 20
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of
> fs.s3a.connection.establish.timeout is 5000
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.timeout is
> 20
> 18/11/01
[jira] [Updated] (IMPALA-7804) Various scanner tests intermittently failing on S3 on different runs
[
https://issues.apache.org/jira/browse/IMPALA-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-7804:
--
Priority: Blocker (was: Critical)
> Various scanner tests intermittently failing on S3 on different runs
>
>
> Key: IMPALA-7804
> URL: https://issues.apache.org/jira/browse/IMPALA-7804
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: David Knupp
>Priority: Blocker
> Labels: S3
>
> The failures have to do with getting AWS client credentials.
> *query_test/test_scanners.py:696: in test_decimal_encodings*
> _Stacktrace_
> {noformat}
> query_test/test_scanners.py:696: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:438: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:260: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E -255.00,-255.00,-255.00 == -255.00,-255.00,-255.00
> E -255.00,-255.00,-255.00 != -65535.00,-65535.00,-65535.00
> E -65535.00,-65535.00,-65535.00 != -999.99,-999.99,-999.99
> E -65535.00,-65535.00,-65535.00 !=
> 0.00,-.99,-.99
> E -999.99,-999.99,-999.99 != 0.00,0.00,0.00
> E -999.99,-999.99,-999.99 !=
> 0.00,.99,.99
> E 0.00,-.99,-.99 !=
> 255.00,255.00,255.00
> E 0.00,-.99,-.99 !=
> 65535.00,65535.00,65535.00
> E 0.00,0.00,0.00 != 999.99,999.99,999.99
> E 0.00,0.00,0.00 != None
> E 0.00,.99,.99 != None
> E 0.00,.99,.99 != None
> E 255.00,255.00,255.00 != None
> E 255.00,255.00,255.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 65535.00,65535.00,65535.00 != None
> E 999.99,999.99,999.99 != None
> E 999.99,999.99,999.99 != None
> E Number of rows returned (expected vs actual): 18 != 9
> {noformat}
> _Standard Error_
> {noformat}
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_huge_num_rows_76a09ef1` CASCADE;
> -- 2018-11-01 09:42:41,140 INFO MainThread: Started query
> 4c4bc0e7b69d7641:130ffe73
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_huge_num_rows_76a09ef1`;
> -- 2018-11-01 09:42:42,402 INFO MainThread: Started query
> e34d714d6a62cba1:2a8544d0
> -- 2018-11-01 09:42:42,405 INFO MainThread: Created database
> "test_huge_num_rows_76a09ef1" for test ID
> "query_test/test_scanners.py::TestParquet::()::test_huge_num_rows[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
> 18/11/01 09:42:43 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for
> impala-test-uswest2-1
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Propagating entries under
> fs.s3a.bucket.impala-test-uswest2-1.
> 18/11/01 09:42:43 WARN impl.MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
> 18/11/01 09:42:43 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
> period at 10 second(s).
> 18/11/01 09:42:43 INFO impl.MetricsSystemImpl: s3a-file-system metrics system
> started
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: For URI s3a://impala-test-uswest2-1/,
> using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider
> com.amazonaws.auth.InstanceProfileCredentialsProvider@15bbf42f
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.maximum is
> 1500
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.attempts.maximum is 20
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of
> fs.s3a.connection.establish.timeout is 5000
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.timeout is
> 20
> 18/11/01 09:42:43 DEBUG s3a.S3AUtils: Value of
[jira] [Commented] (IMPALA-7754) Expressions sometimes not re-analyzed after rewrite
[
https://issues.apache.org/jira/browse/IMPALA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680326#comment-16680326
]
Paul Rogers commented on IMPALA-7754:
-
Another misfire, in {{PlannerTest}}, {{hdfs.test}}:
{noformat}
# IMPALA-5180: Test that predicates not touching a partition column are ignored
in
# partition pruning
select count(*) from
(select * from functional.alltypes) T
where random() = 100
...
00:SCAN HDFS [functional.alltypes]
partitions=24/24 files=24 size=478.45KB
predicates: functional.alltypes.int_col = 1, FALSE OR 1 + random() * 1 = 100
{noformat}
Should be:
{noformat}
00:SCAN HDFS [functional.alltypes]
partitions=24/24 files=24 size=478.45KB
predicates: functional.alltypes.int_col = 1, (1 + random() * 1 = 100)
{noformat}
> Expressions sometimes not re-analyzed after rewrite
> ---
>
> Key: IMPALA-7754
> URL: https://issues.apache.org/jira/browse/IMPALA-7754
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Priority: Major
>
> The analyzer has a chain of rules which fire in order without (as noted
> above) repeats. The result of rule A (rewriting conditional functions) is fed
> into rule B (simplify CASE). Each rule requires that analysis be done so that
> attributes of expressions can be picked out.
> As it turns out, in the current code, this is rather ad-hoc. The
> {{SimplifyConditionalsRule}} re-analyzes its result as part of the fix for
> IMPALA-5125, but others do not, leading to optimizations not working. In
> particular, in a chain of rewrites for {{IS DISTINCT FROM}}, certain rules
> didn't fire because previous rules left new expressions in an un-analyzed
> state. This is a bug.
> The fix is to analyze the result any time a rule fires, before passing the
> result to the next rule.
> {code:java}
> private Expr applyRuleBottomUp(Expr expr, ExprRewriteRule rule, Analyzer
> analyzer)
> throws AnalysisException {
> ...
> Expr rewrittenExpr = rule.apply(expr, analyzer);
> if (rewrittenExpr != expr) {
> ++numChanges_;
> rewrittenExpr.analyze(analyzer); // Add me!
> }}
> return rewrittenExpr;
> }
> {code}
> There are several places that the above logic appears: make the change in all
> of them.
> Then, in rules that simply refused to run if an expression is to analyzed:
> {code:java}
> public class SimplifyDistinctFromRule implements ExprRewriteRule {
> public Expr apply(Expr expr, Analyzer analyzer) {
> if (!expr.isAnalyzed()) return expr;
> {code}
> Replace this with an assertion that analysis must have been done:
> {code:java}
> public class SimplifyDistinctFromRule implements ExprRewriteRule {
> public Expr apply(Expr expr, Analyzer analyzer) {
> assert expr.isAnalyzed();
> {code}
> To be safe, the assertion fires only in "debug" mode, not in production.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7838) Planner's handling of parenthesized expressions is awkward
[
https://issues.apache.org/jira/browse/IMPALA-7838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated IMPALA-7838:
Description:
Consider two simple queries:
{code:sql}
SELECT 2 + 3 * 4 FROM ...
SELECT (2 + 3) * 4 FROM ...
{code}
The parenthesis are required in infix notation to override default precedence
rules. When parsed, the difference shows up as different tree structures:
{noformat}
(+ 2 (* 3 4))
(* (+ 2 3) 4)
{noformat}
Impala's expression nodes often wish to render the parse nodes back to SQL. A
simple traversal will produce {{2 + 3 * 4}} in both cases, which is wrong. A
fully parenthesized version would be correct, but a nuisance:
{noformat}
(2 + (3 * 4))
((2 + 3) * 4)
{noformat}
To recreate the user's original parenthisization, Impala tags each expression
node with whether it was wrapped in parenthesis when parsed.
All of this works as long as we leave the parse tree unchanged. But, if we do
rewrites, we end up in a very confused state. For example, after constant
folding should the above be enclosed in parenthesis or not?
Take a more complex case:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE (foo.a > 10) AND (bar.b = 20)
{code}
Do we need to preserve the parenthesis in the emitted plan? In the above, no,
we don't: they don't add information. Yet, the planner tries to preserve them.
{noformat}
predicates: (foo.a > 10)
{noformat}
By contrast:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE foo.a > 10 AND bar.b = 20
{code}
Produces:
{noformat}
predicates: foo.a > 10
{noformat}
Yet, functionally, the two are identical: the plan differences are unnecessary
and are just noise.
Again, if a rewrite occurs, it is not clear whether parenthesis should or
should not be preserved.
Today, a rewrite discards parenthesis (the rewritten node never has the
{{printSqlInParenthesis}} flag set.) Yet, that could, conceivably, change the
meaning of an expression when printed:
{noformat}
a AND (b OR c OR FALSE) --> a AND b OR c -- what the user sees
(a AND (b OR c OR FALSE)) --> (a AND (b OR c)) -- parse nodes
(a AND (b OR c OR FALSE)) --> ((a AND b) OR c) -- user's parse
{noformat}
The first is what the user would see, the second is how the parse tree
represents the rewritten statement, the third is how the user would interpret
the {{toSql()}} form with missing parenthesis.
The problem is, we are approaching the problem incorrectly. Since we perform
rewrites, we should use parenthesis only when to override precedence:
{noformat}
foo.a + foo.b * 3
(foo.a + foo.b) * 3
{noformat}
That is, parenthesis should be generated based on precedence rules, *not* based
on the original SQL source text. That way, parenthesis will be both consistent
and accurate in the emitted, rewritten expressions.
was:
Consider two simple queries:
{code:sql}
SELECT 2 + 3 * 4 FROM ...
SELECT (2 + 3) * 4 FROM ...
{code}
The parenthesis are required in infix notation to override default precedence
rules. When parsed, the difference shows up as different tree structures:
{noformat}
(+ 2 (* 3 4))
(* (+ 2 3) 4)
{noformat}
Impala's expression nodes often wish to render the parse nodes back to SQL. A
simple traversal will produce {{2 + 3 * 4}} in both cases, which is wrong. A
fully parenthesized version would be correct, but a nuisance:
{noformat}
(2 + (3 * 4))
((2 + 3) * 4)
{noformat}
To recreate the user's original parenthisization, Impala tags each expression
node with whether it was wrapped in parenthesis when parsed.
All of this works as long as we leave the parse tree unchanged. But, if we do
rewrites, we end up in a very confused state. For example, after constant
folding should the above be enclosed in parenthesis or not?
Take a more complex case:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE (foo.a > 10) AND (bar.b = 20)
{code}
Do we need to preserve the parenthesis in the emitted plan? In the above, no,
we don't: they don't add information. Yet, the planner tries to preserve them.
{noformat}
predicates: (foo.a > 10)
{noformat}
By contrast:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE foo.a > 10 AND bar.b = 20
{code}
Produces:
{noformat}
predicates: foo.a > 10
{noformat}
Yet, functionally, the two are identical: the plan differences are unnecessary
and are just noise.
Again, if a rewrite occurs, it is not clear whether parenthesis should or
should not be preserved.
The problem is, we are approaching the problem incorrectly. Since we perform
rewrites, we should use parenthesis only when to override precedence:
{noformat}
foo.a + foo.b * 3
(foo.a + foo.b) * 3
{noformat}
That is, parenthesis should be generated based on precedence rules, *not* based
on the original SQL source text. That way, parenthesis will be both consistent
and accurate in the emitted, rewritten expressions.
> Planner's handling of parenthesized expressions is awkward
>
[jira] [Commented] (IMPALA-7754) Expressions sometimes not re-analyzed after rewrite
[
https://issues.apache.org/jira/browse/IMPALA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680342#comment-16680342
]
Paul Rogers commented on IMPALA-7754:
-
Another misfire, in {{PlannerTest}}, {{constant-propagation.test}}:
{noformat}
# Non-trivial expr substitution (non-constant)
select count(*) from
(select * from functional.alltypes where int_col = 10) T
where T.int_col = 10 and T.int_col > 1 and T.int_col = 10 and T.int_col =
T.int_col and
(coalesce(NULL, T.int_col) + random() * T.tinyint_col = 100 OR
coalesce(NULL, T.int_col) + T.int_col = 20)
PLAN
PLAN-ROOT SINK
|
01:AGGREGATE [FINALIZE]
| output: count(*)
|
00:SCAN HDFS [functional.alltypes]
partitions=24/24 files=24 size=478.45KB
predicates: int_col = 10, TRUE OR 10 + random() *
functional.alltypes.tinyint_col = 100
{noformat}
Should be:
{noformat}
00:SCAN HDFS [functional.alltypes]
partitions=24/24 files=24 size=478.45KB
predicates: int_col = 10
{noformat}
> Expressions sometimes not re-analyzed after rewrite
> ---
>
> Key: IMPALA-7754
> URL: https://issues.apache.org/jira/browse/IMPALA-7754
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Priority: Major
>
> The analyzer has a chain of rules which fire in order without (as noted
> above) repeats. The result of rule A (rewriting conditional functions) is fed
> into rule B (simplify CASE). Each rule requires that analysis be done so that
> attributes of expressions can be picked out.
> As it turns out, in the current code, this is rather ad-hoc. The
> {{SimplifyConditionalsRule}} re-analyzes its result as part of the fix for
> IMPALA-5125, but others do not, leading to optimizations not working. In
> particular, in a chain of rewrites for {{IS DISTINCT FROM}}, certain rules
> didn't fire because previous rules left new expressions in an un-analyzed
> state. This is a bug.
> The fix is to analyze the result any time a rule fires, before passing the
> result to the next rule.
> {code:java}
> private Expr applyRuleBottomUp(Expr expr, ExprRewriteRule rule, Analyzer
> analyzer)
> throws AnalysisException {
> ...
> Expr rewrittenExpr = rule.apply(expr, analyzer);
> if (rewrittenExpr != expr) {
> ++numChanges_;
> rewrittenExpr.analyze(analyzer); // Add me!
> }}
> return rewrittenExpr;
> }
> {code}
> There are several places that the above logic appears: make the change in all
> of them.
> Then, in rules that simply refused to run if an expression is to analyzed:
> {code:java}
> public class SimplifyDistinctFromRule implements ExprRewriteRule {
> public Expr apply(Expr expr, Analyzer analyzer) {
> if (!expr.isAnalyzed()) return expr;
> {code}
> Replace this with an assertion that analysis must have been done:
> {code:java}
> public class SimplifyDistinctFromRule implements ExprRewriteRule {
> public Expr apply(Expr expr, Analyzer analyzer) {
> assert expr.isAnalyzed();
> {code}
> To be safe, the assertion fires only in "debug" mode, not in production.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7838) Planner's handling of parenthesized expressions is awkward
Paul Rogers created IMPALA-7838:
---
Summary: Planner's handling of parenthesized expressions is awkward
Key: IMPALA-7838
URL: https://issues.apache.org/jira/browse/IMPALA-7838
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Consider two simple queries:
{code:sql}
SELECT 2 + 3 * 4 FROM ...
SELECT (2 + 3) * 4 FROM ...
{code}
The parenthesis are required in infix notation to override default precedence
rules. When parsed, the difference shows up as different tree structures:
{noformat}
(+ 2 (* 3 4))
(* (+ 2 3) 4)
{noformat}
Impala's expression nodes often wish to render the parse nodes back to SQL. A
simple traversal will produce {{2 + 3 * 4}} in both cases, which is wrong. A
fully parenthesized version would be correct, but a nuisance:
{noformat}
(2 + (3 * 4))
((2 + 3) * 4)
{noformat}
To recreate the user's original parenthisization, Impala tags each expression
node with whether it was wrapped in parenthesis when parsed.
All of this works as long as we leave the parse tree unchanged. But, if we do
rewrites, we end up in a very confused state. For example, after constant
folding should the above be enclosed in parenthesis or not?
Take a more complex case:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE (foo.a > 10) AND (bar.b = 20)
{code}
Do we need to preserve the parenthesis in the emitted plan? In the above, no,
we don't: they don't add information. Yet, the planner tries to preserve them.
{noformat}
predicates: (foo.a > 10)
{noformat}
By contrast:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE foo.a > 10 AND bar.b = 20
{code}
Produces:
{noformat}
predicates: foo.a > 10
{noformat}
Yet, functionally, the two are identical: the plan differences are unnecessary
and are just noise.
Again, if a rewrite occurs, it is not clear whether parenthesis should or
should not be preserved.
The problem is, we are approaching the problem incorrectly. Since we perform
rewrites, we should use parenthesis only when to override precedence:
{noformat}
foo.a + foo.b * 3
(foo.a + foo.b) * 3
{noformat}
That is, parenthesis should be generated based on precedence rules, *not* based
on the original SQL source text. That way, parenthesis will be both consistent
and accurate in the emitted, rewritten expressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7838) Planner's handling of parenthesized expressions is awkward
Paul Rogers created IMPALA-7838:
---
Summary: Planner's handling of parenthesized expressions is awkward
Key: IMPALA-7838
URL: https://issues.apache.org/jira/browse/IMPALA-7838
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.0
Reporter: Paul Rogers
Consider two simple queries:
{code:sql}
SELECT 2 + 3 * 4 FROM ...
SELECT (2 + 3) * 4 FROM ...
{code}
The parenthesis are required in infix notation to override default precedence
rules. When parsed, the difference shows up as different tree structures:
{noformat}
(+ 2 (* 3 4))
(* (+ 2 3) 4)
{noformat}
Impala's expression nodes often wish to render the parse nodes back to SQL. A
simple traversal will produce {{2 + 3 * 4}} in both cases, which is wrong. A
fully parenthesized version would be correct, but a nuisance:
{noformat}
(2 + (3 * 4))
((2 + 3) * 4)
{noformat}
To recreate the user's original parenthisization, Impala tags each expression
node with whether it was wrapped in parenthesis when parsed.
All of this works as long as we leave the parse tree unchanged. But, if we do
rewrites, we end up in a very confused state. For example, after constant
folding should the above be enclosed in parenthesis or not?
Take a more complex case:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE (foo.a > 10) AND (bar.b = 20)
{code}
Do we need to preserve the parenthesis in the emitted plan? In the above, no,
we don't: they don't add information. Yet, the planner tries to preserve them.
{noformat}
predicates: (foo.a > 10)
{noformat}
By contrast:
{code:sql}
SELECT * FROM foo, bar ON ... WHERE foo.a > 10 AND bar.b = 20
{code}
Produces:
{noformat}
predicates: foo.a > 10
{noformat}
Yet, functionally, the two are identical: the plan differences are unnecessary
and are just noise.
Again, if a rewrite occurs, it is not clear whether parenthesis should or
should not be preserved.
The problem is, we are approaching the problem incorrectly. Since we perform
rewrites, we should use parenthesis only when to override precedence:
{noformat}
foo.a + foo.b * 3
(foo.a + foo.b) * 3
{noformat}
That is, parenthesis should be generated based on precedence rules, *not* based
on the original SQL source text. That way, parenthesis will be both consistent
and accurate in the emitted, rewritten expressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7837) SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
[
https://issues.apache.org/jira/browse/IMPALA-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-7837:
--
Target Version: Impala 3.2.0 (was: Impala 3.1.0)
> SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
>
>
> Key: IMPALA-7837
> URL: https://issues.apache.org/jira/browse/IMPALA-7837
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Michael Brown
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Attachments: impala-logs.tar.gz
>
>
> This test is not raising the expected exception on a *release build*:
> {noformat}
> QUERY
> # Query should fail due to exceeding scan bytes limit.
> set SCAN_BYTES_LIMIT="100M";
> select count(*) from tpch.lineitem l1,tpch.lineitem l2, tpch.lineitem l3 where
> l1.l_suppkey = l2.l_linenumber and l1.l_orderkey = l2.l_orderkey and
> l1.l_orderkey = l3.l_orderkey group by l1.l_comment, l2.l_comment
> having count(*) = 99
> CATCH
> row_regex:.*terminated due to scan bytes limit of 100.00 M.
> {noformat}
> {noformat}
> Stacktrace
> query_test/test_resource_limits.py:46: in test_resource_limits
> self.run_test_case('QueryTest/query-resource-limits', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex:.*terminated due to scan
> bytes limit of 100.00 M.*
> {noformat}
> It fails deterministically in CI (3 times in a row). I can't find a query
> profile matching the query ID for some reason, but I've attached logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7837) SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
[
https://issues.apache.org/jira/browse/IMPALA-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Brown updated IMPALA-7837:
--
Affects Version/s: (was: Impala 3.1.0)
Impala 3.2.0
> SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
>
>
> Key: IMPALA-7837
> URL: https://issues.apache.org/jira/browse/IMPALA-7837
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Michael Brown
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Attachments: impala-logs.tar.gz
>
>
> This test is not raising the expected exception on a *release build*:
> {noformat}
> QUERY
> # Query should fail due to exceeding scan bytes limit.
> set SCAN_BYTES_LIMIT="100M";
> select count(*) from tpch.lineitem l1,tpch.lineitem l2, tpch.lineitem l3 where
> l1.l_suppkey = l2.l_linenumber and l1.l_orderkey = l2.l_orderkey and
> l1.l_orderkey = l3.l_orderkey group by l1.l_comment, l2.l_comment
> having count(*) = 99
> CATCH
> row_regex:.*terminated due to scan bytes limit of 100.00 M.
> {noformat}
> {noformat}
> Stacktrace
> query_test/test_resource_limits.py:46: in test_resource_limits
> self.run_test_case('QueryTest/query-resource-limits', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex:.*terminated due to scan
> bytes limit of 100.00 M.*
> {noformat}
> It fails deterministically in CI (3 times in a row). I can't find a query
> profile matching the query ID for some reason, but I've attached logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7148) test_profile_fragment_instances is flaky
[ https://issues.apache.org/jira/browse/IMPALA-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680127#comment-16680127 ] ASF subversion and git services commented on IMPALA-7148: - Commit 7299a6aedbbb7563e465dcfc6a20a981e9f1b804 in impala's branch refs/heads/master from Michael Ho [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=7299a6a ] IMPALA-7148: Make test_profile_fragment_instances() more robust test_profile_fragment_instances() makes assumption about number of instances of each scan node in the distributed query plan. The number of instances for each scan nodes can actually vary depending on how data is loaded and scheduler's decision. This change relaxes the check for number of instances of each scan node is a multiple of 3 which is the number of scan nodes in the plan. Change-Id: I08b068c21e9637575c85f4d54be9f7c56c106bf1 Reviewed-on: http://gerrit.cloudera.org:8080/11906 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > test_profile_fragment_instances is flaky > > > Key: IMPALA-7148 > URL: https://issues.apache.org/jira/browse/IMPALA-7148 > Project: IMPALA > Issue Type: Test >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Critical > > The underlying bug in IMPALA-6338 was causing a large number of flaky tests > and was tricky to fix given our present coordinator runtime profile handling. > We decided to temporarily disable these tests pending work on the coordinator > (MPALA-2990/IMPALA-4063). Once that work is done, we should re-enable these > tests. > The tests have been re-enabled since IMPALA-4063 has been fixed. However, > there seems to be some flakiness with some tests not related to IMPALA-4063 > or IMPALA-6338. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7208) Consider using quickstack instead of pstack
[ https://issues.apache.org/jira/browse/IMPALA-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuj Phadke resolved IMPALA-7208. - Resolution: Won't Fix I am resolving this since this did not work on certain versions of Centos for me. > Consider using quickstack instead of pstack > --- > > Key: IMPALA-7208 > URL: https://issues.apache.org/jira/browse/IMPALA-7208 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Priority: Major > Labels: supportability > Attachments: quickstack.out > > > This is an alternative to heavyweight methods like pstack and gdb that can > block the process while they're collecting stacks: > https://github.com/yoshinorim/quickstack . Yoshinori has a lot of experience > troubleshooting production systems so his recommendation carries some weight > for me. > I tried it out and it seems to work as advertised so far. The binary is > pretty small and easy to build so isn't an unreasonable dependency. > [~bharathv][~dgarg][~anujphadke] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7148) test_profile_fragment_instances is flaky
[ https://issues.apache.org/jira/browse/IMPALA-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho resolved IMPALA-7148. Resolution: Fixed Fix Version/s: Impala 3.2.0 All tests have been re-enabled and the flaky test_profile_fragment_instances() has been addressed. > test_profile_fragment_instances is flaky > > > Key: IMPALA-7148 > URL: https://issues.apache.org/jira/browse/IMPALA-7148 > Project: IMPALA > Issue Type: Test >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Critical > Fix For: Impala 3.2.0 > > > The underlying bug in IMPALA-6338 was causing a large number of flaky tests > and was tricky to fix given our present coordinator runtime profile handling. > We decided to temporarily disable these tests pending work on the coordinator > (MPALA-2990/IMPALA-4063). Once that work is done, we should re-enable these > tests. > The tests have been re-enabled since IMPALA-4063 has been fixed. However, > there seems to be some flakiness with some tests not related to IMPALA-4063 > or IMPALA-6338. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
[ https://issues.apache.org/jira/browse/IMPALA-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7836: Component/s: Docs > Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort > conversion > --- > > Key: IMPALA-7836 > URL: https://issues.apache.org/jira/browse/IMPALA-7836 > Project: IMPALA > Issue Type: Sub-task > Components: Docs, Frontend >Affects Versions: Impala 2.9.0 >Reporter: Sahil Takiar >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we > should document. The changes in IMPALA-5004 work by estimating the amount of > memory required to run a TopN operator. The memory estimate is based on the > size of the individual tuples that need to be processed by the TopN operator, > as well as the sum of the limit and offset in the query. TopN operators don't > spill to disk so they have to keep all rows they process in memory. > If the estimated size of the working set of the TopN operator exceeds the > threshold of 'topn_bytes_limit' the TopN operator will be replaced with a > Sort operator. The Sort operator can spill to disk, but it processes all the > data (the limit and offset have no affect). So switching to Sort might incur > performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7208) Consider using quickstack instead of pstack
[ https://issues.apache.org/jira/browse/IMPALA-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuj Phadke resolved IMPALA-7208. - Resolution: Won't Fix I am resolving this since this did not work on certain versions of Centos for me. > Consider using quickstack instead of pstack > --- > > Key: IMPALA-7208 > URL: https://issues.apache.org/jira/browse/IMPALA-7208 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Tim Armstrong >Priority: Major > Labels: supportability > Attachments: quickstack.out > > > This is an alternative to heavyweight methods like pstack and gdb that can > block the process while they're collecting stacks: > https://github.com/yoshinorim/quickstack . Yoshinori has a lot of experience > troubleshooting production systems so his recommendation carries some weight > for me. > I tried it out and it seems to work as advertised so far. The binary is > pretty small and easy to build so isn't an unreasonable dependency. > [~bharathv][~dgarg][~anujphadke] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
[ https://issues.apache.org/jira/browse/IMPALA-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7836: Target Version: Impala 3.1.0 > Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort > conversion > --- > > Key: IMPALA-7836 > URL: https://issues.apache.org/jira/browse/IMPALA-7836 > Project: IMPALA > Issue Type: Sub-task > Components: Docs, Frontend >Affects Versions: Impala 2.9.0 >Reporter: Sahil Takiar >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we > should document. The changes in IMPALA-5004 work by estimating the amount of > memory required to run a TopN operator. The memory estimate is based on the > size of the individual tuples that need to be processed by the TopN operator, > as well as the sum of the limit and offset in the query. TopN operators don't > spill to disk so they have to keep all rows they process in memory. > If the estimated size of the working set of the TopN operator exceeds the > threshold of 'topn_bytes_limit' the TopN operator will be replaced with a > Sort operator. The Sort operator can spill to disk, but it processes all the > data (the limit and offset have no affect). So switching to Sort might incur > performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
[ https://issues.apache.org/jira/browse/IMPALA-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7836: Labels: future_release_doc (was: ) > Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort > conversion > --- > > Key: IMPALA-7836 > URL: https://issues.apache.org/jira/browse/IMPALA-7836 > Project: IMPALA > Issue Type: Sub-task > Components: Docs, Frontend >Affects Versions: Impala 2.9.0 >Reporter: Sahil Takiar >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we > should document. The changes in IMPALA-5004 work by estimating the amount of > memory required to run a TopN operator. The memory estimate is based on the > size of the individual tuples that need to be processed by the TopN operator, > as well as the sum of the limit and offset in the query. TopN operators don't > spill to disk so they have to keep all rows they process in memory. > If the estimated size of the working set of the TopN operator exceeds the > threshold of 'topn_bytes_limit' the TopN operator will be replaced with a > Sort operator. The Sort operator can spill to disk, but it processes all the > data (the limit and offset have no affect). So switching to Sort might incur > performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680135#comment-16680135 ] Tim Armstrong commented on IMPALA-7834: --- If you have complete stack traces for all threads that is often useful. > Impala different types of crashes > - > > Key: IMPALA-7834 > URL: https://issues.apache.org/jira/browse/IMPALA-7834 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Manikandan R >Priority: Major > Labels: CDH5, crash > > Off late, We had witnessed different types of crashes in cluster. I don't see > any similarities among crashes stack traces and also not able to reproduce. > Below are the stack traces occurred in different daemons at different > timings. I do have complete stack traces and let me know if it helps for > further debugging. > 1) > 10-1-33-172 > Nov 6 18:04 > Thread 1 (Thread 0x7f018b423700 (LWP 96325)): > #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7f0aad280185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7f0aad422593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7f0aad28568f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7f0a44ca3000 in ?? () > No symbol table info available. > #8 0x00dba5c1 in > impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () > No symbol table info available. > #9 0x00dba924 in > impala::HdfsParquetScanner::AssembleRows(std::vector std::allocator > const&, impala::RowBatch*, > bool*) () > No symbol table info available. > #10 0x00dbf5f6 in > impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () > No symbol table info available. > #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () > No symbol table info available. > #12 0x00d835e6 in > impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*, > impala::io::ScanRange*) () > No symbol table info available. > #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () > No symbol table info available. > #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function, impala::Promise*) () > No symbol table info available. > #15 0x00d173c4 in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function, > impala::Promise*), boost::_bi::list4, > boost::_bi::value, boost::_bi::value >, > boost::_bi::value*> > > >::run() () > No symbol table info available. > #16 0x0128fada in thread_proxy () > No symbol table info available. > #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 > No symbol table info available. > Nov 1 11:21 > Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): > #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7fc297d12185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fc297eb4593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7fc297d1768f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7fc1dd9f in ?? () > No symbol table info available. > #8 0x00fdd9f9 in > impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () > No symbol table info available. > #9 0x00b74d6d in impala::FragmentInstanceState::Open() () > No symbol table info available. > #10 0x00b763ab in impala::FragmentInstanceState::Exec() () > No symbol table info available. > #11 0x00b65b38 in > impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () > No
[jira] [Commented] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680137#comment-16680137 ] Tim Armstrong commented on IMPALA-7834: --- What was the exact version, also? CDH releases often include some bugfixes not in the base Apache Impala release so that can help to rule things out. > Impala different types of crashes > - > > Key: IMPALA-7834 > URL: https://issues.apache.org/jira/browse/IMPALA-7834 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Manikandan R >Priority: Major > Labels: CDH5, crash > > Off late, We had witnessed different types of crashes in cluster. I don't see > any similarities among crashes stack traces and also not able to reproduce. > Below are the stack traces occurred in different daemons at different > timings. I do have complete stack traces and let me know if it helps for > further debugging. > 1) > 10-1-33-172 > Nov 6 18:04 > Thread 1 (Thread 0x7f018b423700 (LWP 96325)): > #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7f0aad280185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7f0aad422593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7f0aad28568f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7f0a44ca3000 in ?? () > No symbol table info available. > #8 0x00dba5c1 in > impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () > No symbol table info available. > #9 0x00dba924 in > impala::HdfsParquetScanner::AssembleRows(std::vector std::allocator > const&, impala::RowBatch*, > bool*) () > No symbol table info available. > #10 0x00dbf5f6 in > impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () > No symbol table info available. > #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () > No symbol table info available. > #12 0x00d835e6 in > impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*, > impala::io::ScanRange*) () > No symbol table info available. > #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () > No symbol table info available. > #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function, impala::Promise*) () > No symbol table info available. > #15 0x00d173c4 in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function, > impala::Promise*), boost::_bi::list4, > boost::_bi::value, boost::_bi::value >, > boost::_bi::value*> > > >::run() () > No symbol table info available. > #16 0x0128fada in thread_proxy () > No symbol table info available. > #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 > No symbol table info available. > Nov 1 11:21 > Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): > #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7fc297d12185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fc297eb4593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7fc297d1768f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7fc1dd9f in ?? () > No symbol table info available. > #8 0x00fdd9f9 in > impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () > No symbol table info available. > #9 0x00b74d6d in impala::FragmentInstanceState::Open() () > No symbol table info available. > #10 0x00b763ab in impala::FragmentInstanceState::Exec() () > No symbol table info available. > #11 0x00b65b38
[jira] [Updated] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-7834: -- Labels: CDH5 crash (was: CDH5) > Impala different types of crashes > - > > Key: IMPALA-7834 > URL: https://issues.apache.org/jira/browse/IMPALA-7834 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Manikandan R >Priority: Major > Labels: CDH5, crash > > Off late, We had witnessed different types of crashes in cluster. I don't see > any similarities among crashes stack traces and also not able to reproduce. > Below are the stack traces occurred in different daemons at different > timings. I do have complete stack traces and let me know if it helps for > further debugging. > 1) > 10-1-33-172 > Nov 6 18:04 > Thread 1 (Thread 0x7f018b423700 (LWP 96325)): > #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7f0aad280185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7f0aad422593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7f0aad28568f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7f0a44ca3000 in ?? () > No symbol table info available. > #8 0x00dba5c1 in > impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () > No symbol table info available. > #9 0x00dba924 in > impala::HdfsParquetScanner::AssembleRows(std::vector std::allocator > const&, impala::RowBatch*, > bool*) () > No symbol table info available. > #10 0x00dbf5f6 in > impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () > No symbol table info available. > #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () > No symbol table info available. > #12 0x00d835e6 in > impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*, > impala::io::ScanRange*) () > No symbol table info available. > #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () > No symbol table info available. > #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function, impala::Promise*) () > No symbol table info available. > #15 0x00d173c4 in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function, > impala::Promise*), boost::_bi::list4, > boost::_bi::value, boost::_bi::value >, > boost::_bi::value*> > > >::run() () > No symbol table info available. > #16 0x0128fada in thread_proxy () > No symbol table info available. > #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 > No symbol table info available. > Nov 1 11:21 > Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): > #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7fc297d12185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fc297eb4593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7fc297d1768f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7fc1dd9f in ?? () > No symbol table info available. > #8 0x00fdd9f9 in > impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () > No symbol table info available. > #9 0x00b74d6d in impala::FragmentInstanceState::Open() () > No symbol table info available. > #10 0x00b763ab in impala::FragmentInstanceState::Exec() () > No symbol table info available. > #11 0x00b65b38 in > impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () > No symbol table info available. > #12 0x00d16c83 in
[jira] [Resolved] (IMPALA-7148) test_profile_fragment_instances is flaky
[ https://issues.apache.org/jira/browse/IMPALA-7148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho resolved IMPALA-7148. Resolution: Fixed Fix Version/s: Impala 3.2.0 All tests have been re-enabled and the flaky test_profile_fragment_instances() has been addressed. > test_profile_fragment_instances is flaky > > > Key: IMPALA-7148 > URL: https://issues.apache.org/jira/browse/IMPALA-7148 > Project: IMPALA > Issue Type: Test >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Critical > Fix For: Impala 3.2.0 > > > The underlying bug in IMPALA-6338 was causing a large number of flaky tests > and was tricky to fix given our present coordinator runtime profile handling. > We decided to temporarily disable these tests pending work on the coordinator > (MPALA-2990/IMPALA-4063). Once that work is done, we should re-enable these > tests. > The tests have been re-enabled since IMPALA-4063 has been fixed. However, > there seems to be some flakiness with some tests not related to IMPALA-4063 > or IMPALA-6338. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-7828) test_mem_leak() is flaky
[
https://issues.apache.org/jira/browse/IMPALA-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-7828.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
Workaround has been merged.
> test_mem_leak() is flaky
>
>
> Key: IMPALA-7828
> URL: https://issues.apache.org/jira/browse/IMPALA-7828
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.2.0
>
>
> Before IMPALA-4063, the error message detected during
> {{FragmentInstanceState::Close()}} was always lost. After IMPALA-4063, we may
> sometimes get the error message in {{FragmentInstanceState::Close()}}. It's
> non-deterministic as the fragment instance thread may race with the query
> state thread which reports the final status. The test currently tries to
> handle this non-determinism by using "row_regex:.*" in the {{--ERRORS--}}
> section but it doesn't seem to work.
> Let's unbreak the test for now. Longer run, we need to move the point in
> which all fragment instances are done after
> {{FragmentInstanceState::Close()}} so things will become deterministic. This
> may have implication to query completion time so performance may need to be
> evaluated.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:313: in test_ir_functions
> self.run_test_case('QueryTest/udf-no-expr-rewrite', vector,
> use_db=unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:362: in verify_raw_results
> verify_errors(expected_errors, actual_errors)
> common/test_result_verifier.py:314: in verify_errors
> VERIFIER_MAP['VERIFY_IS_EQUAL'](expected, actual)
> common/test_result_verifier.py:271: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E row_regex:.* != None
> E Number of rows returned (expected vs actual): 1 != 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7828) test_mem_leak() is flaky
[
https://issues.apache.org/jira/browse/IMPALA-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-7828.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
Workaround has been merged.
> test_mem_leak() is flaky
>
>
> Key: IMPALA-7828
> URL: https://issues.apache.org/jira/browse/IMPALA-7828
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.2.0
>
>
> Before IMPALA-4063, the error message detected during
> {{FragmentInstanceState::Close()}} was always lost. After IMPALA-4063, we may
> sometimes get the error message in {{FragmentInstanceState::Close()}}. It's
> non-deterministic as the fragment instance thread may race with the query
> state thread which reports the final status. The test currently tries to
> handle this non-determinism by using "row_regex:.*" in the {{--ERRORS--}}
> section but it doesn't seem to work.
> Let's unbreak the test for now. Longer run, we need to move the point in
> which all fragment instances are done after
> {{FragmentInstanceState::Close()}} so things will become deterministic. This
> may have implication to query completion time so performance may need to be
> evaluated.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:313: in test_ir_functions
> self.run_test_case('QueryTest/udf-no-expr-rewrite', vector,
> use_db=unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:362: in verify_raw_results
> verify_errors(expected_errors, actual_errors)
> common/test_result_verifier.py:314: in verify_errors
> VERIFIER_MAP['VERIFY_IS_EQUAL'](expected, actual)
> common/test_result_verifier.py:271: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E row_regex:.* != None
> E Number of rows returned (expected vs actual): 1 != 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
Fredy Wijaya created IMPALA-7835:
Summary: Creating a role with the same name as the user name with
object ownership enabled can cause INVALIDATE METADATA to hang
Key: IMPALA-7835
URL: https://issues.apache.org/jira/browse/IMPALA-7835
Project: IMPALA
Issue Type: Bug
Components: Catalog
Affects Versions: Impala 3.1.0
Reporter: Fredy Wijaya
Assignee: Fredy Wijaya
Start Impala with object ownership enabled.
{noformat}
[localhost:21000] default> create role impdev;
[localhost:21000] default> grant all on server to role impdev;
[localhost:21000] default> grant role impdev to group impdev;
[localhost:21000] default> invalidate metadata; -- this will hang
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-5004) Switch to sorting node for large TopN queries
[ https://issues.apache.org/jira/browse/IMPALA-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved IMPALA-5004. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Switch to sorting node for large TopN queries > - > > Key: IMPALA-5004 > URL: https://issues.apache.org/jira/browse/IMPALA-5004 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.9.0 >Reporter: Lars Volker >Assignee: Sahil Takiar >Priority: Major > Fix For: Impala 3.1.0 > > > As explained by [~tarmstrong] in IMPALA-4995: > bq. We should also consider switching to the sort operator for large limits. > This allows it to spill. The memory requirements for TopN also are > problematic for large limits, since it would allocate large vectors that are > untracked and also require a large amount of contiguous memory. > There's already logic to select TopN vs. Sort: > [planner/SingleNodePlanner.java#L289|https://github.com/apache/incubator-impala/blob/master/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L289] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (IMPALA-6924) Compute stats profiles should include reference to child queries
[ https://issues.apache.org/jira/browse/IMPALA-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6924: -- Labels: observability supportability (was: observability) > Compute stats profiles should include reference to child queries > > > Key: IMPALA-6924 > URL: https://issues.apache.org/jira/browse/IMPALA-6924 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Tim Armstrong >Priority: Major > Labels: observability, supportability > > "Compute stats" queries spawn off child queries that do most of the work. > It's non-trivial to track down the child queries and get their profiles if > something goes wrong. We really should have, at a minimum, the query IDs of > the child queries. in the profile. Maybe we could do better than that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6293) Shell commands run by Impala can fail when using the Java debugger
[ https://issues.apache.org/jira/browse/IMPALA-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680024#comment-16680024 ] Philip Zeyliger commented on IMPALA-6293: - Be aware that the way we initialize our JVM is via HDFS's libhdfs code, and JAVA_TOOL_OPTIONS is (perhaps implicitly) being used from there. For our own subprocesses, we should probably be scrubbing the environment. > Shell commands run by Impala can fail when using the Java debugger > -- > > Key: IMPALA-6293 > URL: https://issues.apache.org/jira/browse/IMPALA-6293 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Joe McDonnell >Priority: Major > > Impala has several parameters that specify shell commands for Impala to run: > s3a_access_key_cmd > s3a_secret_key_cmd > ssl_private_key_password_cmd > webserver_private_key_password_cmd > When debugging the JVM inside the Impala process, it is useful to specify > JAVA_TOOL_OPTIONS to run the Java debugger on a particular port. However, > JAVA_TOOL_OPTIONS remains in the environment, so it is passed to these shell > commands. If any of these shell commands run java, then that JVM will attempt > to use the JAVA_TOOL_OPTIONS specified and thus try to bind to the same port. > The Impala process JVM is already bound to that port, so this will fail. > Several of these commands run at startup, so Impala will fail to startup with > the Java debugger. > Impala should be careful about the environment variables that get passed to > these shell programs. In particular, JAVA_TOOL_OPTIONS should be scrubbed of > any Java debugger configuration to avoid these port conflicts. It might be > best to simply null out JAVA_TOOL_OPTIONS for these commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7834) Impala different types of crashes
Manikandan R created IMPALA-7834: Summary: Impala different types of crashes Key: IMPALA-7834 URL: https://issues.apache.org/jira/browse/IMPALA-7834 Project: IMPALA Issue Type: Bug Reporter: Manikandan R Off late, We had witnessed different types of crashes in cluster. I don't see any similarities among crashes stack traces and also not able to reproduce. Below are the stack traces occurred in different daemons at different timings. I do have complete stack traces and let me know if it helps for further debugging. 1) 10-1-33-172 Nov 6 18:04 Thread 1 (Thread 0x7f018b423700 (LWP 96325)): #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x7f0aad280185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #3 0x7f0aad422593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #4 0x7f0aad28568f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #6 No symbol table info available. #7 0x7f0a44ca3000 in ?? () No symbol table info available. #8 0x00dba5c1 in impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () No symbol table info available. #9 0x00dba924 in impala::HdfsParquetScanner::AssembleRows(std::vector > const&, impala::RowBatch*, bool*) () No symbol table info available. #10 0x00dbf5f6 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () No symbol table info available. #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () No symbol table info available. #12 0x00d835e6 in impala::HdfsScanNode::ProcessSplit(std::vector > const&, impala::MemPool*, impala::io::ScanRange*) () No symbol table info available. #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () No symbol table info available. #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function, impala::Promise*) () No symbol table info available. #15 0x00d173c4 in boost::detail::thread_data, impala::Promise*), boost::_bi::list4, boost::_bi::value, boost::_bi::value >, boost::_bi::value*> > > >::run() () No symbol table info available. #16 0x0128fada in thread_proxy () No symbol table info available. #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 No symbol table info available. Nov 1 11:21 Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x7fc297d12185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #3 0x7fc297eb4593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #4 0x7fc297d1768f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #6 No symbol table info available. #7 0x7fc1dd9f in ?? () No symbol table info available. #8 0x00fdd9f9 in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () No symbol table info available. #9 0x00b74d6d in impala::FragmentInstanceState::Open() () No symbol table info available. #10 0x00b763ab in impala::FragmentInstanceState::Exec() () No symbol table info available. #11 0x00b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () No symbol table info available. #12 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function, impala::Promise*) () No symbol table info available. #13 0x00d173c4 in boost::detail::thread_data, impala::Promise*), boost::_bi::list4, boost::_bi::value, boost::_bi::value >, boost::_bi::value*> > > >::run() () No symbol table info available. #14 0x0128fada in thread_proxy () No symbol table info available. #15 0x7fc295d8edd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #16 0x7fc295ab8b3d in clone () from /lib64/libc.so.6 No symbol table info available. 2)
[jira] [Updated] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
[
https://issues.apache.org/jira/browse/IMPALA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fredy Wijaya updated IMPALA-7835:
-
Description:
Start Impala with object ownership enabled.
{noformat}
[localhost:21000] default> create database foo;
[localhost:21000] default> create role impdev;
[localhost:21000] default> grant all on server to role impdev;
[localhost:21000] default> grant role impdev to group impdev;
[localhost:21000] default> invalidate metadata; -- this will hang
{noformat}
was:
Start Impala with object ownership enabled.
{noformat}
[localhost:21000] default> create role impdev;
[localhost:21000] default> grant all on server to role impdev;
[localhost:21000] default> grant role impdev to group impdev;
[localhost:21000] default> invalidate metadata; -- this will hang
{noformat}
> Creating a role with the same name as the user name with object ownership
> enabled can cause INVALIDATE METADATA to hang
> ---
>
> Key: IMPALA-7835
> URL: https://issues.apache.org/jira/browse/IMPALA-7835
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Major
>
> Start Impala with object ownership enabled.
> {noformat}
> [localhost:21000] default> create database foo;
> [localhost:21000] default> create role impdev;
> [localhost:21000] default> grant all on server to role impdev;
> [localhost:21000] default> grant role impdev to group impdev;
> [localhost:21000] default> invalidate metadata; -- this will hang
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7791) Aggregation Node memory estimates don't account for number of fragment instances
[
https://issues.apache.org/jira/browse/IMPALA-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nilangekar resolved IMPALA-7791.
--
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Aggregation Node memory estimates don't account for number of fragment
> instances
>
>
> Key: IMPALA-7791
> URL: https://issues.apache.org/jira/browse/IMPALA-7791
> Project: IMPALA
> Issue Type: Sub-task
>Affects Versions: Impala 3.1.0
>Reporter: Pooja Nilangekar
>Assignee: Pooja Nilangekar
>Priority: Blocker
> Fix For: Impala 3.1.0
>
>
> AggregationNode's memory estimates are calculated based on the input
> cardinality of the node, without accounting for the division of input data
> across fragment instances. This results in very high memory estimates. In
> reality, the nodes often use only a part of this memory.
> Example query:
> {code:java}
> [localhost:21000] default> select distinct * from tpch.lineitem limit 5;
> {code}
> Summary:
> {code:java}
> +--++--+--+---++---+---+---+
> | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem
> | Est. Peak Mem | Detail
>
>
>
>
>|
> +--++--+--+---++---+---+---+
> | 04:EXCHANGE | 1 | 21.24us | 21.24us | 5 | 5 | 48.00 KB
> | 16.00 KB | UNPARTITIONED
>
>
>
>
>|
> | 03:AGGREGATE | 3 | 5.11s| 5.15s| 15| 5 | 576.21
> MB | 1.62 GB | FINALIZE
>
>
>
>
> |
> | 02:EXCHANGE | 3 | 709.75ms | 728.91ms | 6.00M | 6.00M | 5.46 MB
> | 10.78 MB |
> HASH(tpch.lineitem.l_orderkey,tpch.lineitem.l_partkey,tpch.lineitem.l_suppkey,tpch.lineitem.l_linenumber,tpch.lineitem.l_quantity,tpch.lineitem.l_extendedprice,tpch.lineitem.l_discount,tpch.lineitem.l_tax,tpch.lineitem.l_returnflag,tpch.lineitem.l_linestatus,tpch.lineitem.l_shipdate,tpch.lineitem.l_commitdate,tpch.lineitem.l_receiptdate,tpch.lineitem.l_shipinstruct,tpch.lineitem.l_shipmode,tpch.lineitem.l_comment)
> |
> | 01:AGGREGATE | 3 | 4.37s| 4.70s| 6.00M | 6.00M | 36.77 MB
> | 1.62 GB | STREAMING
>
>
>
>
>|
> | 00:SCAN HDFS | 3 | 437.14ms | 480.60ms | 6.00M |
[jira] [Created] (IMPALA-7833) Audit and fix other string builtins for long string handling
Tim Armstrong created IMPALA-7833:
-
Summary: Audit and fix other string builtins for long string
handling
Key: IMPALA-7833
URL: https://issues.apache.org/jira/browse/IMPALA-7833
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.0, Impala 2.11.0, Impala 3.1.0
Reporter: Tim Armstrong
Following on from IMPALA-7822, there are some other string builtins that seem
to follow the same pattern of having a string size overflow an int passed into
the StringVal constructor. I think in some cases we get lucky and it works out,
but others it seems possible to crash given the right input values.
Here are some examples of cases where we can hit such bugs:
{noformat}
select lpad('foo', 17179869184 , ' ');
select rpad('foo', 17179869184 , ' ');
select space(17179869184 );
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5004) Switch to sorting node for large TopN queries
[ https://issues.apache.org/jira/browse/IMPALA-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved IMPALA-5004. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 > Switch to sorting node for large TopN queries > - > > Key: IMPALA-5004 > URL: https://issues.apache.org/jira/browse/IMPALA-5004 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.9.0 >Reporter: Lars Volker >Assignee: Sahil Takiar >Priority: Major > Fix For: Impala 3.1.0 > > > As explained by [~tarmstrong] in IMPALA-4995: > bq. We should also consider switching to the sort operator for large limits. > This allows it to spill. The memory requirements for TopN also are > problematic for large limits, since it would allocate large vectors that are > untracked and also require a large amount of contiguous memory. > There's already logic to select TopN vs. Sort: > [planner/SingleNodePlanner.java#L289|https://github.com/apache/incubator-impala/blob/master/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L289] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
[
https://issues.apache.org/jira/browse/IMPALA-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7835 started by Fredy Wijaya.
> Creating a role with the same name as the user name with object ownership
> enabled can cause INVALIDATE METADATA to hang
> ---
>
> Key: IMPALA-7835
> URL: https://issues.apache.org/jira/browse/IMPALA-7835
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Major
>
> Start Impala with object ownership enabled.
> {noformat}
> [localhost:21000] default> create role impdev;
> [localhost:21000] default> grant all on server to role impdev;
> [localhost:21000] default> grant role impdev to group impdev;
> [localhost:21000] default> invalidate metadata; -- this will hang
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7837) SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
Michael Brown created IMPALA-7837:
-
Summary: SCAN_BYTES_LIMIT="100M" test failing to raise exception
in release build
Key: IMPALA-7837
URL: https://issues.apache.org/jira/browse/IMPALA-7837
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Michael Brown
Assignee: Bikramjeet Vig
Attachments: impala-logs.tar.gz
This test is not raising the expected exception on a *release build*:
{noformat}
QUERY
# Query should fail due to exceeding scan bytes limit.
set SCAN_BYTES_LIMIT="100M";
select count(*) from tpch.lineitem l1,tpch.lineitem l2, tpch.lineitem l3 where
l1.l_suppkey = l2.l_linenumber and l1.l_orderkey = l2.l_orderkey and
l1.l_orderkey = l3.l_orderkey group by l1.l_comment, l2.l_comment
having count(*) = 99
CATCH
row_regex:.*terminated due to scan bytes limit of 100.00 M.
{noformat}
{noformat}
Stacktrace
query_test/test_resource_limits.py:46: in test_resource_limits
self.run_test_case('QueryTest/query-resource-limits', vector)
common/impala_test_suite.py:482: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: row_regex:.*terminated due to scan
bytes limit of 100.00 M.*
{noformat}
It fails deterministically in CI (3 times in a row). I can't find a query
profile matching the query ID for some reason, but I've attached logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IMPALA-7733) TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
[
https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7733:
--
Labels: broken-build flaky (was: )
> TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
> -
>
> Key: IMPALA-7733
> URL: https://issues.apache.org/jira/browse/IMPALA-7733
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build, flaky
>
> I see two examples in the past two months or so where this test fails due to
> a rename error on S3. The test's stacktrace looks like this:
> {noformat}
> query_test/test_insert_parquet.py:112: in test_insert_parquet
> self.run_test_case('insert_parquet', vector, unique_database,
> multiple_impalad=True)
> common/impala_test_suite.py:408: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:625: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:176: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:350: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:371: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Error(s) moving partition files. First error (of 1) was:
> Hdfs op (RENAME
> s3a:///test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> TO
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq)
> failed, error was:
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> E Error(5): Input/output error{noformat}
> Since we know this happens once in a while, some ideas to deflake it:
> * retry
> * check for this specific issue... if we think its platform flakiness, then
> we should skip it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7834) Impala different types of crashes
[ https://issues.apache.org/jira/browse/IMPALA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated IMPALA-7834: - Affects Version/s: Impala 2.11.0 Labels: CDH5 (was: ) Component/s: Backend > Impala different types of crashes > - > > Key: IMPALA-7834 > URL: https://issues.apache.org/jira/browse/IMPALA-7834 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Manikandan R >Priority: Major > Labels: CDH5 > > Off late, We had witnessed different types of crashes in cluster. I don't see > any similarities among crashes stack traces and also not able to reproduce. > Below are the stack traces occurred in different daemons at different > timings. I do have complete stack traces and let me know if it helps for > further debugging. > 1) > 10-1-33-172 > Nov 6 18:04 > Thread 1 (Thread 0x7f018b423700 (LWP 96325)): > #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7f0aad280185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7f0aad422593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7f0aad28568f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7f0a44ca3000 in ?? () > No symbol table info available. > #8 0x00dba5c1 in > impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () > No symbol table info available. > #9 0x00dba924 in > impala::HdfsParquetScanner::AssembleRows(std::vector std::allocator > const&, impala::RowBatch*, > bool*) () > No symbol table info available. > #10 0x00dbf5f6 in > impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () > No symbol table info available. > #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () > No symbol table info available. > #12 0x00d835e6 in > impala::HdfsScanNode::ProcessSplit(std::vector std::allocator > const&, impala::MemPool*, > impala::io::ScanRange*) () > No symbol table info available. > #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () > No symbol table info available. > #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, > std::string const&, boost::function, impala::Promise*) () > No symbol table info available. > #15 0x00d173c4 in boost::detail::thread_data void (*)(std::string const&, std::string const&, boost::function, > impala::Promise*), boost::_bi::list4, > boost::_bi::value, boost::_bi::value >, > boost::_bi::value*> > > >::run() () > No symbol table info available. > #16 0x0128fada in thread_proxy () > No symbol table info available. > #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 > No symbol table info available. > #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 > No symbol table info available. > Nov 1 11:21 > Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): > #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x7fc297d12185 in os::abort(bool) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #3 0x7fc297eb4593 in VMError::report_and_die() () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #4 0x7fc297d1768f in JVM_handle_linux_signal () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from > /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so > No symbol table info available. > #6 > No symbol table info available. > #7 0x7fc1dd9f in ?? () > No symbol table info available. > #8 0x00fdd9f9 in > impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () > No symbol table info available. > #9 0x00b74d6d in impala::FragmentInstanceState::Open() () > No symbol table info available. > #10 0x00b763ab in impala::FragmentInstanceState::Exec() () > No symbol table info available. > #11 0x00b65b38 in > impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () > No symbol table
[jira] [Commented] (IMPALA-6293) Shell commands run by Impala can fail when using the Java debugger
[ https://issues.apache.org/jira/browse/IMPALA-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679871#comment-16679871 ] Fredy Wijaya commented on IMPALA-6293: -- Wouldn't it be better to have a flag like --jvm_args to debug Impala instead of relying on JAVA_TOOL_OPTIONS? > Shell commands run by Impala can fail when using the Java debugger > -- > > Key: IMPALA-6293 > URL: https://issues.apache.org/jira/browse/IMPALA-6293 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Joe McDonnell >Priority: Major > > Impala has several parameters that specify shell commands for Impala to run: > s3a_access_key_cmd > s3a_secret_key_cmd > ssl_private_key_password_cmd > webserver_private_key_password_cmd > When debugging the JVM inside the Impala process, it is useful to specify > JAVA_TOOL_OPTIONS to run the Java debugger on a particular port. However, > JAVA_TOOL_OPTIONS remains in the environment, so it is passed to these shell > commands. If any of these shell commands run java, then that JVM will attempt > to use the JAVA_TOOL_OPTIONS specified and thus try to bind to the same port. > The Impala process JVM is already bound to that port, so this will fail. > Several of these commands run at startup, so Impala will fail to startup with > the Java debugger. > Impala should be careful about the environment variables that get passed to > these shell programs. In particular, JAVA_TOOL_OPTIONS should be scrubbed of > any Java debugger configuration to avoid these port conflicts. It might be > best to simply null out JAVA_TOOL_OPTIONS for these commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4063) Make fragment instance reports per-query (or per-host) instead of per-fragment instance.
[ https://issues.apache.org/jira/browse/IMPALA-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679475#comment-16679475 ] ASF subversion and git services commented on IMPALA-4063: - Commit d1aa1c009f62500ae2ee8cd915751b0d42bee911 in impala's branch refs/heads/master from Michael Ho [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d1aa1c0 ] IMPALA-7828: A temporary workaround for flaky UDF test (test_mem_leak()) Before IMPALA-4063, the error message detected during FragmentInstanceState::Close() was always lost. After IMPALA-4063, we may sometimes get the error message in FragmentInstanceState::Close(). It's non-deterministic as the fragment instance thread may race with the query state thread which reports the final status. The UDF test currently tries to handle this non-determinism by using "row_regex:.*" in the ERRORS section but it doesn't always seem to work. This change workarounds the issue by commenting out the ERRORS section in udf-no-expr-rewrite.text for now. The actual fix will be done in IMPALA-7829. Change-Id: I6a55d5ad1a5a7278e7390f60854a8df28c1b9f28 Reviewed-on: http://gerrit.cloudera.org:8080/11900 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Make fragment instance reports per-query (or per-host) instead of > per-fragment instance. > > > Key: IMPALA-4063 > URL: https://issues.apache.org/jira/browse/IMPALA-4063 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Affects Versions: Impala 2.7.0 >Reporter: Sailesh Mukil >Assignee: Michael Ho >Priority: Major > Labels: impala-scalability-sprint-08-13-2018, performance > Fix For: Impala 3.2.0 > > > Currently we send a report per-fragment instance to the coordinator every 5 > seconds (by default; modifiable via query option 'status_report_interval'). > For queries with a large number of fragment instances, this generates > tremendous amounts of network traffic to the coordinator, which will only be > aggravated with higher a DOP. > We should instead queue per-fragment instance reports and send out a > per-query report to the coordinator instead. > For code references, see: > PlanFragmentExecutor:: ReportProfile() > PlanFragmentExecutor:: SendReport() > FragmentExecState:: ReportStatusCb() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
Sahil Takiar created IMPALA-7836: Summary: Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion Key: IMPALA-7836 URL: https://issues.apache.org/jira/browse/IMPALA-7836 Project: IMPALA Issue Type: Sub-task Components: Frontend Affects Versions: Impala 2.9.0 Reporter: Sahil Takiar Assignee: Alex Rodoni IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we should document. The changes in IMPALA-5004 work by estimating the amount of memory required to run a TopN operator. The memory estimate is based on the size of the individual tuples that need to be processed by the TopN operator, as well as the sum of the limit and offset in the query. TopN operators don't spill to disk so they have to keep all rows they process in memory. If the estimated size of the working set of the TopN operator exceeds the threshold of 'topn_bytes_limit' the TopN operator will be replaced with a Sort operator. The Sort operator can spill to disk, but it processes all the data (the limit and offset have no affect). So switching to Sort might incur performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IMPALA-7528) Division by zero when computing cardinalities of many to many joins on NULL columns
[
https://issues.apache.org/jira/browse/IMPALA-7528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679477#comment-16679477
]
ASF subversion and git services commented on IMPALA-7528:
-
Commit 898c515882dc64e1c98bff073f5529ea63a20cfe in impala's branch
refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=898c515 ]
IMPALA-7528: Fix division by zero when computing cardinalities
This patch fixes a case where there can be a division by zero when
computing cardinalities of many to many joins on NULL columns.
Testing:
Added a planner test case
Change-Id: Ieedd51d3ad6131a4ea2a5883f05104e6a0f2cd14
Reviewed-on: http://gerrit.cloudera.org:8080/11901
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Division by zero when computing cardinalities of many to many joins on NULL
> columns
> ---
>
> Key: IMPALA-7528
> URL: https://issues.apache.org/jira/browse/IMPALA-7528
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.12.0
>Reporter: Balazs Jeszenszky
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: planner
>
> The following:
> {code:java}
> | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | Per-Host Resources: mem-estimate=33.94MB mem-reservation=1.94MB|
> | 02:HASH JOIN [INNER JOIN, BROADCAST] |
> | | hash predicates: b.code = a.code|
> | | fk/pk conjuncts: none |
> | | runtime filters: RF000 <- a.code|
> | | mem-estimate=1.94MB mem-reservation=1.94MB spill-buffer=64.00KB |
> | | tuple-ids=1,0 row-size=163B cardinality=9223372036854775807 |
> < Estimation due to overflow.
> | | |
> | |--03:EXCHANGE [BROADCAST] |
> | | | mem-estimate=0B mem-reservation=0B |
> | | | tuple-ids=0 row-size=82B cardinality=823 |
> | | | |
> | | F01:PLAN FRAGMENT [RANDOM] hosts=1 instances=1 |
> | | Per-Host Resources: mem-estimate=32.00MB mem-reservation=0B |
> | | 00:SCAN HDFS [default.sample_07 a, RANDOM] |
> | | partitions=1/1 files=1 size=44.98KB |
> | | stats-rows=823 extrapolated-rows=disabled|
> | | table stats: rows=823 size=44.98KB |
> | | column stats: all|
> | | mem-estimate=32.00MB mem-reservation=0B |
> | | tuple-ids=0 row-size=82B cardinality=823 |
> | | |
> | 01:SCAN HDFS [default.sample_08 b, RANDOM] |
> |partitions=1/1 files=1 size=44.99KB |
> |runtime filters: RF000 -> b.code|
> |stats-rows=823 extrapolated-rows=disabled |
> |table stats: rows=823 size=44.99KB |
> |column stats: all |
> |mem-estimate=32.00MB mem-reservation=0B |
> |tuple-ids=1 row-size=82B cardinality=823|
> ++
> {code}
> is the result of both join columns having 0 as NDV.
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/org/apache/impala/planner/JoinNode.java#L368
> should handle this more gracefully.
> IMPALA-7310 makes it a bit more likely that someone will run into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7837) SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
Michael Brown created IMPALA-7837:
-
Summary: SCAN_BYTES_LIMIT="100M" test failing to raise exception
in release build
Key: IMPALA-7837
URL: https://issues.apache.org/jira/browse/IMPALA-7837
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.1.0
Reporter: Michael Brown
Assignee: Bikramjeet Vig
Attachments: impala-logs.tar.gz
This test is not raising the expected exception on a *release build*:
{noformat}
QUERY
# Query should fail due to exceeding scan bytes limit.
set SCAN_BYTES_LIMIT="100M";
select count(*) from tpch.lineitem l1,tpch.lineitem l2, tpch.lineitem l3 where
l1.l_suppkey = l2.l_linenumber and l1.l_orderkey = l2.l_orderkey and
l1.l_orderkey = l3.l_orderkey group by l1.l_comment, l2.l_comment
having count(*) = 99
CATCH
row_regex:.*terminated due to scan bytes limit of 100.00 M.
{noformat}
{noformat}
Stacktrace
query_test/test_resource_limits.py:46: in test_resource_limits
self.run_test_case('QueryTest/query-resource-limits', vector)
common/impala_test_suite.py:482: in run_test_case
assert False, "Expected exception: %s" % expected_str
E AssertionError: Expected exception: row_regex:.*terminated due to scan
bytes limit of 100.00 M.*
{noformat}
It fails deterministically in CI (3 times in a row). I can't find a query
profile matching the query ID for some reason, but I've attached logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7504) ParseKerberosPrincipal() should use krb5_parse_name() instead
[
https://issues.apache.org/jira/browse/IMPALA-7504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7504:
--
Target Version: Impala 3.2.0 (was: Impala 3.1.0)
> ParseKerberosPrincipal() should use krb5_parse_name() instead
> -
>
> Key: IMPALA-7504
> URL: https://issues.apache.org/jira/browse/IMPALA-7504
> Project: IMPALA
> Issue Type: Improvement
> Components: Security
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Michael Ho
>Priority: Minor
> Labels: ramp-up
>
> [~tlipcon] pointed out during code review that we should be using
> krb5_parse_name() to parse the principal instead of creating our own
> bq. I wonder whether we should just be using krb5_parse_name here instead of
> implementing our own parsing? According to
> [http://web.mit.edu/kerberos/krb5-1.15/doc/appdev/refs/api/krb5_parse_name.html]
> there are various escapings, etc, that this function isn't currently
> supporting.
> We currently do the following to parse the principal:
> {noformat}
> vector names;
> split(names, principal, is_any_of("/"));
> if (names.size() != 2) return Status(TErrorCode::BAD_PRINCIPAL_FORMAT,
> principal);
> *service_name = names[0];
> string remaining_principal = names[1];
> split(names, remaining_principal, is_any_of("@"));
> if (names.size() != 2) return Status(TErrorCode::BAD_PRINCIPAL_FORMAT,
> principal);
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4063) Make fragment instance reports per-query (or per-host) instead of per-fragment instance.
[ https://issues.apache.org/jira/browse/IMPALA-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679474#comment-16679474 ] ASF subversion and git services commented on IMPALA-4063: - Commit d1aa1c009f62500ae2ee8cd915751b0d42bee911 in impala's branch refs/heads/master from Michael Ho [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d1aa1c0 ] IMPALA-7828: A temporary workaround for flaky UDF test (test_mem_leak()) Before IMPALA-4063, the error message detected during FragmentInstanceState::Close() was always lost. After IMPALA-4063, we may sometimes get the error message in FragmentInstanceState::Close(). It's non-deterministic as the fragment instance thread may race with the query state thread which reports the final status. The UDF test currently tries to handle this non-determinism by using "row_regex:.*" in the ERRORS section but it doesn't always seem to work. This change workarounds the issue by commenting out the ERRORS section in udf-no-expr-rewrite.text for now. The actual fix will be done in IMPALA-7829. Change-Id: I6a55d5ad1a5a7278e7390f60854a8df28c1b9f28 Reviewed-on: http://gerrit.cloudera.org:8080/11900 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Make fragment instance reports per-query (or per-host) instead of > per-fragment instance. > > > Key: IMPALA-4063 > URL: https://issues.apache.org/jira/browse/IMPALA-4063 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Affects Versions: Impala 2.7.0 >Reporter: Sailesh Mukil >Assignee: Michael Ho >Priority: Major > Labels: impala-scalability-sprint-08-13-2018, performance > Fix For: Impala 3.2.0 > > > Currently we send a report per-fragment instance to the coordinator every 5 > seconds (by default; modifiable via query option 'status_report_interval'). > For queries with a large number of fragment instances, this generates > tremendous amounts of network traffic to the coordinator, which will only be > aggravated with higher a DOP. > We should instead queue per-fragment instance reports and send out a > per-query report to the coordinator instead. > For code references, see: > PlanFragmentExecutor:: ReportProfile() > PlanFragmentExecutor:: SendReport() > FragmentExecState:: ReportStatusCb() -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7834) Impala different types of crashes
Manikandan R created IMPALA-7834: Summary: Impala different types of crashes Key: IMPALA-7834 URL: https://issues.apache.org/jira/browse/IMPALA-7834 Project: IMPALA Issue Type: Bug Reporter: Manikandan R Off late, We had witnessed different types of crashes in cluster. I don't see any similarities among crashes stack traces and also not able to reproduce. Below are the stack traces occurred in different daemons at different timings. I do have complete stack traces and let me know if it helps for further debugging. 1) 10-1-33-172 Nov 6 18:04 Thread 1 (Thread 0x7f018b423700 (LWP 96325)): #0 0x7f0aaaf5e207 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x7f0aaaf5fa38 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x7f0aad280185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #3 0x7f0aad422593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #4 0x7f0aad28568f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #5 0x7f0aad27bbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #6 No symbol table info available. #7 0x7f0a44ca3000 in ?? () No symbol table info available. #8 0x00dba5c1 in impala::HdfsParquetScanner::TransferScratchTuples(impala::RowBatch*) () No symbol table info available. #9 0x00dba924 in impala::HdfsParquetScanner::AssembleRows(std::vector > const&, impala::RowBatch*, bool*) () No symbol table info available. #10 0x00dbf5f6 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) () No symbol table info available. #11 0x00db9ba7 in impala::HdfsParquetScanner::ProcessSplit() () No symbol table info available. #12 0x00d835e6 in impala::HdfsScanNode::ProcessSplit(std::vector > const&, impala::MemPool*, impala::io::ScanRange*) () No symbol table info available. #13 0x00d85115 in impala::HdfsScanNode::ScannerThread() () No symbol table info available. #14 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function, impala::Promise*) () No symbol table info available. #15 0x00d173c4 in boost::detail::thread_data, impala::Promise*), boost::_bi::list4, boost::_bi::value, boost::_bi::value >, boost::_bi::value*> > > >::run() () No symbol table info available. #16 0x0128fada in thread_proxy () No symbol table info available. #17 0x7f0aab2fcdd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #18 0x7f0aab026b3d in clone () from /lib64/libc.so.6 No symbol table info available. Nov 1 11:21 Thread 1 (Thread 0x7fb3bffe9700 (LWP 50334)): #0 0x7fc2959f0207 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x7fc2959f18f8 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x7fc297d12185 in os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #3 0x7fc297eb4593 in VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #4 0x7fc297d1768f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #5 0x7fc297d0dbe3 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so No symbol table info available. #6 No symbol table info available. #7 0x7fc1dd9f in ?? () No symbol table info available. #8 0x00fdd9f9 in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) () No symbol table info available. #9 0x00b74d6d in impala::FragmentInstanceState::Open() () No symbol table info available. #10 0x00b763ab in impala::FragmentInstanceState::Exec() () No symbol table info available. #11 0x00b65b38 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () No symbol table info available. #12 0x00d16c83 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function, impala::Promise*) () No symbol table info available. #13 0x00d173c4 in boost::detail::thread_data, impala::Promise*), boost::_bi::list4, boost::_bi::value, boost::_bi::value >, boost::_bi::value*> > > >::run() () No symbol table info available. #14 0x0128fada in thread_proxy () No symbol table info available. #15 0x7fc295d8edd5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #16 0x7fc295ab8b3d in clone () from /lib64/libc.so.6 No symbol table info available. 2)
[jira] [Commented] (IMPALA-5004) Switch to sorting node for large TopN queries
[ https://issues.apache.org/jira/browse/IMPALA-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680041#comment-16680041 ] Sahil Takiar commented on IMPALA-5004: -- [~arodoni_cloudera] filed IMPALA-7836 > Switch to sorting node for large TopN queries > - > > Key: IMPALA-5004 > URL: https://issues.apache.org/jira/browse/IMPALA-5004 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.9.0 >Reporter: Lars Volker >Assignee: Sahil Takiar >Priority: Major > Fix For: Impala 3.1.0 > > > As explained by [~tarmstrong] in IMPALA-4995: > bq. We should also consider switching to the sort operator for large limits. > This allows it to spill. The memory requirements for TopN also are > problematic for large limits, since it would allocate large vectors that are > untracked and also require a large amount of contiguous memory. > There's already logic to select TopN vs. Sort: > [planner/SingleNodePlanner.java#L289|https://github.com/apache/incubator-impala/blob/master/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L289] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7824) Running INVALIDATE METADATA with authorization enabled can cause a hang if Sentry is unavailable
[ https://issues.apache.org/jira/browse/IMPALA-7824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679478#comment-16679478 ] ASF subversion and git services commented on IMPALA-7824: - Commit 76842acc34f05760defb6e803df96fe61699ede0 in impala's branch refs/heads/master from [~fredyw] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=76842ac ] IMPALA-7824: INVALIDATE METADATA should not hang when Sentry is unavailable Before this patch, running INVALIDATE METADATA when Sentry is unavailable could cause Impala query to hang. PolicyReader thread in SentryProxy is used by two use cases, one as a background thread that periodically refreshes Sentry policy and another one as a synchronous operation for INVALIDATE METADATA. For the background thread, we need to swallow any exception thrown while refreshing the Sentry policy in order to not kill the background thread. For a synchronous reset operation, such as INVALIDATE METADATA, swallowing an exception causes the Impala catalog to wait indefinitely for authorization catalog objects that never get processed due to Sentry being unavailable. The patch updates the code by not swallowing any exception in INVALIDATE METADATA and return the exception to the caller. Testing: - Ran all FE tests - Added a new E2E test - Ran all E2E authorization tests Change-Id: Icff987a6184f62a338faadfdc1a0d349d912fc37 Reviewed-on: http://gerrit.cloudera.org:8080/11897 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Running INVALIDATE METADATA with authorization enabled can cause a hang if > Sentry is unavailable > > > Key: IMPALA-7824 > URL: https://issues.apache.org/jira/browse/IMPALA-7824 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > > Steps to reproduce: > 1. Start Impala with authorization > 2. Kill Sentry > 3. Run INVALIDATE METADATA -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-6924) Compute stats profiles should include reference to child queries
[ https://issues.apache.org/jira/browse/IMPALA-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6924: -- Description: "Compute stats" queries spawn off child queries that do most of the work. It's non-trivial to track down the child queries and get their profiles if something goes wrong. We really should have, at a minimum, the query IDs of the child queries in the parent's profile and vice-versa. was:"Compute stats" queries spawn off child queries that do most of the work. It's non-trivial to track down the child queries and get their profiles if something goes wrong. We really should have, at a minimum, the query IDs of the child queries. in the profile. Maybe we could do better than that. > Compute stats profiles should include reference to child queries > > > Key: IMPALA-6924 > URL: https://issues.apache.org/jira/browse/IMPALA-6924 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Tim Armstrong >Priority: Major > Labels: observability, supportability > > "Compute stats" queries spawn off child queries that do most of the work. > It's non-trivial to track down the child queries and get their profiles if > something goes wrong. We really should have, at a minimum, the query IDs of > the child queries in the parent's profile and vice-versa. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7835) Creating a role with the same name as the user name with object ownership enabled can cause INVALIDATE METADATA to hang
Fredy Wijaya created IMPALA-7835:
Summary: Creating a role with the same name as the user name with
object ownership enabled can cause INVALIDATE METADATA to hang
Key: IMPALA-7835
URL: https://issues.apache.org/jira/browse/IMPALA-7835
Project: IMPALA
Issue Type: Bug
Components: Catalog
Affects Versions: Impala 3.1.0
Reporter: Fredy Wijaya
Assignee: Fredy Wijaya
Start Impala with object ownership enabled.
{noformat}
[localhost:21000] default> create role impdev;
[localhost:21000] default> grant all on server to role impdev;
[localhost:21000] default> grant role impdev to group impdev;
[localhost:21000] default> invalidate metadata; -- this will hang
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7836) Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion
Sahil Takiar created IMPALA-7836: Summary: Impala 3.1 Doc: New query option 'topn_bytes_limit' for TopN to Sort conversion Key: IMPALA-7836 URL: https://issues.apache.org/jira/browse/IMPALA-7836 Project: IMPALA Issue Type: Sub-task Components: Frontend Affects Versions: Impala 2.9.0 Reporter: Sahil Takiar Assignee: Alex Rodoni IMPALA-5004 adds a new query level option called 'topn_bytes_limit' that we should document. The changes in IMPALA-5004 work by estimating the amount of memory required to run a TopN operator. The memory estimate is based on the size of the individual tuples that need to be processed by the TopN operator, as well as the sum of the limit and offset in the query. TopN operators don't spill to disk so they have to keep all rows they process in memory. If the estimated size of the working set of the TopN operator exceeds the threshold of 'topn_bytes_limit' the TopN operator will be replaced with a Sort operator. The Sort operator can spill to disk, but it processes all the data (the limit and offset have no affect). So switching to Sort might incur performance penalties, but it will require less memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7083) AnalysisException for GROUP BY and ORDER BY expressions that are folded to constants from 2.9 onwards
[
https://issues.apache.org/jira/browse/IMPALA-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7083:
--
Target Version: Impala 3.2.0 (was: Impala 3.1.0)
> AnalysisException for GROUP BY and ORDER BY expressions that are folded to
> constants from 2.9 onwards
> -
>
> Key: IMPALA-7083
> URL: https://issues.apache.org/jira/browse/IMPALA-7083
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.9.0
>Reporter: Eric Lin
>Priority: Critical
> Labels: regression
>
> To reproduce, please run below impala query:
> {code}
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (a int);
> SELECT (
> CASE
>WHEN (1 =1)
>THEN 1
>ELSE a
> end) AS b
> FROM test
> GROUP BY 1
> ORDER BY (
> CASE
>WHEN (1 =1)
>THEN 1
>ELSE a
> end);
> {code}
> It will fail with below error:
> {code}
> ERROR: AnalysisException: ORDER BY expression not produced by aggregation
> output (missing from GROUP BY clause?): (CASE WHEN TRUE THEN 1 ELSE a END)
> {code}
> However, if I replace column name "a" as a constant value, it works:
> {code}
> SELECT (
> CASE
>WHEN (1 =1)
>THEN 1
>ELSE 2
> end) AS b
> FROM test
> GROUP BY 1
> ORDER BY (
> CASE
>WHEN (1 =1)
>THEN 1
>ELSE 2
> end);
> {code}
> This issue is identified in CDH5.12.x (Impala 2.9), and no issues in 5.11.x
> (Impala 2.8).
> We know that it can be worked around by re-write as below:
> {code}
> SELECT (
> CASE
>WHEN (1 =1)
>THEN 1
>ELSE a
> end) AS b
> FROM test
> GROUP BY 1
> ORDER BY 1;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7754) Expressions sometimes not re-analyzed after rewrite
[
https://issues.apache.org/jira/browse/IMPALA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679132#comment-16679132
]
Paul Rogers edited comment on IMPALA-7754 at 11/8/18 1:54 AM:
--
There be dragons here. It seems that, without rewrites, we get slightly wrong
results in certain pathological queries. Consider this case in
{{test-empty.sql}}:
{noformat}
# Constant conjunct in the ON-clause of an outer join is
# assigned to the join.
select *
from functional.alltypessmall a
left outer join functional.alltypestiny b
on (a.id = b.id and 1 + 1 > 10)
PLAN
Max Per-Host Resource Reservation: Memory=1.95MB Threads=3
Per-Host Resource Estimates: Memory=66MB
PLAN-ROOT SINK
|
02:HASH JOIN [LEFT OUTER JOIN]
| hash predicates: a.id = b.id
| other join predicates: FALSE
|
|--01:SCAN HDFS [functional.alltypestiny b]
| partitions=4/4 files=4 size=460B
|
00:SCAN HDFS [functional.alltypessmall a]
partitions=4/4 files=4 size=6.32KB
{noformat}
If we re-analyze expressions after rewrite, we get the following, which seems
more accurate:
{noformat}
Max Per-Host Resource Reservation: Memory=16.00KB Threads=3
Per-Host Resource Estimates: Memory=64MB
PLAN-ROOT SINK
|
02:NESTED LOOP JOIN [LEFT OUTER JOIN]
| join predicates: (FALSE)
{noformat}
Note that, in the current expected results (the top example), the predicate
{{a.id = b.id}} is pushed down, ignoring the fact that it is part of a
predicate which reduces to {{FALSE}}. In the second version, the planner has
correctly identified that the join produces no rows. Also, the memory cost is
lower for the second example, Hence, the current behavior (first example)
appears wrong.
The reason for this difference boils down to the lack of re-analysis in the
expression rewriter. Lets' take it step-by-step.
* Pass {{a.id = b.id and 1 + 1 > 10}} to the rewriter.
* Nothing can be done with {{a.id = b.id}}.
* Constant folding: {{1 + 1 > 10}} becomes {{FALSE}}.
* Normalize binary predicates: {{a.id = b.id AND FALSE}} becomes {{FALSE AND
a.id = b.id}}.
* The result of the above is not analyzed (it is a new node), so the call to
simplify the above is skipped.
When re-analysis is added, the last step above changes to:
* Simplify conditional: {{FALSE AND a.id = b.id}} becomes {{FALSE}}.
Thus, the second plan shown above is the correct one, the first one is the
result of an incomplete expression rewrite.
We can verify this by creating a "baseline" query and running it through
{{PlannerTest}}: replace the ON clause with a simple {{FALSE}}:
{noformat}
select *
from functional.alltypessmall a
left outer join functional.alltypestiny b
on FALSE
PLAN
Max Per-Host Resource Reservation: Memory=16.00KB Threads=3
Per-Host Resource Estimates: Memory=64MB
PLAN-ROOT SINK
|
02:NESTED LOOP JOIN [LEFT OUTER JOIN]
| join predicates: FALSE
|
|--01:SCAN HDFS [functional.alltypestiny b]
| partitions=4/4 files=4 size=460B
|
00:SCAN HDFS [functional.alltypessmall a]
partitions=4/4 files=4 size=6.32KB
{noformat}
The rewritten form of the query shown at the top of this node should produce an
identical plan.
was (Author: paul.rogers):
There be dragons here. It seems that, without rewrites, we get slightly wrong
results in certain pathological queries. Consider this case in
{{test-empty.sql}}:
{noformat}
# Constant conjunct in the ON-clause of an outer join is
# assigned to the join.
select *
from functional.alltypessmall a
left outer join functional.alltypestiny b
on (a.id = b.id and 1 + 1 > 10)
PLAN
Max Per-Host Resource Reservation: Memory=1.95MB Threads=3
Per-Host Resource Estimates: Memory=66MB
PLAN-ROOT SINK
|
02:HASH JOIN [LEFT OUTER JOIN]
| hash predicates: a.id = b.id
| other join predicates: FALSE
|
|--01:SCAN HDFS [functional.alltypestiny b]
| partitions=4/4 files=4 size=460B
|
00:SCAN HDFS [functional.alltypessmall a]
partitions=4/4 files=4 size=6.32KB
{noformat}
If we re-analyze expressions after rewrite, we get the following, which seems
more accurate:
{noformat}
Max Per-Host Resource Reservation: Memory=16.00KB Threads=3
Per-Host Resource Estimates: Memory=64MB
PLAN-ROOT SINK
|
02:NESTED LOOP JOIN [LEFT OUTER JOIN]
| join predicates: (FALSE)
{noformat}
Note that, in the current expected results (the top example), the predicate
{{a.id = b.id}} is pushed down, ignoring the fact that it is part of a
predicate which reduces to {{FALSE}}. In the second version, the planner has
correctly identified that the join produces no rows. Also, the memory cost is
lower for the second example, Hence, the current behavior (first example)
appears wrong.
> Expressions sometimes not re-analyzed after rewrite
> ---
>
> Key: IMPALA-7754
> URL: https://issues.apache.org/jira/browse/IMPALA-7754
> Project: IMPALA
> Issue Type: Bug
>
[jira] [Resolved] (IMPALA-7822) Crash in repeat() constructing strings > 1GB
[
https://issues.apache.org/jira/browse/IMPALA-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7822.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Crash in repeat() constructing strings > 1GB
>
>
> Key: IMPALA-7822
> URL: https://issues.apache.org/jira/browse/IMPALA-7822
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0,
> Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0,
> Impala 3.0, Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: crash
> Fix For: Impala 3.1.0
>
>
> {noformat}
> select repeat('x', 1024 * 1024 * 1024 * 1024);
> {noformat}
> {noformat}
> #0 0x04551aa3 in
> google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) ()
> #1 0x7f72cdc0f362 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #2 0x7f72cdc138b9 in JVM_handle_linux_signal ()
>from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #3 0x7f72cdc06f78 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #4
> #5 0x7f72cab8a06c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x030e19da in impala::StringFunctions::Repeat (context=0x721f800,
> str=..., n=...)
> at be/src/exprs/string-functions-ir.cc:100
> #7 0x03145c32 in
> impala::ScalarFnCall::InterpretEval (
> this=0xaf2cb40, eval=0xd26ccc0, row=0x0) at
> be/src/exprs/scalar-fn-call.cc:485
> #8 0x0312bd85 in impala::ScalarFnCall::GetStringVal (this=0xaf2cb40,
> eval=0xd26ccc0,
> row=0x0) at be/src/exprs/scalar-fn-call.cc:599
> #9 0x030da116 in impala::ScalarExprEvaluator::GetValue
> (this=0xd26ccc0, expr=..., row=0x0)
> at be/src/exprs/scalar-expr-evaluator.cc:299
> #10 0x030d9da5 in impala::ScalarExprEvaluator::GetValue
> (this=0xd26ccc0, row=0x0)
> at be/src/exprs/scalar-expr-evaluator.cc:250
> #11 0x01fc59bc in
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (
> env=0xd25b1e0, caller_class=0x7f724d523280,
> thrift_expr_batch=0x7f724d523298,
> thrift_query_ctx_bytes=0x7f724d523290) at be/src/service/fe-support.cc:237
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7829) Send the final profile only after all fragment instances have been closed
[
https://issues.apache.org/jira/browse/IMPALA-7829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679476#comment-16679476
]
ASF subversion and git services commented on IMPALA-7829:
-
Commit d1aa1c009f62500ae2ee8cd915751b0d42bee911 in impala's branch
refs/heads/master from Michael Ho
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d1aa1c0 ]
IMPALA-7828: A temporary workaround for flaky UDF test (test_mem_leak())
Before IMPALA-4063, the error message detected during
FragmentInstanceState::Close() was always lost. After
IMPALA-4063, we may sometimes get the error message in
FragmentInstanceState::Close(). It's non-deterministic
as the fragment instance thread may race with the query
state thread which reports the final status. The UDF test
currently tries to handle this non-determinism by using
"row_regex:.*" in the ERRORS section but it doesn't
always seem to work.
This change workarounds the issue by commenting out the
ERRORS section in udf-no-expr-rewrite.text for now.
The actual fix will be done in IMPALA-7829.
Change-Id: I6a55d5ad1a5a7278e7390f60854a8df28c1b9f28
Reviewed-on: http://gerrit.cloudera.org:8080/11900
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Send the final profile only after all fragment instances have been closed
> -
>
> Key: IMPALA-7829
> URL: https://issues.apache.org/jira/browse/IMPALA-7829
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.2.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Critical
>
> As shown in IMPALA-7828. there is some non-determinism on whether the errors
> detected in {{FragmentInstanceState::Close()}} will show up in the final
> profile sent to the coordinator. The reason is that the current code marks a
> fragment instance "done" after {{ExecInternal()}} completes but before
> {{Close()}} is called. There is a window in between when the final status
> report may be sent before {{Close()}} finishes. We should consider not
> sending the final report until {{Close()}} is done but we probably need to
> understand whether it has any implication to the overall reported query
> execution time. It should have no effect to the "First Rows Available" time
> for a query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7837) SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
[
https://issues.apache.org/jira/browse/IMPALA-7837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680092#comment-16680092
]
Michael Brown commented on IMPALA-7837:
---
[~bikramjeet.vig] Gave this to you to look at, please reassign if necessary.
> SCAN_BYTES_LIMIT="100M" test failing to raise exception in release build
>
>
> Key: IMPALA-7837
> URL: https://issues.apache.org/jira/browse/IMPALA-7837
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Attachments: impala-logs.tar.gz
>
>
> This test is not raising the expected exception on a *release build*:
> {noformat}
> QUERY
> # Query should fail due to exceeding scan bytes limit.
> set SCAN_BYTES_LIMIT="100M";
> select count(*) from tpch.lineitem l1,tpch.lineitem l2, tpch.lineitem l3 where
> l1.l_suppkey = l2.l_linenumber and l1.l_orderkey = l2.l_orderkey and
> l1.l_orderkey = l3.l_orderkey group by l1.l_comment, l2.l_comment
> having count(*) = 99
> CATCH
> row_regex:.*terminated due to scan bytes limit of 100.00 M.
> {noformat}
> {noformat}
> Stacktrace
> query_test/test_resource_limits.py:46: in test_resource_limits
> self.run_test_case('QueryTest/query-resource-limits', vector)
> common/impala_test_suite.py:482: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex:.*terminated due to scan
> bytes limit of 100.00 M.*
> {noformat}
> It fails deterministically in CI (3 times in a row). I can't find a query
> profile matching the query ID for some reason, but I've attached logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7769) Handle CAST(NULL AS type) in rewrites
[
https://issues.apache.org/jira/browse/IMPALA-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679202#comment-16679202
]
Paul Rogers commented on IMPALA-7769:
-
An example failure in {{PlannerTest.testEmpt()}}, for the following case:
{noformat}
# IMPALA-1860: INSERT/CTAS should evaluate and apply constant predicates.
with t as (select * from functional.alltypes where coalesce(NULL) > 10)
insert into functional.alltypes partition(year, month)
select * from t
{noformat}
Monitor the rewrite output. We get:
{noformat}
Before: coalesce(NULL) > 10
After: NULL > 10
{noformat}
Here, the {{NULL}} on the second line is actually {{CAST(NULL AS TINYINT)}}.
The expression should have been further rewritten to just {{NULL}}. The problem
is that the constant folding rule looks for expressions that contains literals,
but here is needs to also look for a CAST of a NULL literal.
> Handle CAST(NULL AS type) in rewrites
> -
>
> Key: IMPALA-7769
> URL: https://issues.apache.org/jira/browse/IMPALA-7769
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Priority: Minor
>
> Consider the following query:
> {code:sql}
> SELECT IFNULL(NULL + 1, id) FROM alltypessmall;
> {code}
> Visualize the rewritten query after analysis (using the new
> {{FullRewriteTest}}.) We get:
> {code:sql}
> SELECT CASE WHEN CAST(NULL AS INT) IS NULL THEN id ELSE NULL END FROM
> alltypessmall
> {code}
> (This is what the expression actually contains. The {{toSql()}} method lies
> and says that the statement is:
> {code:sql}
> SELECT CASE WHEN NULL IS NULL THEN id ELSE NULL END FROM alltypessmall
> {code}
> which causes confusion when debugging.)
> Expected:
> {code:sql}
> SELECT id FROM alltypessmall
> {code}
> Another case:
> {code:sql}
> CASE WHEN NULL + 1 THEN 10 ELSE 20 END
> {code}
> Ensure the {{NULL + 1}} is constant folded with a cast. Then, in {{CASE}}:
> {code:java}
> for (int i = loopStart; i < numChildren - 1; i += 2) {
> if (expr.getChild(i).isLiteral()) {
> canSimplify = true;
> break;
> }
> }
> {code}
> The {{isLiteral()}} method won’t match the cast and the simplification won’t
> fire.
> The reason is that there are multiple rewrites, one of which has a flaw.
> # Rewrite {{IFNULL to CASE}}
> # Rewrite {{NULL + 1}} to {{CAST(NULL AS SMALLINT)}}
> # Try to rewrite {{CAST(NULL AS SMALLINT) IS NULL}}, but fail because CAST is
> not a literal.
> In addition to the {{CASE}} issue above, the {{FoldConstantsRule}} itself is
> tripped up. It is supposed to simplify the {{... IS NULL}} expression above,
> but does not.
> The code in question in {{FoldConstantsRule.apply()}} is:
> {code:java}
> for (Expr child: expr.getChildren()) if (!child.isLiteral()) return expr;
> {code}
> In fact, this check is too restrictive. Need a new {{isLiteralLike()}} which
> should work like {{IsNullLiteral()}}:
> * True if the node is a literal.
> * True if the node is a cast of a literal.
> (Can't change {{isLiteral()}} since there are places that assume that this
> existing predicate indicates that the node is exactly a {{LiteralExpr}}.)
> Impala already has a predicate that does what is needed: {{isConstant()}}.
> However, the code in {{FoldConstantsRule.apply()}} specifically excludes
> calling it:
> {code:java}
> // Avoid calling Expr.isConstant() because that would lead to repeated
> traversals
> // of the Expr tree. Assumes the bottom-up application of this rule.
> Constant
> // children should have been folded at this point.
> {code}
> The new method solves the repeated traversal problem. With it, the test query
> now simplifies to the expected result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7822) Crash in repeat() constructing strings > 1GB
[
https://issues.apache.org/jira/browse/IMPALA-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-7822.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Crash in repeat() constructing strings > 1GB
>
>
> Key: IMPALA-7822
> URL: https://issues.apache.org/jira/browse/IMPALA-7822
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0,
> Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0,
> Impala 3.0, Impala 3.1.0
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: crash
> Fix For: Impala 3.1.0
>
>
> {noformat}
> select repeat('x', 1024 * 1024 * 1024 * 1024);
> {noformat}
> {noformat}
> #0 0x04551aa3 in
> google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) ()
> #1 0x7f72cdc0f362 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #2 0x7f72cdc138b9 in JVM_handle_linux_signal ()
>from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #3 0x7f72cdc06f78 in ?? () from
> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
> #4
> #5 0x7f72cab8a06c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x030e19da in impala::StringFunctions::Repeat (context=0x721f800,
> str=..., n=...)
> at be/src/exprs/string-functions-ir.cc:100
> #7 0x03145c32 in
> impala::ScalarFnCall::InterpretEval (
> this=0xaf2cb40, eval=0xd26ccc0, row=0x0) at
> be/src/exprs/scalar-fn-call.cc:485
> #8 0x0312bd85 in impala::ScalarFnCall::GetStringVal (this=0xaf2cb40,
> eval=0xd26ccc0,
> row=0x0) at be/src/exprs/scalar-fn-call.cc:599
> #9 0x030da116 in impala::ScalarExprEvaluator::GetValue
> (this=0xd26ccc0, expr=..., row=0x0)
> at be/src/exprs/scalar-expr-evaluator.cc:299
> #10 0x030d9da5 in impala::ScalarExprEvaluator::GetValue
> (this=0xd26ccc0, row=0x0)
> at be/src/exprs/scalar-expr-evaluator.cc:250
> #11 0x01fc59bc in
> Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (
> env=0xd25b1e0, caller_class=0x7f724d523280,
> thrift_expr_batch=0x7f724d523298,
> thrift_query_ctx_bytes=0x7f724d523290) at be/src/service/fe-support.cc:237
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IMPALA-7733) TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
[
https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-7733:
--
Target Version: Impala 3.2.0 (was: Impala 3.1.0)
> TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
> -
>
> Key: IMPALA-7733
> URL: https://issues.apache.org/jira/browse/IMPALA-7733
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Assignee: Tianyi Wang
>Priority: Blocker
> Labels: broken-build, flaky
>
> I see two examples in the past two months or so where this test fails due to
> a rename error on S3. The test's stacktrace looks like this:
> {noformat}
> query_test/test_insert_parquet.py:112: in test_insert_parquet
> self.run_test_case('insert_parquet', vector, unique_database,
> multiple_impalad=True)
> common/impala_test_suite.py:408: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:625: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:176: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:350: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:371: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Error(s) moving partition files. First error (of 1) was:
> Hdfs op (RENAME
> s3a:///test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> TO
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq)
> failed, error was:
> s3a:///test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed/.4e45cd68bcddd451-3c7156ed0002_803672621_dir/4e45cd68bcddd451-3c7156ed0002_448261088_data.0.parq
> E Error(5): Input/output error{noformat}
> Since we know this happens once in a while, some ideas to deflake it:
> * retry
> * check for this specific issue... if we think its platform flakiness, then
> we should skip it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7828) test_mem_leak() is flaky
[
https://issues.apache.org/jira/browse/IMPALA-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679473#comment-16679473
]
ASF subversion and git services commented on IMPALA-7828:
-
Commit d1aa1c009f62500ae2ee8cd915751b0d42bee911 in impala's branch
refs/heads/master from Michael Ho
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d1aa1c0 ]
IMPALA-7828: A temporary workaround for flaky UDF test (test_mem_leak())
Before IMPALA-4063, the error message detected during
FragmentInstanceState::Close() was always lost. After
IMPALA-4063, we may sometimes get the error message in
FragmentInstanceState::Close(). It's non-deterministic
as the fragment instance thread may race with the query
state thread which reports the final status. The UDF test
currently tries to handle this non-determinism by using
"row_regex:.*" in the ERRORS section but it doesn't
always seem to work.
This change workarounds the issue by commenting out the
ERRORS section in udf-no-expr-rewrite.text for now.
The actual fix will be done in IMPALA-7829.
Change-Id: I6a55d5ad1a5a7278e7390f60854a8df28c1b9f28
Reviewed-on: http://gerrit.cloudera.org:8080/11900
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> test_mem_leak() is flaky
>
>
> Key: IMPALA-7828
> URL: https://issues.apache.org/jira/browse/IMPALA-7828
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
>
> Before IMPALA-4063, the error message detected during
> {{FragmentInstanceState::Close()}} was always lost. After IMPALA-4063, we may
> sometimes get the error message in {{FragmentInstanceState::Close()}}. It's
> non-deterministic as the fragment instance thread may race with the query
> state thread which reports the final status. The test currently tries to
> handle this non-determinism by using "row_regex:.*" in the {{--ERRORS--}}
> section but it doesn't seem to work.
> Let's unbreak the test for now. Longer run, we need to move the point in
> which all fragment instances are done after
> {{FragmentInstanceState::Close()}} so things will become deterministic. This
> may have implication to query completion time so performance may need to be
> evaluated.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:313: in test_ir_functions
> self.run_test_case('QueryTest/udf-no-expr-rewrite', vector,
> use_db=unique_database)
> common/impala_test_suite.py:496: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:358: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:362: in verify_raw_results
> verify_errors(expected_errors, actual_errors)
> common/test_result_verifier.py:314: in verify_errors
> VERIFIER_MAP['VERIFY_IS_EQUAL'](expected, actual)
> common/test_result_verifier.py:271: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E row_regex:.* != None
> E Number of rows returned (expected vs actual): 1 != 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7818) Standardize use of Expr predicates
[
https://issues.apache.org/jira/browse/IMPALA-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679472#comment-16679472
]
ASF subversion and git services commented on IMPALA-7818:
-
Commit a3d0f30ba61b63cf227d7bf100ebd2176914e51f in impala's branch
refs/heads/master from [~paul-rogers]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=a3d0f30 ]
IMPALA-7818: Standardize use of Expr predicates
The Expr node provide two kinds of queries about node categories:
predicates and isMumble() functions. This change standardizes two
existing methods to use predicates instead.
First, the existing isLiteral() method is replaced by a new
IS_LITERAL predicate.
The key purpose is to clean up the confusing use of the existing
isNullLiteral() method, which actually is a check for either a null
literal or a cast of a null. To make this clearer, replaced this
method with a new IS_NULL_VALUE() predicate.
Added a new IS_NULL_LITERAL predicate for the case when a node must be
exactly the NULL literal, not a cast to NULL. Replaced instances of
foo instanceof NullLiteral with a use of the new IS_NULL_LITERAL
predicate. (This revealed bugs which will be addressed separately.)
Added an IS_NON_NULL_LITERAL to replace the two-method idiom used in
several places.
Tests: No functional change. Reran all tests to ensure nothing changed.
Change-Id: I09a089c0f2484246e5c6444b78990fa9260c036a
Reviewed-on: http://gerrit.cloudera.org:8080/11887
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Standardize use of Expr predicates
> --
>
> Key: IMPALA-7818
> URL: https://issues.apache.org/jira/browse/IMPALA-7818
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The {{Expr}} (expression) class in the frontend has many handy predicates
> such as {{IS_BINARY_PREDICATE}} and so on. (Note the confusing terminology:
> there are subclasses of {{Expr}} which are SQL predicates. The predicates
> discussed here are those based on the Guava {{Predicate}} class.)
> The class also has predicate-like methods: {{isLiteral()}} and
> {{isNullLiteral()}}. As it has evolved, {{isNullLiteral()}} checks not only
> for a null literal, but also a cast of a null to some type. These functions
> are in the base class, but do their work by checking the type of the
> expression.
> It would be cleaner to make these methods into predicates with names that
> follow their function: {{IS_LITERAL}}, and {{IS_NULL_VALUE}}.
> Further, there are a few places in the code that check for a non-null literal
> using an and of conditions, This would be cleaner using a new
> {{IS_NON_NULL_LITERAL}} predicate.
> These changes put us in position to add new predicates as a result of work in
> the pipeline; this refactoring is done as a separate change to keep the other
> commit smaller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7833) Audit and fix other string builtins for long string handling
Tim Armstrong created IMPALA-7833:
-
Summary: Audit and fix other string builtins for long string
handling
Key: IMPALA-7833
URL: https://issues.apache.org/jira/browse/IMPALA-7833
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.0, Impala 2.11.0, Impala 3.1.0
Reporter: Tim Armstrong
Following on from IMPALA-7822, there are some other string builtins that seem
to follow the same pattern of having a string size overflow an int passed into
the StringVal constructor. I think in some cases we get lucky and it works out,
but others it seems possible to crash given the right input values.
Here are some examples of cases where we can hit such bugs:
{noformat}
select lpad('foo', 17179869184 , ' ');
select rpad('foo', 17179869184 , ' ');
select space(17179869184 );
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
