[jira] [Resolved] (IMPALA-10337) DCHECK hit at SpillableRowBatchQueue when row size exceed max reservation
[
https://issues.apache.org/jira/browse/IMPALA-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riza Suminto resolved IMPALA-10337.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
> DCHECK hit at SpillableRowBatchQueue when row size exceed max reservation
> -
>
> Key: IMPALA-10337
> URL: https://issues.apache.org/jira/browse/IMPALA-10337
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
> Fix For: Impala 4.0
>
>
> While working on IMPALA-9856, I found that the following DCHECK in
> SpillableRowBatchQueue::AddBatch consistently hit when result spooling is
> enabled and row size is larger than resource_profile_.max_reservation,
> causing impalad to crash.
>
> [https://github.com/apache/impala/blob/eea617b/be/src/runtime/spillable-row-batch-queue.cc#L97]
> We can reproduce this issue by adding the following query options in
> query_test/test_insert.py::TestInsertQueries::test_insert_large_string
> {code:java}
> self.client.set_configuration_option("spool_query_results", "1")
> self.client.set_configuration_option("max_row_size", "257mb"){code}
> Additionally, setting max_result_spooling_mem to 512MB will increase
> resource_profile_.max_reservation to fit the large row and avoid this DCHECK.
> Instead of DCHECK, I think impalad should return error status, suggesting
> that user need to set larger max_result_spooling_mem.
> Another solution is to also consider max_row_size when computing
> maxMemReservationBytes in PlanRootSink.java.
>
> [https://github.com/apache/impala/blob/eea617b/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java#L74]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IMPALA-10337) DCHECK hit at SpillableRowBatchQueue when row size exceed max reservation
[
https://issues.apache.org/jira/browse/IMPALA-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riza Suminto resolved IMPALA-10337.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
> DCHECK hit at SpillableRowBatchQueue when row size exceed max reservation
> -
>
> Key: IMPALA-10337
> URL: https://issues.apache.org/jira/browse/IMPALA-10337
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
> Fix For: Impala 4.0
>
>
> While working on IMPALA-9856, I found that the following DCHECK in
> SpillableRowBatchQueue::AddBatch consistently hit when result spooling is
> enabled and row size is larger than resource_profile_.max_reservation,
> causing impalad to crash.
>
> [https://github.com/apache/impala/blob/eea617b/be/src/runtime/spillable-row-batch-queue.cc#L97]
> We can reproduce this issue by adding the following query options in
> query_test/test_insert.py::TestInsertQueries::test_insert_large_string
> {code:java}
> self.client.set_configuration_option("spool_query_results", "1")
> self.client.set_configuration_option("max_row_size", "257mb"){code}
> Additionally, setting max_result_spooling_mem to 512MB will increase
> resource_profile_.max_reservation to fit the large row and avoid this DCHECK.
> Instead of DCHECK, I think impalad should return error status, suggesting
> that user need to set larger max_result_spooling_mem.
> Another solution is to also consider max_row_size when computing
> maxMemReservationBytes in PlanRootSink.java.
>
> [https://github.com/apache/impala/blob/eea617b/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java#L74]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10337) DCHECK hit at SpillableRowBatchQueue when row size exceed max reservation
[
https://issues.apache.org/jira/browse/IMPALA-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244392#comment-17244392
]
ASF subversion and git services commented on IMPALA-10337:
--
Commit 2004a87edfa3a78e89623f395e8697047fe3c984 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2004a87 ]
IMPALA-10337: Consider MAX_ROW_SIZE when computing max reservation
PlanRootSink can fail silently if result spooling is enabled and
maxMemReservationBytes is less than 2 * MAX_ROW_SIZE. This happens
because results are spilled using a SpillableRowBatchQueue which needs 2
buffer (read and write) with at least MAX_ROW_SIZE bytes per buffer.
This patch fixes this by setting a lower bound of 2 * MAX_ROW_SIZE while
computing the min reservation for the PlanRootSink.
Testing:
- Pass exhaustive tests.
- Add e2e TestResultSpoolingMaxReservation.
- Lower MAX_ROW_SIZE on tests where MAX_RESULT_SPOOLING_MEM is set to
extremely low value. Also verify that PLAN_ROOT_SINK's ReservationLimit
remain unchanged after lowering the MAX_ROW_SIZE.
Change-Id: Id7138e1e034ea5d1cd15cf8de399690e52a9d726
Reviewed-on: http://gerrit.cloudera.org:8080/16765
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> DCHECK hit at SpillableRowBatchQueue when row size exceed max reservation
> -
>
> Key: IMPALA-10337
> URL: https://issues.apache.org/jira/browse/IMPALA-10337
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
>
> While working on IMPALA-9856, I found that the following DCHECK in
> SpillableRowBatchQueue::AddBatch consistently hit when result spooling is
> enabled and row size is larger than resource_profile_.max_reservation,
> causing impalad to crash.
>
> [https://github.com/apache/impala/blob/eea617b/be/src/runtime/spillable-row-batch-queue.cc#L97]
> We can reproduce this issue by adding the following query options in
> query_test/test_insert.py::TestInsertQueries::test_insert_large_string
> {code:java}
> self.client.set_configuration_option("spool_query_results", "1")
> self.client.set_configuration_option("max_row_size", "257mb"){code}
> Additionally, setting max_result_spooling_mem to 512MB will increase
> resource_profile_.max_reservation to fit the large row and avoid this DCHECK.
> Instead of DCHECK, I think impalad should return error status, suggesting
> that user need to set larger max_result_spooling_mem.
> Another solution is to also consider max_row_size when computing
> maxMemReservationBytes in PlanRootSink.java.
>
> [https://github.com/apache/impala/blob/eea617b/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java#L74]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10011) Building the Debian 8 container in the toolchain fails on an expired key
[
https://issues.apache.org/jira/browse/IMPALA-10011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244351#comment-17244351
]
Laszlo Gaal commented on IMPALA-10011:
--
I've just linked IMPALA-10378, which basically preempts it -- or that one can
be resolved as a duplicate and we can document the Won't Fix and the sunsetting
here. My apologies for the duplicate filing.
> Building the Debian 8 container in the toolchain fails on an expired key
>
>
> Key: IMPALA-10011
> URL: https://issues.apache.org/jira/browse/IMPALA-10011
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Laszlo Gaal
>Assignee: Laszlo Gaal
>Priority: Major
>
> Investigating and fixing IMPALA-9985 required a toolchain rebuild reaching
> down to rebuilding the Docker images used for the toolchian build (this
> happens using the Dockerfiles in the docker subdirectory of the
> native-toolchain repo).
> During the rebuild the Debian 8 container failed with this error:
> {code}
>
> GPG error: http://archive.debian.org jessie Release: The following signatures
> were invalid: KEYEXPIRED 1587841717
> .
> WARNING: The following packages cannot be authenticated!
> libgdbm3 libpopt0 libedit2 libgpm2 libkeyutils1 python-minimal mime-support
> [list of about 100 packages]
> [91mE: There are problems and -y was used without --force-yes
> [0mThe command '/bin/sh -c apt-get update && apt-get install -y autoconf
> bison build-essential ccache curl flex git
>groff-base libffi-dev libkrb5-dev libncurses5-dev
> libreadline-dev libsasl2-dev libssl-dev libtool-bin
> libz-dev lsb-release pigz pkg-config python-dev
> rsync texinfo unzip vim-common wget' returned a
> non-zero code: 100
> {code}
> Debian 8 is past its end-of-life, which happened on June 30th, 2020; the
> expiration of repo keys is probably one of the side effects of this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10378) Retire support for Debian 8
[
https://issues.apache.org/jira/browse/IMPALA-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244350#comment-17244350
]
Laszlo Gaal commented on IMPALA-10378:
--
I should have remembered the older ticket. This one preempts it I guess, so
IMPALA-10011 can be resolved as Won't Fix
> Retire support for Debian 8
> ---
>
> Key: IMPALA-10378
> URL: https://issues.apache.org/jira/browse/IMPALA-10378
> Project: IMPALA
> Issue Type: Task
>Reporter: Laszlo Gaal
>Priority: Major
>
> Debian 8 ("jessie") reached end-of-life on 2020-06-30; see the announcement
> at https://www.debian.org/News/2020/20200709.
> With no security updates provided, it is not practical for Impala to keep
> supporting Debian 8, so I propose retiring support for this OS version.
> This means that the native-toolchain project will not build (and publish)
> Debian 8 tarballs for new versions.
> This is also a workaround for the recent problems around repo signature
> verification seen for Debian 8 package repos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10295) Fix analytic limit pushdown when no predicates are present
[ https://issues.apache.org/jira/browse/IMPALA-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-10295. Fix Version/s: Impala 4.0 Resolution: Fixed > Fix analytic limit pushdown when no predicates are present > -- > > Key: IMPALA-10295 > URL: https://issues.apache.org/jira/browse/IMPALA-10295 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Blocker > Labels: correctness > Fix For: Impala 4.0 > > > This is to fix case 1 of the parent JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10295) Fix analytic limit pushdown when no predicates are present
[ https://issues.apache.org/jira/browse/IMPALA-10295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-10295. Fix Version/s: Impala 4.0 Resolution: Fixed > Fix analytic limit pushdown when no predicates are present > -- > > Key: IMPALA-10295 > URL: https://issues.apache.org/jira/browse/IMPALA-10295 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Affects Versions: Impala 4.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Blocker > Labels: correctness > Fix For: Impala 4.0 > > > This is to fix case 1 of the parent JIRA. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMPALA-6434) Add support to decode RLE_DICTIONARY encoded pages
[ https://issues.apache.org/jira/browse/IMPALA-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-6434: -- Labels: newbie parquet ramp-up (was: parquet) > Add support to decode RLE_DICTIONARY encoded pages > -- > > Key: IMPALA-6434 > URL: https://issues.apache.org/jira/browse/IMPALA-6434 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Lars Volker >Priority: Major > Labels: newbie, parquet, ramp-up > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244303#comment-17244303 ] junwen yang commented on IMPALA-10370: -- [~tarmstrong], thanks. We have updated the detected results as attached impala_thrift_incompatibility.txt-v2 by filtering the files you mentioned. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt, > impala_thrift_incompatibility.txt-v2.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junwen yang updated IMPALA-10370: - Attachment: impala_thrift_incompatibility.txt > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt, > impala_thrift_incompatibility.txt-v2.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junwen yang updated IMPALA-10370: - Attachment: impala_thrift_incompatibility.txt-v2.txt > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt, > impala_thrift_incompatibility.txt-v2.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junwen yang updated IMPALA-10370: - Attachment: (was: impala_thrift_incompatibility.txt) > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt, > impala_thrift_incompatibility.txt-v2.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9884) TestAdmissionControllerStress.test_mem_limit failing occasionally
[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244283#comment-17244283
]
Tim Armstrong commented on IMPALA-9884:
---
I tried just making the query run longer but ran into some different issues:
{noformat}
FAILURES
=
TestAdmissionControllerStress.test_mem_limit[num_queries: 30 | protocol:
beeswax | table_format: t
ext/none | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 5000, '
disable_codegen': False, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0} | submission_de
lay_ms: 150 | round_robin_submission: True]
tests/custom_cluster/test_admission_controller.py:1865: in test_mem_limit
{'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
tests/custom_cluster/test_admission_controller.py:1752: in run_admission_test
self.end_admitted_queries(num_to_end)
tests/custom_cluster/test_admission_controller.py:1531: in end_admitted_queries
assert (time() - start_time < STRESS_TIMEOUT),\
E AssertionError: Timed out waiting 120 seconds for query end
E assert (1607109115.855676 - 1607108995.685362) < 120
E+ where 1607109115.855676 = time()
{noformat}
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: impalad-executors.tar.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-06.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9884) TestAdmissionControllerStress.test_mem_limit failing occasionally
[
https://issues.apache.org/jira/browse/IMPALA-9884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-9884:
--
Priority: Critical (was: Blocker)
> TestAdmissionControllerStress.test_mem_limit failing occasionally
> -
>
> Key: IMPALA-9884
> URL: https://issues.apache.org/jira/browse/IMPALA-9884
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: impalad-executors.tar.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1925.vpc.cloudera.com.jenkins.log.INFO.20201017-06.23933.gz
>
>
> Recently, I saw this test failing with the exception trace below.
> {noformat}
> custom_cluster/test_admission_controller.py:1782: in test_mem_limit
> {'request_pool': self.pool_name, 'mem_limit': query_mem_limit})
> custom_cluster/test_admission_controller.py:1638: in run_admission_test
> assert metric_deltas['dequeued'] == 0,\
> E AssertionError: Queued queries should not run until others are made to
> finish
> E assert 1 == 0
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9930) Introduce new admission control rpc service
[ https://issues.apache.org/jira/browse/IMPALA-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-9930. Fix Version/s: Impala 4.0 Resolution: Fixed > Introduce new admission control rpc service > --- > > Key: IMPALA-9930 > URL: https://issues.apache.org/jira/browse/IMPALA-9930 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > Fix For: Impala 4.0 > > > In order to support the eventual single admission controller, we'll need to > create rpc definitions for the service. > The rpcs that will need to be defined are outlined in the design doc: > https://docs.google.com/document/d/1nLovzk4rwZLct2yl-eT7VMHFBP7LswcJp7RqiP59RAc/edit -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9930) Introduce new admission control rpc service
[ https://issues.apache.org/jira/browse/IMPALA-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-9930. Fix Version/s: Impala 4.0 Resolution: Fixed > Introduce new admission control rpc service > --- > > Key: IMPALA-9930 > URL: https://issues.apache.org/jira/browse/IMPALA-9930 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > Fix For: Impala 4.0 > > > In order to support the eventual single admission controller, we'll need to > create rpc definitions for the service. > The rpcs that will need to be defined are outlined in the design doc: > https://docs.google.com/document/d/1nLovzk4rwZLct2yl-eT7VMHFBP7LswcJp7RqiP59RAc/edit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-9453) S3 build failed with many strange symptoms
[
https://issues.apache.org/jira/browse/IMPALA-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9453.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
I'm going to close this. I think this has some pieces that are related to
dataload's s3guard interaction, but also other stuff. I think the fuzz failure
is IMPALA-9831.
If we see these symptoms again, we can open a new Jira.
> S3 build failed with many strange symptoms
> --
>
> Key: IMPALA-9453
> URL: https://issues.apache.org/jira/browse/IMPALA-9453
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Tim Armstrong
>Priority: Critical
> Labels: broken-build, crash
> Fix For: Not Applicable
>
> Attachments: impalad.ERROR, impalad_node1.ERROR, impalad_node2.ERROR
>
>
> There were a lot of incorrect results:
> {noformat}
> uery_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 7300 != 6990
> Stacktrace
> query_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 7300 != 6990
> Standard Error
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 7300 != 6990
> {noformat}
> The impalads eventually crashed:
> {noformat}
> F0302 00:50:55.607841 483 parquet-page-reader.cc:67]
> e24eb0839fa75423:8ac0bf730002] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a impala::HdfsParquetScanner::GetNextInternal()
> @ 0x2c8403e impala::HdfsParquetScanner::ProcessSplit()
> @ 0x28b826b impala::HdfsScanNode::ProcessSplit()
> @ 0x28b7440 impala::HdfsScanNode::ScannerThread()
> @ 0x28b679d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x28b8d91
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x20aa1e9 boost::function0<>::operator()()
> @ 0x266a84a impala::Thread::SuperviseThread()
> @ 0x2672ace boost::_bi::list5<>::operator()<>()
> @ 0x26729f2 boost::_bi::bind_t<>::operator()()
> @ 0x26729b5 boost::detail::thread_data<>::run()
> @ 0x3e98b19 thread_proxy
> @ 0x7f8c2ccefe24 start_thread
> @ 0x7f8c2985b34c __clone
> {noformat}
> {noformat}
> F0302 00:50:41.466794 32643 parquet-page-reader.cc:67]
> dd48a46583bea9c8:e3be6412] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee
[jira] [Resolved] (IMPALA-9453) S3 build failed with many strange symptoms
[
https://issues.apache.org/jira/browse/IMPALA-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9453.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
I'm going to close this. I think this has some pieces that are related to
dataload's s3guard interaction, but also other stuff. I think the fuzz failure
is IMPALA-9831.
If we see these symptoms again, we can open a new Jira.
> S3 build failed with many strange symptoms
> --
>
> Key: IMPALA-9453
> URL: https://issues.apache.org/jira/browse/IMPALA-9453
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Tim Armstrong
>Priority: Critical
> Labels: broken-build, crash
> Fix For: Not Applicable
>
> Attachments: impalad.ERROR, impalad_node1.ERROR, impalad_node2.ERROR
>
>
> There were a lot of incorrect results:
> {noformat}
> uery_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 7300 != 6990
> Stacktrace
> query_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 7300 != 6990
> Standard Error
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 7300 != 6990
> {noformat}
> The impalads eventually crashed:
> {noformat}
> F0302 00:50:55.607841 483 parquet-page-reader.cc:67]
> e24eb0839fa75423:8ac0bf730002] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a impala::HdfsParquetScanner::GetNextInternal()
> @ 0x2c8403e impala::HdfsParquetScanner::ProcessSplit()
> @ 0x28b826b impala::HdfsScanNode::ProcessSplit()
> @ 0x28b7440 impala::HdfsScanNode::ScannerThread()
> @ 0x28b679d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x28b8d91
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x20aa1e9 boost::function0<>::operator()()
> @ 0x266a84a impala::Thread::SuperviseThread()
> @ 0x2672ace boost::_bi::list5<>::operator()<>()
> @ 0x26729f2 boost::_bi::bind_t<>::operator()()
> @ 0x26729b5 boost::detail::thread_data<>::run()
> @ 0x3e98b19 thread_proxy
> @ 0x7f8c2ccefe24 start_thread
> @ 0x7f8c2985b34c __clone
> {noformat}
> {noformat}
> F0302 00:50:41.466794 32643 parquet-page-reader.cc:67]
> dd48a46583bea9c8:e3be6412] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee
[jira] [Resolved] (IMPALA-8581) Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where applicable
[ https://issues.apache.org/jira/browse/IMPALA-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8581. --- Fix Version/s: Not Applicable Resolution: Duplicate https://github.com/apache/impala/commit/ac87278b169422091af1c03fcd2101516372defb > Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where > applicable > -- > > Key: IMPALA-8581 > URL: https://issues.apache.org/jira/browse/IMPALA-8581 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Joe McDonnell >Priority: Minor > Labels: newbie > Fix For: Not Applicable > > > When doing a put or copyFromLocal via the HDFS commandline, it puts the file > into a temporary file and then moves it into place. The "-d" option skips the > temporary file and copies the file directly into place. On S3, this avoids an > extra copy and can also avoid consistency issues. We should look into using > the "-d" option across our tests. Our usages of the HDFS commandline > generally don't require atomicity of this file copy, so using "-d" is > unlikely to impact HDFS tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8581) Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where applicable
[ https://issues.apache.org/jira/browse/IMPALA-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8581. --- Fix Version/s: Not Applicable Resolution: Duplicate https://github.com/apache/impala/commit/ac87278b169422091af1c03fcd2101516372defb > Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where > applicable > -- > > Key: IMPALA-8581 > URL: https://issues.apache.org/jira/browse/IMPALA-8581 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Joe McDonnell >Priority: Minor > Labels: newbie > Fix For: Not Applicable > > > When doing a put or copyFromLocal via the HDFS commandline, it puts the file > into a temporary file and then moves it into place. The "-d" option skips the > temporary file and copies the file directly into place. On S3, this avoids an > extra copy and can also avoid consistency issues. We should look into using > the "-d" option across our tests. Our usages of the HDFS commandline > generally don't require atomicity of this file copy, so using "-d" is > unlikely to impact HDFS tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IMPALA-10378) Retire support for Debian 8
[
https://issues.apache.org/jira/browse/IMPALA-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244253#comment-17244253
]
Tim Armstrong commented on IMPALA-10378:
Makes sense
> Retire support for Debian 8
> ---
>
> Key: IMPALA-10378
> URL: https://issues.apache.org/jira/browse/IMPALA-10378
> Project: IMPALA
> Issue Type: Task
>Reporter: Laszlo Gaal
>Priority: Major
>
> Debian 8 ("jessie") reached end-of-life on 2020-06-30; see the announcement
> at https://www.debian.org/News/2020/20200709.
> With no security updates provided, it is not practical for Impala to keep
> supporting Debian 8, so I propose retiring support for this OS version.
> This means that the native-toolchain project will not build (and publish)
> Debian 8 tarballs for new versions.
> This is also a workaround for the recent problems around repo signature
> verification seen for Debian 8 package repos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10011) Building the Debian 8 container in the toolchain fails on an expired key
[
https://issues.apache.org/jira/browse/IMPALA-10011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244249#comment-17244249
]
Joe McDonnell commented on IMPALA-10011:
[~laszlog] What is left for this JIRA? I am comfortable removing all Debian 8
support. This is what we discussed upstream. Debian 8 can be maintained
independently if someone wishes to do so, but we shouldn't leave it in
native-toolchain.
> Building the Debian 8 container in the toolchain fails on an expired key
>
>
> Key: IMPALA-10011
> URL: https://issues.apache.org/jira/browse/IMPALA-10011
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Laszlo Gaal
>Assignee: Laszlo Gaal
>Priority: Major
>
> Investigating and fixing IMPALA-9985 required a toolchain rebuild reaching
> down to rebuilding the Docker images used for the toolchian build (this
> happens using the Dockerfiles in the docker subdirectory of the
> native-toolchain repo).
> During the rebuild the Debian 8 container failed with this error:
> {code}
>
> GPG error: http://archive.debian.org jessie Release: The following signatures
> were invalid: KEYEXPIRED 1587841717
> .
> WARNING: The following packages cannot be authenticated!
> libgdbm3 libpopt0 libedit2 libgpm2 libkeyutils1 python-minimal mime-support
> [list of about 100 packages]
> [91mE: There are problems and -y was used without --force-yes
> [0mThe command '/bin/sh -c apt-get update && apt-get install -y autoconf
> bison build-essential ccache curl flex git
>groff-base libffi-dev libkrb5-dev libncurses5-dev
> libreadline-dev libsasl2-dev libssl-dev libtool-bin
> libz-dev lsb-release pigz pkg-config python-dev
> rsync texinfo unzip vim-common wget' returned a
> non-zero code: 100
> {code}
> Debian 8 is past its end-of-life, which happened on June 30th, 2020; the
> expiration of repo keys is probably one of the side effects of this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9004) TestCompressedFormats is broken for text files
[
https://issues.apache.org/jira/browse/IMPALA-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9004.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
This was fixed as part of IMPALA-10005.
> TestCompressedFormats is broken for text files
> --
>
> Key: IMPALA-9004
> URL: https://issues.apache.org/jira/browse/IMPALA-9004
> Project: IMPALA
> Issue Type: Test
>Reporter: Sahil Takiar
>Priority: Major
> Fix For: Impala 4.0
>
>
> While working on IMPALA-8950, we made a fix to {{TestCompressedFormats}} so
> that it actually checks the exit status of the {{hdfs dfs -cp}} command,
> turns out that this command has been silently failing whenever
> {{test_compressed_formats}} runs with {{file_format}} = {{text}}.
> For some reason, data load writes compressed text files with their
> corresponding file compression suffix, but for compressed seq/rc files, it
> does not:
> {code:java}
> hdfs dfs -ls /test-warehouse/tinytable_seq_*
> Found 1 items
> -rwxr-xr-x 3 systest supergroup325 2019-08-22 14:32
> /test-warehouse/tinytable_seq_bzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup215 2019-08-22 14:32
> /test-warehouse/tinytable_seq_def/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup260 2019-08-22 14:32
> /test-warehouse/tinytable_seq_gzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup301 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_bzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup209 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_def/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup242 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_gzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup233 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_snap/00_0
> Found 2 items
> -rwxr-xr-x 3 systest supergroup243 2019-08-22 14:32
> /test-warehouse/tinytable_seq_snap/00_0
> hdfs dfs -ls /test-warehouse/tinytable_text_*
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 59 2019-08-22 14:32
> /test-warehouse/tinytable_text_bzip/00_0.bz2
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 28 2019-08-22 14:32
> /test-warehouse/tinytable_text_def/00_0.deflate
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 40 2019-08-22 14:32
> /test-warehouse/tinytable_text_gzip/00_0.gz
> Found 2 items
> -rwxr-xr-x 3 systest supergroup 87 2019-08-22 14:32
> /test-warehouse/tinytable_text_lzo/00_0.lzo
> -rw-r--r-- 3 systest supergroup 8 2019-08-22 14:42
> /test-warehouse/tinytable_text_lzo/00_0.lzo.index
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 41 2019-08-22 14:32
> /test-warehouse/tinytable_text_snap/00_0.snappy{code}
> Not sure if that is by design or not, but it is causing the tests to fail for
> all text files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IMPALA-9004) TestCompressedFormats is broken for text files
[
https://issues.apache.org/jira/browse/IMPALA-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9004.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
This was fixed as part of IMPALA-10005.
> TestCompressedFormats is broken for text files
> --
>
> Key: IMPALA-9004
> URL: https://issues.apache.org/jira/browse/IMPALA-9004
> Project: IMPALA
> Issue Type: Test
>Reporter: Sahil Takiar
>Priority: Major
> Fix For: Impala 4.0
>
>
> While working on IMPALA-8950, we made a fix to {{TestCompressedFormats}} so
> that it actually checks the exit status of the {{hdfs dfs -cp}} command,
> turns out that this command has been silently failing whenever
> {{test_compressed_formats}} runs with {{file_format}} = {{text}}.
> For some reason, data load writes compressed text files with their
> corresponding file compression suffix, but for compressed seq/rc files, it
> does not:
> {code:java}
> hdfs dfs -ls /test-warehouse/tinytable_seq_*
> Found 1 items
> -rwxr-xr-x 3 systest supergroup325 2019-08-22 14:32
> /test-warehouse/tinytable_seq_bzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup215 2019-08-22 14:32
> /test-warehouse/tinytable_seq_def/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup260 2019-08-22 14:32
> /test-warehouse/tinytable_seq_gzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup301 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_bzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup209 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_def/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup242 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_gzip/00_0
> Found 1 items
> -rwxr-xr-x 3 systest supergroup233 2019-08-22 14:32
> /test-warehouse/tinytable_seq_record_snap/00_0
> Found 2 items
> -rwxr-xr-x 3 systest supergroup243 2019-08-22 14:32
> /test-warehouse/tinytable_seq_snap/00_0
> hdfs dfs -ls /test-warehouse/tinytable_text_*
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 59 2019-08-22 14:32
> /test-warehouse/tinytable_text_bzip/00_0.bz2
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 28 2019-08-22 14:32
> /test-warehouse/tinytable_text_def/00_0.deflate
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 40 2019-08-22 14:32
> /test-warehouse/tinytable_text_gzip/00_0.gz
> Found 2 items
> -rwxr-xr-x 3 systest supergroup 87 2019-08-22 14:32
> /test-warehouse/tinytable_text_lzo/00_0.lzo
> -rw-r--r-- 3 systest supergroup 8 2019-08-22 14:42
> /test-warehouse/tinytable_text_lzo/00_0.lzo.index
> Found 1 items
> -rwxr-xr-x 3 systest supergroup 41 2019-08-22 14:32
> /test-warehouse/tinytable_text_snap/00_0.snappy{code}
> Not sure if that is by design or not, but it is causing the tests to fail for
> all text files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10378) Retire support for Debian 8
Laszlo Gaal created IMPALA-10378:
Summary: Retire support for Debian 8
Key: IMPALA-10378
URL: https://issues.apache.org/jira/browse/IMPALA-10378
Project: IMPALA
Issue Type: Task
Reporter: Laszlo Gaal
Debian 8 ("jessie") reached end-of-life on 2020-06-30; see the announcement at
https://www.debian.org/News/2020/20200709.
With no security updates provided, it is not practical for Impala to keep
supporting Debian 8, so I propose retiring support for this OS version.
This means that the native-toolchain project will not build (and publish)
Debian 8 tarballs for new versions.
This is also a workaround for the recent problems around repo signature
verification seen for Debian 8 package repos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10378) Retire support for Debian 8
Laszlo Gaal created IMPALA-10378:
Summary: Retire support for Debian 8
Key: IMPALA-10378
URL: https://issues.apache.org/jira/browse/IMPALA-10378
Project: IMPALA
Issue Type: Task
Reporter: Laszlo Gaal
Debian 8 ("jessie") reached end-of-life on 2020-06-30; see the announcement at
https://www.debian.org/News/2020/20200709.
With no security updates provided, it is not practical for Impala to keep
supporting Debian 8, so I propose retiring support for this OS version.
This means that the native-toolchain project will not build (and publish)
Debian 8 tarballs for new versions.
This is also a workaround for the recent problems around repo signature
verification seen for Debian 8 package repos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IMPALA-9112) Consider removing hdfsExists calls when writing files to S3
[
https://issues.apache.org/jira/browse/IMPALA-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9112.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
The consistency aspect of this will no longer apply. S3 now has strong
consistency: [https://aws.amazon.com/s3/consistency/]
I'm going to close this. We can consider performance aspects in a separate JIRA.
> Consider removing hdfsExists calls when writing files to S3
> ---
>
> Key: IMPALA-9112
> URL: https://issues.apache.org/jira/browse/IMPALA-9112
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: Not Applicable
>
>
> There are a few places in the backend where we call {{hdfsExists}} before
> writing out a file. This can cause issues when writing data to S3, because S3
> can cache 404 Not Found errors. This issue manifests itself with errors such
> as:
> {code:java}
> ERROR: Error(s) moving partition files. First error (of 1) was: Hdfs op
> (RENAME
> s3a://[bucket-name]/[table-name]/_impala_insert_staging/3943ae7ccf00711e_59606d88/.3943ae7ccf00711e-59606d88000b_562151879_dir/year=2015/3943ae7ccf00711e-59606d88000b_1994902389_data.0.parq
> TO
> s3a://[bucket-name]/[table-name]/3943ae7ccf00711e-59606d88000b_1994902389_data.0.parq)
> failed, error was:
> s3a://[bucket-name]/[table-name]/_impala_insert_staging/3943ae7ccf00711e_59606d88/.3943ae7ccf00711e-59606d88000b_562151879_dir/year=2015/3943ae7ccf00711e-59606d88000b_1994902389_data.0.parq
> Error(5): Input/output error
> Root cause: AmazonS3Exception: Not Found (Service: Amazon S3; Status Code:
> 404; Error Code: 404 Not Found; Request ID: []; S3 Extended Request ID:
> []){code}
> HADOOP-13884, HADOOP-13950, HADOOP-16490 - the HDFS clients allow specifying
> an "overwrite" option when creating a file; this can avoid doing any HEAD
> requests when opening a file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9112) Consider removing hdfsExists calls when writing files to S3
[
https://issues.apache.org/jira/browse/IMPALA-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9112.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
The consistency aspect of this will no longer apply. S3 now has strong
consistency: [https://aws.amazon.com/s3/consistency/]
I'm going to close this. We can consider performance aspects in a separate JIRA.
> Consider removing hdfsExists calls when writing files to S3
> ---
>
> Key: IMPALA-9112
> URL: https://issues.apache.org/jira/browse/IMPALA-9112
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: Not Applicable
>
>
> There are a few places in the backend where we call {{hdfsExists}} before
> writing out a file. This can cause issues when writing data to S3, because S3
> can cache 404 Not Found errors. This issue manifests itself with errors such
> as:
> {code:java}
> ERROR: Error(s) moving partition files. First error (of 1) was: Hdfs op
> (RENAME
> s3a://[bucket-name]/[table-name]/_impala_insert_staging/3943ae7ccf00711e_59606d88/.3943ae7ccf00711e-59606d88000b_562151879_dir/year=2015/3943ae7ccf00711e-59606d88000b_1994902389_data.0.parq
> TO
> s3a://[bucket-name]/[table-name]/3943ae7ccf00711e-59606d88000b_1994902389_data.0.parq)
> failed, error was:
> s3a://[bucket-name]/[table-name]/_impala_insert_staging/3943ae7ccf00711e_59606d88/.3943ae7ccf00711e-59606d88000b_562151879_dir/year=2015/3943ae7ccf00711e-59606d88000b_1994902389_data.0.parq
> Error(5): Input/output error
> Root cause: AmazonS3Exception: Not Found (Service: Amazon S3; Status Code:
> 404; Error Code: 404 Not Found; Request ID: []; S3 Extended Request ID:
> []){code}
> HADOOP-13884, HADOOP-13950, HADOOP-16490 - the HDFS clients allow specifying
> an "overwrite" option when creating a file; this can avoid doing any HEAD
> requests when opening a file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IMPALA-6331) slow s3 operations in unit tests
[
https://issues.apache.org/jira/browse/IMPALA-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-6331.
---
Fix Version/s: Not Applicable
Resolution: Cannot Reproduce
This has not reproduced anytime recently. This may have been related to some
tests running on an s3 bucket in a different region.
> slow s3 operations in unit tests
>
>
> Key: IMPALA-6331
> URL: https://issues.apache.org/jira/browse/IMPALA-6331
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Infrastructure, Perf Investigation
>Affects Versions: Impala 2.10.0
>Reporter: Vuk Ercegovac
>Priority: Major
> Fix For: Not Applicable
>
>
> Seeing some slowness for certain fs tests when run against S3. Since they in
> the hour range, I think its useful to track it down and distinguish between a
> test infrastructure issue vs. an impala/s3 issue.
> Here are logs from a recent run (note the time gaps in the log entries):
> 20:07:37
> query_test/test_hdfs_file_mods.py::TestHdfsFileMods::test_file_modifications[modification_type:
> move_file | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 20:08:19
> query_test/test_hdfs_file_mods.py::TestHdfsFileMods::test_file_modifications[modification_type:
> append | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 22:59:55
> query_test/test_insert.py::TestInsertQueries::test_insert[exec_option:
> {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 03:40:11
> query_test/test_insert.py::TestInsertQueries::test_insert[exec_option:
> {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 03:41:12
> query_test/test_insert.py::TestInsertQueries::test_insert_overwrite[exec_option:
> {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> There are several other tests that take in the 10 minute to minute range.
> Highlighting above the longest ones (> hour).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IMPALA-6331) slow s3 operations in unit tests
[
https://issues.apache.org/jira/browse/IMPALA-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-6331.
---
Fix Version/s: Not Applicable
Resolution: Cannot Reproduce
This has not reproduced anytime recently. This may have been related to some
tests running on an s3 bucket in a different region.
> slow s3 operations in unit tests
>
>
> Key: IMPALA-6331
> URL: https://issues.apache.org/jira/browse/IMPALA-6331
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Infrastructure, Perf Investigation
>Affects Versions: Impala 2.10.0
>Reporter: Vuk Ercegovac
>Priority: Major
> Fix For: Not Applicable
>
>
> Seeing some slowness for certain fs tests when run against S3. Since they in
> the hour range, I think its useful to track it down and distinguish between a
> test infrastructure issue vs. an impala/s3 issue.
> Here are logs from a recent run (note the time gaps in the log entries):
> 20:07:37
> query_test/test_hdfs_file_mods.py::TestHdfsFileMods::test_file_modifications[modification_type:
> move_file | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 20:08:19
> query_test/test_hdfs_file_mods.py::TestHdfsFileMods::test_file_modifications[modification_type:
> append | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 22:59:55
> query_test/test_insert.py::TestInsertQueries::test_insert[exec_option:
> {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 03:40:11
> query_test/test_insert.py::TestInsertQueries::test_insert[exec_option:
> {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> 03:41:12
> query_test/test_insert.py::TestInsertQueries::test_insert_overwrite[exec_option:
> {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none] PASSED
> There are several other tests that take in the 10 minute to minute range.
> Highlighting above the longest ones (> hour).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9759) Revisit integration of snapshot dataload with s3guard
[
https://issues.apache.org/jira/browse/IMPALA-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9759.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
S3 now has strong consistency, which will render S3Guard obsolete. No
improvements are planned for this codepath, so I'm closing this.
> Revisit integration of snapshot dataload with s3guard
> -
>
> Key: IMPALA-9759
> URL: https://issues.apache.org/jira/browse/IMPALA-9759
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Joe McDonnell
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Not Applicable
>
>
> Sometimes, the s3 jobs (which use s3guard for consistency) sees test failures
> due to missing files from the dataload snapshot (see bottom). This may be
> related to the interaction of snapshot loading with s3guard. We should nail
> down exactly the right procedure for loading the snapshot. Currently, we do
> the following:
> 1. Remove any data from the s3bucket via the s3 commandline
> 2. Create the s3guard dynamodb table (or reuse existing one if a previous job
> failed without deleting the old dynamodb table)
> 3. Prune any existing entries from that table
> 4. Load the snapshot to the s3 bucket
> In theory, this leave s3guard with an empty dynamodb table and an s3bucket
> with data. As tests progress and try to access the s3 bucket, s3guard would
> see that there is no entry in the dynamodb table and then check the
> underlying s3 bucket.
> We need to revisit these steps and verify that everything is being done
> correctly.
> {noformat}
> metadata/test_metadata_query_statements.py:70: in test_show_stats
> self.run_test_case('QueryTest/show-stats', vector, "functional")
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1
> == '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1
> E '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10
> == '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10
> E '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11
> == '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11
> E '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12
> == '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12
> E '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2
> == '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2
> E '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3
> == '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3
> E '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4
> == '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4
> E '2009','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5
> != '2009','5',0,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5
> E '2009','6',300,1,'19.71KB','NOT CACHED','NOT
>
[jira] [Resolved] (IMPALA-9759) Revisit integration of snapshot dataload with s3guard
[
https://issues.apache.org/jira/browse/IMPALA-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9759.
---
Fix Version/s: Not Applicable
Resolution: Won't Fix
S3 now has strong consistency, which will render S3Guard obsolete. No
improvements are planned for this codepath, so I'm closing this.
> Revisit integration of snapshot dataload with s3guard
> -
>
> Key: IMPALA-9759
> URL: https://issues.apache.org/jira/browse/IMPALA-9759
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Joe McDonnell
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Not Applicable
>
>
> Sometimes, the s3 jobs (which use s3guard for consistency) sees test failures
> due to missing files from the dataload snapshot (see bottom). This may be
> related to the interaction of snapshot loading with s3guard. We should nail
> down exactly the right procedure for loading the snapshot. Currently, we do
> the following:
> 1. Remove any data from the s3bucket via the s3 commandline
> 2. Create the s3guard dynamodb table (or reuse existing one if a previous job
> failed without deleting the old dynamodb table)
> 3. Prune any existing entries from that table
> 4. Load the snapshot to the s3 bucket
> In theory, this leave s3guard with an empty dynamodb table and an s3bucket
> with data. As tests progress and try to access the s3 bucket, s3guard would
> see that there is no entry in the dynamodb table and then check the
> underlying s3 bucket.
> We need to revisit these steps and verify that everything is being done
> correctly.
> {noformat}
> metadata/test_metadata_query_statements.py:70: in test_show_stats
> self.run_test_case('QueryTest/show-stats', vector, "functional")
> common/impala_test_suite.py:687: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1
> == '2009','1',310,1,'19.95KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=1
> E '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10
> == '2009','10',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=10
> E '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11
> == '2009','11',300,1,'19.71KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=11
> E '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12
> == '2009','12',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=12
> E '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2
> == '2009','2',280,1,'18.12KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=2
> E '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3
> == '2009','3',310,1,'20.06KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=3
> E '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4
> == '2009','4',300,1,'19.61KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=4
> E '2009','5',310,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5
> != '2009','5',0,1,'20.36KB','NOT CACHED','NOT
> CACHED','TEXT','false','s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=5
> E '2009','6',300,1,'19.71KB','NOT CACHED','NOT
>
[jira] [Resolved] (IMPALA-8311) Mitigate s3 consistency issues for test_scanners_fuzz
[ https://issues.apache.org/jira/browse/IMPALA-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8311. --- Resolution: Fixed All subtasks are complete. S3 now has strong consistency, so there is no future work here. > Mitigate s3 consistency issues for test_scanners_fuzz > - > > Key: IMPALA-8311 > URL: https://issues.apache.org/jira/browse/IMPALA-8311 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > Labels: s3 > > IMPALA-7804 tries to reduce the frequency of S3 inconsistency issues by: > a. reordering some operations to give more time to S3 to get consistent > b. by adding -f to -put calls, which removes checking whether there is > already a file with given path, making the PUT the first operation > tests/test_scanners_fuzz.py does -copyFromLocal calls without -f and -d, > which is probably the cause of some of the inconsistency issues -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8311) Mitigate s3 consistency issues for test_scanners_fuzz
[ https://issues.apache.org/jira/browse/IMPALA-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8311. --- Resolution: Fixed All subtasks are complete. S3 now has strong consistency, so there is no future work here. > Mitigate s3 consistency issues for test_scanners_fuzz > - > > Key: IMPALA-8311 > URL: https://issues.apache.org/jira/browse/IMPALA-8311 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > Labels: s3 > > IMPALA-7804 tries to reduce the frequency of S3 inconsistency issues by: > a. reordering some operations to give more time to S3 to get consistent > b. by adding -f to -put calls, which removes checking whether there is > already a file with given path, making the PUT the first operation > tests/test_scanners_fuzz.py does -copyFromLocal calls without -f and -d, > which is probably the cause of some of the inconsistency issues -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-8736) test_hive_parquet_codec_interop fails on S3
[ https://issues.apache.org/jira/browse/IMPALA-8736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8736. --- Fix Version/s: Impala 3.3.0 Resolution: Fixed > test_hive_parquet_codec_interop fails on S3 > --- > > Key: IMPALA-8736 > URL: https://issues.apache.org/jira/browse/IMPALA-8736 > Project: IMPALA > Issue Type: Test >Reporter: Abhishek Rawat >Priority: Major > Fix For: Impala 3.3.0 > > > The test case fails when trying to do hive inserts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-8736) test_hive_parquet_codec_interop fails on S3
[ https://issues.apache.org/jira/browse/IMPALA-8736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell resolved IMPALA-8736. --- Fix Version/s: Impala 3.3.0 Resolution: Fixed > test_hive_parquet_codec_interop fails on S3 > --- > > Key: IMPALA-8736 > URL: https://issues.apache.org/jira/browse/IMPALA-8736 > Project: IMPALA > Issue Type: Test >Reporter: Abhishek Rawat >Priority: Major > Fix For: Impala 3.3.0 > > > The test case fails when trying to do hive inserts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9058) S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
[
https://issues.apache.org/jira/browse/IMPALA-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9058.
---
Resolution: Won't Fix
We will be disabling S3guard for tests since S3 now has strong consistency -
https://aws.amazon.com/s3/consistency/
> S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
> --
>
> Key: IMPALA-9058
> URL: https://issues.apache.org/jira/browse/IMPALA-9058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> I've seen this happen several times now, S3 tests intermittently fail with an
> error such as:
> {code:java}
> Query aborted:InternalException: Error adding partitions E CAUSED BY:
> MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
> resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ResourceNotFoundException; Request ID:
> 8T9IS939MDI7ASOB0IJCC34J3NVV4KQNSO5AEMVJF66Q9ASUAAJG) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9453) S3 build failed with many strange symptoms
[
https://issues.apache.org/jira/browse/IMPALA-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244202#comment-17244202
]
Tim Armstrong commented on IMPALA-9453:
---
We may be able to close this - we will be disabling S3guard for tests since S3
now has strong consistency - https://aws.amazon.com/s3/consistency/
> S3 build failed with many strange symptoms
> --
>
> Key: IMPALA-9453
> URL: https://issues.apache.org/jira/browse/IMPALA-9453
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Tim Armstrong
>Priority: Critical
> Labels: broken-build, crash
> Attachments: impalad.ERROR, impalad_node1.ERROR, impalad_node2.ERROR
>
>
> There were a lot of incorrect results:
> {noformat}
> uery_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 7300 != 6990
> Stacktrace
> query_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 7300 != 6990
> Standard Error
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 7300 != 6990
> {noformat}
> The impalads eventually crashed:
> {noformat}
> F0302 00:50:55.607841 483 parquet-page-reader.cc:67]
> e24eb0839fa75423:8ac0bf730002] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a impala::HdfsParquetScanner::GetNextInternal()
> @ 0x2c8403e impala::HdfsParquetScanner::ProcessSplit()
> @ 0x28b826b impala::HdfsScanNode::ProcessSplit()
> @ 0x28b7440 impala::HdfsScanNode::ScannerThread()
> @ 0x28b679d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x28b8d91
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x20aa1e9 boost::function0<>::operator()()
> @ 0x266a84a impala::Thread::SuperviseThread()
> @ 0x2672ace boost::_bi::list5<>::operator()<>()
> @ 0x26729f2 boost::_bi::bind_t<>::operator()()
> @ 0x26729b5 boost::detail::thread_data<>::run()
> @ 0x3e98b19 thread_proxy
> @ 0x7f8c2ccefe24 start_thread
> @ 0x7f8c2985b34c __clone
> {noformat}
> {noformat}
> F0302 00:50:41.466794 32643 parquet-page-reader.cc:67]
> dd48a46583bea9c8:e3be6412] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239
[jira] [Resolved] (IMPALA-9058) S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
[
https://issues.apache.org/jira/browse/IMPALA-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-9058.
---
Resolution: Won't Fix
We will be disabling S3guard for tests since S3 now has strong consistency -
https://aws.amazon.com/s3/consistency/
> S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
> --
>
> Key: IMPALA-9058
> URL: https://issues.apache.org/jira/browse/IMPALA-9058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> I've seen this happen several times now, S3 tests intermittently fail with an
> error such as:
> {code:java}
> Query aborted:InternalException: Error adding partitions E CAUSED BY:
> MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
> resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ResourceNotFoundException; Request ID:
> 8T9IS939MDI7ASOB0IJCC34J3NVV4KQNSO5AEMVJF66Q9ASUAAJG) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IMPALA-9058) S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
[
https://issues.apache.org/jira/browse/IMPALA-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244204#comment-17244204
]
Tim Armstrong commented on IMPALA-9058:
---
We may be able to close this - we will be disabling S3guard for tests since S3
now has strong consistency - https://aws.amazon.com/s3/consistency/
> S3 tests failing with FileNotFoundException getVersionMarkerItem on ../VERSION
> --
>
> Key: IMPALA-9058
> URL: https://issues.apache.org/jira/browse/IMPALA-9058
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
>
> I've seen this happen several times now, S3 tests intermittently fail with an
> error such as:
> {code:java}
> Query aborted:InternalException: Error adding partitions E CAUSED BY:
> MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
> resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ResourceNotFoundException; Request ID:
> 8T9IS939MDI7ASOB0IJCC34J3NVV4KQNSO5AEMVJF66Q9ASUAAJG) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10251) test_decimal_queries and test_tpcds_queries may run out of memory on ASAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244198#comment-17244198
]
Tim Armstrong commented on IMPALA-10251:
I'm not going to have time to investigate. I think we'd need to solve this by
either increasing the memory on the test machines, reducing test parallelism
for ASAN, or enabling some kind of admission control for tests (like we did on
the upstream dockerized tests).
> test_decimal_queries and test_tpcds_queries may run out of memory on ASAN
> builds
>
>
> Key: IMPALA-10251
> URL: https://issues.apache.org/jira/browse/IMPALA-10251
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Critical
> Labels: broken-build, flaky
>
> Encountered these test failures due to running out of memory:
> {code:java}
> query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> beeswax | exec_option: {'disable_codegen_rows_threshold': 0,
> 'disable_codegen': 'false', 'decimal_v2': 'true', 'batch_size': 1} |
> table_format:
> orc/def/block]query_test.test_tpcds_queries.TestTpcdsQuery.test_tpcds_q51a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> hs2 | exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen':
> 'true', 'decimal_v2': 'true', 'batch_size': 1} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q18a[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q23_1[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q4[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] {code}
> The error is
> {code}
> query_test/test_decimal_queries.py:55: in test_queries
> self.run_test_case('QueryTest/decimal', vector)
> common/impala_test_suite.py:662: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:600: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:909: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Failed to get minimum memory reservation of 68.01 MB on
> daemon impala-ec2-centos74-r5-4xlarge-ondemand-109b.vpc.cloudera.com:27001
> for query 2b41e9326825c48d:15a4d931 due to following error: Failed to
> increase reservation by 68.01 MB because it would exceed the applicable
> reservation limit for the "Process" ReservationTracker:
> reservation_limit=10.20 GB reservation=10.16 GB used_reservation=0
> child_reservations=10.16 GB
> E The top 5 queries that allocated memory under this tracker are:
> E Query(ca4c47a813876f70:d626ca0c): Reservation=9.60 GB
> ReservationLimit=9.60 GB OtherMemory=115.84 MB Total=9.71 GB Peak=9.71 GB
> E Query(3f4e984e6b54204d:2597d87f): Reservation=426.77 MB
> ReservationLimit=9.60 GB OtherMemory=113.19 MB Total=539.95 MB Peak=540.00 MB
> E Query(6c4aca5cba7141b7:cb0d9a40): Limit=512.00 MB
> Reservation=68.03 MB ReservationLimit=409.60 MB OtherMemory=2.59 MB
> Total=70.62 MB Peak=71.93 MB
> E Query(e84c2ac37bac9309:7eddf8bb): Reservation=14.94 MB
> ReservationLimit=9.60 GB OtherMemory=5.55 MB Total=20.48 MB Peak=39.46 MB
> E Query(eb44550f4e6aebbd:4b72c396): Reservation=12.31 MB
> ReservationLimit=9.60 GB
[jira] [Assigned] (IMPALA-10251) test_decimal_queries and test_tpcds_queries may run out of memory on ASAN builds
[
https://issues.apache.org/jira/browse/IMPALA-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-10251:
--
Assignee: (was: Tim Armstrong)
> test_decimal_queries and test_tpcds_queries may run out of memory on ASAN
> builds
>
>
> Key: IMPALA-10251
> URL: https://issues.apache.org/jira/browse/IMPALA-10251
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Critical
> Labels: broken-build, flaky
>
> Encountered these test failures due to running out of memory:
> {code:java}
> query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> beeswax | exec_option: {'disable_codegen_rows_threshold': 0,
> 'disable_codegen': 'false', 'decimal_v2': 'true', 'batch_size': 1} |
> table_format:
> orc/def/block]query_test.test_tpcds_queries.TestTpcdsQuery.test_tpcds_q51a[protocol:
> beeswax | exec_option: {'decimal_v2': 0, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_decimal_queries.TestDecimalQueries.test_queries[protocol:
> hs2 | exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen':
> 'true', 'decimal_v2': 'true', 'batch_size': 1} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q18a[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q23_1[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]query_test.test_tpcds_queries.TestTpcdsDecimalV2Query.test_tpcds_q4[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] {code}
> The error is
> {code}
> query_test/test_decimal_queries.py:55: in test_queries
> self.run_test_case('QueryTest/decimal', vector)
> common/impala_test_suite.py:662: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:600: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:909: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Failed to get minimum memory reservation of 68.01 MB on
> daemon impala-ec2-centos74-r5-4xlarge-ondemand-109b.vpc.cloudera.com:27001
> for query 2b41e9326825c48d:15a4d931 due to following error: Failed to
> increase reservation by 68.01 MB because it would exceed the applicable
> reservation limit for the "Process" ReservationTracker:
> reservation_limit=10.20 GB reservation=10.16 GB used_reservation=0
> child_reservations=10.16 GB
> E The top 5 queries that allocated memory under this tracker are:
> E Query(ca4c47a813876f70:d626ca0c): Reservation=9.60 GB
> ReservationLimit=9.60 GB OtherMemory=115.84 MB Total=9.71 GB Peak=9.71 GB
> E Query(3f4e984e6b54204d:2597d87f): Reservation=426.77 MB
> ReservationLimit=9.60 GB OtherMemory=113.19 MB Total=539.95 MB Peak=540.00 MB
> E Query(6c4aca5cba7141b7:cb0d9a40): Limit=512.00 MB
> Reservation=68.03 MB ReservationLimit=409.60 MB OtherMemory=2.59 MB
> Total=70.62 MB Peak=71.93 MB
> E Query(e84c2ac37bac9309:7eddf8bb): Reservation=14.94 MB
> ReservationLimit=9.60 GB OtherMemory=5.55 MB Total=20.48 MB Peak=39.46 MB
> E Query(eb44550f4e6aebbd:4b72c396): Reservation=12.31 MB
> ReservationLimit=9.60 GB OtherMemory=5.85 MB Total=18.16 MB Peak=18.16 MB
> E
> E
> E
> E
> E
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {code}
--
This message was sent by Atlassian Jira
[jira] [Assigned] (IMPALA-10344) Error loading functional_orc_def.alltypesaggmultifiles in hive
[
https://issues.apache.org/jira/browse/IMPALA-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong reassigned IMPALA-10344:
--
Assignee: (was: Tim Armstrong)
> Error loading functional_orc_def.alltypesaggmultifiles in hive
> --
>
> Key: IMPALA-10344
> URL: https://issues.apache.org/jira/browse/IMPALA-10344
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 4.0
>Reporter: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: hive-metastore.out, hive-server2.out,
> load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log
>
>
> Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> Exception when loading 11 partitions in table alltypesaggmultifiles
> {noformat}
> 0: jdbc:hive2://localhost:11050/default> insert into table
> functional_orc_def.al
> ltypesaggmultifiles partition (year, month, day) SELECT id, bool_col,
> tinyint_co
> l, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col,
> st
> ring_col, timestamp_col, year, month, day FROM
> functional.alltypesaggmultifiles
> where id % 4 = 1;
> going to print operations logs
> printed operations logs
> going to print operations logs
> INFO : Compiling
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
> insert into table functional_orc_def.alltypesaggmultifiles partition (year,
> month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month, day FROM functional.alltypesaggmultifiles where
> id % 4 = 1
> INFO : No Stats for functional@alltypesaggmultifiles, Columns: float_col,
> int_col, string_col, bool_col, date_string_col, smallint_col, timestamp_col,
> tinyint_col, bigint_col, id, double_col
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:id,
> type:int, comment:null), FieldSchema(name:bool_col, type:boolean,
> comment:null), FieldSchema(name:tinyint_col, type:tinyint, comment:null),
> FieldSchema(name:smallint_col, type:smallint, comment:null),
> FieldSchema(name:int_col, type:int, comment:null),
> FieldSchema(name:bigint_col, type:bigint, comment:null),
> FieldSchema(name:float_col, type:float, comment:null),
> FieldSchema(name:double_col, type:double, comment:null),
> FieldSchema(name:date_string_col, type:string, comment:null),
> FieldSchema(name:string_col, type:string, comment:null),
> FieldSchema(name:timestamp_col, type:timestamp, comment:null),
> FieldSchema(name:year, type:int, comment:null), FieldSchema(name:month,
> type:int, comment:null), FieldSchema(name:day, type:int, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9);
> Time taken: 0.036 seconds
> INFO : Executing
> command(queryId=jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9):
> insert into table functional_orc_def.alltypesaggmultifiles partition (year,
> month, day) SELECT id, bool_col, tinyint_col, smallint_col, int_col,
> bigint_col, float_col, double_col, date_string_col, string_col,
> timestamp_col, year, month, day FROM functional.alltypesaggmultifiles where
> id % 4 = 1
> INFO : Query ID = jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> jenkins_20201113135913_d2ecc422-ec51-4fe2-9e92-e8d93b01aca9
> INFO : Session is already open
> INFO : Dag name: insert into table functional_orc_def.all...1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1605304119768_0012)
> printed operations logs
> [2K--
> [2K[36;1mVERTICES MODESTATUS TOTAL COMPLETED
> RUNNING PENDING FAILED KILLED
> [22;0m[2K--
> [2KMap 1 .. container SUCCEEDED 42 420
> 0 0 0
> Reducer 2container RUNNING 1 010
> 0 0
> [2K--
> [2K[31;1mVERTICES: 01/02 [=>>-] 97% ELAPSED
> TIME: 1.46 s
> [22;0m[2K--
>
[jira] [Resolved] (IMPALA-10235) Averaged timer profile counters can be negative for trivial queries
[
https://issues.apache.org/jira/browse/IMPALA-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10235.
Fix Version/s: Impala 4.0
Assignee: Tim Armstrong
Resolution: Fixed
This was introduced by part 1 of IMPALA-9382 and fixed by this part of the
below patch.
@@ -1842,7 +2137,8 @@ RuntimeProfileBase::AveragedCounter::GetStats() const {
if (!vals.empty()) {
sort(vals.begin(), vals.end());
result.num_vals = vals.size();
-result.mean = std::accumulate(vals.begin(), vals.end(), 0) /
result.num_vals;
+T sum = std::accumulate(vals.begin(), vals.end(), 0L);
+result.mean = sum / result.num_vals;
result.min = vals[0];
result.max = vals.back();
int end_idx = vals.size() - 1;
@@ -1902,6 +2198,50 @@ void
RuntimeProfileBase::AveragedCounter::PrettyPrintImpl(
commit 9429bd779de986d3e61858bef7e258bd73a2cacd
Author: Tim Armstrong
Date: Sun May 17 16:37:46 2020 -0700
IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
This reworks the status reporting so that serialized
AggregatedRuntimeProfile objects are sent from executors
to coordinators. These profiles are substantially denser
and faster to process for higher mt_dop values. The aggregation
is also done in a single step, merging the aggregated thrift
profile from the executor directly into the final aggregated
profile, instead of converting it to an unaggregated profile
first.
The changes required were:
* A new Update() method for AggregatedRuntimeProfile that
updates the profile from a serialised AggregateRuntimeProfile
for a subset of the instances. The code is generalized from the
existing InitFromThrift() code path.
* Per-fragment reports included in the status report protobuf
when --gen_experimental_profile=true.
* Logic on the coordinator that either consumes serialized
AggregatedRuntimeProfile per fragment, when
--gen_experimental_profile=true, or consumes a serialized
RuntimeProfile per finstance otherwise.
This also adds support for event sequences and time series
in the aggregated profile, so the amount of information
in the aggregated profile is now on par with the basic profile.
We also finish off support for JSON profile. The JSON profile is
more stripped down because we do not need to round-trip profiles
via JSON and it is a much less dense profile representation.
Part 3 will clean up and improve the display of the profile.
Testing:
* Add sanity tests for aggregated runtime profile.
* Add unit tests to exercise aggregation of the various counter types
* Ran core tests.
Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca
Reviewed-on: http://gerrit.cloudera.org:8080/16057
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Averaged timer profile counters can be negative for trivial queries
> ---
>
> Key: IMPALA-10235
> URL: https://issues.apache.org/jira/browse/IMPALA-10235
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Major
> Labels: newbie, ramp-up
> Fix For: Impala 4.0
>
> Attachments: profile-output.txt
>
>
> Steps to reproduce on master:
> {code:java}
> stakiar @ stakiar-desktop -bash ~/Impala 2020-10-13 11:13:02 master
> [74] → ./bin/impala-shell.sh -q "select sleep(100) from functional.alltypes
> limit 25" -p > profile-output.txt
> ...
> Query: select sleep(100) from functional.alltypes limit 25
> Query submitted at: 2020-10-13 11:13:07 (Coordinator:
> http://stakiar-desktop:25000)
> Query progress can be monitored at:
> http://stakiar-desktop:25000/query_plan?query_id=694f94671571d4d1:cdec9db9
> Fetched 25 row(s) in 2.64s
> {code}
> Attached the contents of {{profile-output.txt}}
> Relevant portion of the profile:
> {code:java}
> Averaged Fragment F00:(Total: 2s603ms, non-child: 272.519us, % non-child:
> 0.01%)
> ...
>- CompletionTime: -1665218428.000ns
> ...
>- TotalThreadsTotalWallClockTime: -1686005515.000ns
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 2.151ms
> ...
>- TotalTime: -1691524485.000ns
> {code}
> For whatever reason, this only affects the averaged fragment profile. For
> this query, there was only one coordinator fragment and thus only one
> fragment instance. The coordinator fragment instance showed normal timer
> values:
> {code:java}
> Coordinator Fragment F00:
> ...
> - CompletionTime: 2s629ms
> ...
> - TotalThreadsTotalWallClockTime: 2s608ms
>- TotalThreadsSysTime: 0.000ns
>-
[jira] [Resolved] (IMPALA-10235) Averaged timer profile counters can be negative for trivial queries
[
https://issues.apache.org/jira/browse/IMPALA-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10235.
Fix Version/s: Impala 4.0
Assignee: Tim Armstrong
Resolution: Fixed
This was introduced by part 1 of IMPALA-9382 and fixed by this part of the
below patch.
@@ -1842,7 +2137,8 @@ RuntimeProfileBase::AveragedCounter::GetStats() const {
if (!vals.empty()) {
sort(vals.begin(), vals.end());
result.num_vals = vals.size();
-result.mean = std::accumulate(vals.begin(), vals.end(), 0) /
result.num_vals;
+T sum = std::accumulate(vals.begin(), vals.end(), 0L);
+result.mean = sum / result.num_vals;
result.min = vals[0];
result.max = vals.back();
int end_idx = vals.size() - 1;
@@ -1902,6 +2198,50 @@ void
RuntimeProfileBase::AveragedCounter::PrettyPrintImpl(
commit 9429bd779de986d3e61858bef7e258bd73a2cacd
Author: Tim Armstrong
Date: Sun May 17 16:37:46 2020 -0700
IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
This reworks the status reporting so that serialized
AggregatedRuntimeProfile objects are sent from executors
to coordinators. These profiles are substantially denser
and faster to process for higher mt_dop values. The aggregation
is also done in a single step, merging the aggregated thrift
profile from the executor directly into the final aggregated
profile, instead of converting it to an unaggregated profile
first.
The changes required were:
* A new Update() method for AggregatedRuntimeProfile that
updates the profile from a serialised AggregateRuntimeProfile
for a subset of the instances. The code is generalized from the
existing InitFromThrift() code path.
* Per-fragment reports included in the status report protobuf
when --gen_experimental_profile=true.
* Logic on the coordinator that either consumes serialized
AggregatedRuntimeProfile per fragment, when
--gen_experimental_profile=true, or consumes a serialized
RuntimeProfile per finstance otherwise.
This also adds support for event sequences and time series
in the aggregated profile, so the amount of information
in the aggregated profile is now on par with the basic profile.
We also finish off support for JSON profile. The JSON profile is
more stripped down because we do not need to round-trip profiles
via JSON and it is a much less dense profile representation.
Part 3 will clean up and improve the display of the profile.
Testing:
* Add sanity tests for aggregated runtime profile.
* Add unit tests to exercise aggregation of the various counter types
* Ran core tests.
Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca
Reviewed-on: http://gerrit.cloudera.org:8080/16057
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Averaged timer profile counters can be negative for trivial queries
> ---
>
> Key: IMPALA-10235
> URL: https://issues.apache.org/jira/browse/IMPALA-10235
> Project: IMPALA
> Issue Type: Bug
>Reporter: Sahil Takiar
>Assignee: Tim Armstrong
>Priority: Major
> Labels: newbie, ramp-up
> Fix For: Impala 4.0
>
> Attachments: profile-output.txt
>
>
> Steps to reproduce on master:
> {code:java}
> stakiar @ stakiar-desktop -bash ~/Impala 2020-10-13 11:13:02 master
> [74] → ./bin/impala-shell.sh -q "select sleep(100) from functional.alltypes
> limit 25" -p > profile-output.txt
> ...
> Query: select sleep(100) from functional.alltypes limit 25
> Query submitted at: 2020-10-13 11:13:07 (Coordinator:
> http://stakiar-desktop:25000)
> Query progress can be monitored at:
> http://stakiar-desktop:25000/query_plan?query_id=694f94671571d4d1:cdec9db9
> Fetched 25 row(s) in 2.64s
> {code}
> Attached the contents of {{profile-output.txt}}
> Relevant portion of the profile:
> {code:java}
> Averaged Fragment F00:(Total: 2s603ms, non-child: 272.519us, % non-child:
> 0.01%)
> ...
>- CompletionTime: -1665218428.000ns
> ...
>- TotalThreadsTotalWallClockTime: -1686005515.000ns
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 2.151ms
> ...
>- TotalTime: -1691524485.000ns
> {code}
> For whatever reason, this only affects the averaged fragment profile. For
> this query, there was only one coordinator fragment and thus only one
> fragment instance. The coordinator fragment instance showed normal timer
> values:
> {code:java}
> Coordinator Fragment F00:
> ...
> - CompletionTime: 2s629ms
> ...
> - TotalThreadsTotalWallClockTime: 2s608ms
>- TotalThreadsSysTime: 0.000ns
>-
[jira] [Resolved] (IMPALA-6870) SummaryStatsCounter should be included in averaged profile
[ https://issues.apache.org/jira/browse/IMPALA-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6870. --- Fix Version/s: Impala 4.0 Resolution: Fixed commit 9429bd779de986d3e61858bef7e258bd73a2cacd Author: Tim Armstrong Date: Sun May 17 16:37:46 2020 -0700 IMPALA-9382: part 2/3: aggregate profiles sent to coordinator This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The aggregation is also done in a single step, merging the aggregated thrift profile from the executor directly into the final aggregated profile, instead of converting it to an unaggregated profile first. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. Part 3 will clean up and improve the display of the profile. Testing: * Add sanity tests for aggregated runtime profile. * Add unit tests to exercise aggregation of the various counter types * Ran core tests. Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Reviewed-on: http://gerrit.cloudera.org:8080/16057 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > SummaryStatsCounter should be included in averaged profile > -- > > Key: IMPALA-6870 > URL: https://issues.apache.org/jira/browse/IMPALA-6870 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.12.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: observability > Fix For: Impala 4.0 > > > Summary stats like FooterProcessingTime don't show up in the averaged > fragment. We should be able to merge these counters to produce overall > statistics: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6870) SummaryStatsCounter should be included in averaged profile
[ https://issues.apache.org/jira/browse/IMPALA-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6870. --- Fix Version/s: Impala 4.0 Resolution: Fixed commit 9429bd779de986d3e61858bef7e258bd73a2cacd Author: Tim Armstrong Date: Sun May 17 16:37:46 2020 -0700 IMPALA-9382: part 2/3: aggregate profiles sent to coordinator This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The aggregation is also done in a single step, merging the aggregated thrift profile from the executor directly into the final aggregated profile, instead of converting it to an unaggregated profile first. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. Part 3 will clean up and improve the display of the profile. Testing: * Add sanity tests for aggregated runtime profile. * Add unit tests to exercise aggregation of the various counter types * Ran core tests. Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Reviewed-on: http://gerrit.cloudera.org:8080/16057 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > SummaryStatsCounter should be included in averaged profile > -- > > Key: IMPALA-6870 > URL: https://issues.apache.org/jira/browse/IMPALA-6870 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.12.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: observability > Fix For: Impala 4.0 > > > Summary stats like FooterProcessingTime don't show up in the averaged > fragment. We should be able to merge these counters to produce overall > statistics: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-10377: --- Target Version: Impala 4.0 > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10156) test_unmatched_schema flaky with wrong results
[
https://issues.apache.org/jira/browse/IMPALA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10156.
Fix Version/s: Impala 4.0
Resolution: Fixed
> test_unmatched_schema flaky with wrong results
> --
>
> Key: IMPALA-10156
> URL: https://issues.apache.org/jira/browse/IMPALA-10156
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> {noformat}
> Error Message
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 ==
> 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 !=
> 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 !=
> 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 !=
> 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 !=
> 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None !=
> 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None !=
> 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None
> != 1106,'Name16',94612,5000 E None != 1106,'Name16',94612,5000 E None
> != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,15000 E
> None != 1106,'Name16',94616,5000 E None != 1106,'Name16',94616,5000 E
> None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,15000 E
> None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94612,5000 E
> None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,15000 E
> None != 1106,'Name6',94616,5000 E None != 1106,'Name6',94616,5000 E
> Number of rows returned (expected vs actual): 19 != 38
> Stacktrace
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000
> E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000
> E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000
> E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000
> E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,5000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,15000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,5000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,15000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,5000 !=
[jira] [Resolved] (IMPALA-10156) test_unmatched_schema flaky with wrong results
[
https://issues.apache.org/jira/browse/IMPALA-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10156.
Fix Version/s: Impala 4.0
Resolution: Fixed
> test_unmatched_schema flaky with wrong results
> --
>
> Key: IMPALA-10156
> URL: https://issues.apache.org/jira/browse/IMPALA-10156
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Thomas Tauber-Marshall
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> {noformat}
> Error Message
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 ==
> 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 !=
> 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 !=
> 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 !=
> 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 !=
> 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 !=
> 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 !=
> 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 !=
> 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 !=
> 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 !=
> 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 !=
> 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 !=
> 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None !=
> 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None !=
> 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None
> != 1106,'Name16',94612,5000 E None != 1106,'Name16',94612,5000 E None
> != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,15000 E
> None != 1106,'Name16',94616,5000 E None != 1106,'Name16',94616,5000 E
> None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,15000 E
> None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94612,5000 E
> None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,15000 E
> None != 1106,'Name6',94616,5000 E None != 1106,'Name6',94616,5000 E
> Number of rows returned (expected vs actual): 19 != 38
> Stacktrace
> query_test/test_scanners.py:244: in test_unmatched_schema
> self.run_test_case('QueryTest/test-unmatched-schema', vector)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000
> E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000
> E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000
> E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000
> E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000
> E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000
> E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000
> E 1006,'Name6',94616,5000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,15000 != 1006,'Name16',94612,15000
> E 1106,'Name16',94612,5000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,15000 != 1006,'Name16',94612,5000
> E 1106,'Name16',94616,5000 !=
[jira] [Updated] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong updated IMPALA-10377: --- Fix Version/s: (was: Impala 4.0) > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244104#comment-17244104 ] Tim Armstrong commented on IMPALA-10370: [~zagol] you can exclude BackendGflags.thrift, CatalogInternalService.thrift, CatalogService.thrift, DataSinks.thrift, Data.thrift, Descriptors.thrift, Exprs.thrift, Frontend.thrift, ImpalaInternalService.thrift, JniCatalog.thrift, Logging.thrift, NetworkTest.thrift, Partitions.thrift, Planner.thrift, PlanNodes.thrift, ResourceProfile.thrift, Results.thrift, SqlConstraints.thrift, StatestoreService.thrift, Zip.thrift Those should only be used internally. Some files like Types.thrift have definitions that are used both internally and externally, but I think if you exclude the above files it will avoid most false positives. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244097#comment-17244097 ] Tim Armstrong commented on IMPALA-10377: [~liuyao] thanks for investigating all these issues, it would be great to make these more accurate. We need to be a bit careful if the estimates change a lot because it can disrupt production workloads. Look forward to seeing a patch. > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Fix For: Impala 4.0 > > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10377) Improve the accuracy of resource estimation
[ https://issues.apache.org/jira/browse/IMPALA-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244093#comment-17244093 ] Tim Armstrong commented on IMPALA-10377: I had some ideas about improving accuracy of estimates for the preaggregations, I'll link those here. > Improve the accuracy of resource estimation > --- > > Key: IMPALA-10377 > URL: https://issues.apache.org/jira/browse/IMPALA-10377 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.4.0 >Reporter: liuyao >Assignee: liuyao >Priority: Major > Labels: estimate, memory, statistics > Fix For: Impala 4.0 > > Original Estimate: 120h > Remaining Estimate: 120h > > PlanNode does not consider some factors when estimating memory, this will > cause a large error rate > > AggregationNode > > 1.The memory occupied by hash table's own data structure is not considered. > Hash table inserts a new value, which will add a bucket. The size of a bucket > is 16 bytes. > 2.When estimating the NDV of merge aggregation, if there are multiple > grouping exprs, it may be divided by the number of Fragment Instances several > times, and it should be divided only once. > 3.When estimating the NDV of merge aggregation, and there are multiple > grouping exprs, the estimated memory is much smaller than the actual use. > 4.If there is no grouping exprs, the estimated memory is much larger than the > actual use. > 5.If the NDV of grouping exprs is very small, the estimated memory is much > larger than the actual use. > > SortNode > 1.Estimate the memory usage of external sort. the estimated memory is much > smaller than the actual use. > > > HashJoinNode > 1.The memory occupied by hash table's own data structure is not > considered.Hash Table will keep duplicate data, so the size of DuplicateNode > should be considered. > 2.Hash table will create multiple buckets in advance. The size of these > buckets should be considered. > > KuduScanNode > 1.Estimate memory by scanning all columns,the estimated memory is much larger > than the actual use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8883) Update statistics of ACID tables during writes
[ https://issues.apache.org/jira/browse/IMPALA-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges reassigned IMPALA-8883: Assignee: (was: Attila Jeges) > Update statistics of ACID tables during writes > -- > > Key: IMPALA-8883 > URL: https://issues.apache.org/jira/browse/IMPALA-8883 > Project: IMPALA > Issue Type: Improvement >Reporter: Zoltán Borók-Nagy >Priority: Major > Labels: impala-acid > > When Impala INSERTs or TRUNCATEs an ACID table it simply removes the > COLUMN_STATS_ACCURATE property to invalidate the statistics in order to > prevent Hive using it. > Instead of it Impala should properly update the statistics. It should be > relatively simple for TRUNCATE since it erases all the data, but a bit more > complicated for INSERT, e.g.: > * Properly update _number of distinct values_ > * INSERT OVERWRITE partition should properly update table level _number of > rows_. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8883) Update statistics of ACID tables during writes
[ https://issues.apache.org/jira/browse/IMPALA-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243971#comment-17243971 ] Attila Jeges commented on IMPALA-8883: -- I'm not sure if this is still relevant, but I won't have time to work on this anytime soon. Unassigning. > Update statistics of ACID tables during writes > -- > > Key: IMPALA-8883 > URL: https://issues.apache.org/jira/browse/IMPALA-8883 > Project: IMPALA > Issue Type: Improvement >Reporter: Zoltán Borók-Nagy >Assignee: Attila Jeges >Priority: Major > Labels: impala-acid > > When Impala INSERTs or TRUNCATEs an ACID table it simply removes the > COLUMN_STATS_ACCURATE property to invalidate the statistics in order to > prevent Hive using it. > Instead of it Impala should properly update the statistics. It should be > relatively simple for TRUNCATE since it erases all the data, but a bit more > complicated for INSERT, e.g.: > * Properly update _number of distinct values_ > * INSERT OVERWRITE partition should properly update table level _number of > rows_. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10224) Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC
[ https://issues.apache.org/jira/browse/IMPALA-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges resolved IMPALA-10224. --- Resolution: Fixed > Add startup flag not to expose debug web url via > PingImpalaService/PingImpalaHS2Service RPC > --- > > Key: IMPALA-10224 > URL: https://issues.apache.org/jira/browse/IMPALA-10224 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > > PingImpalaService/PingImpalaHS2Service RPC calls expose the coordinator's > debug web url to clients like impala shell. Since the debug web UI is not > something that end-users will necessarily have access to, we should have a > server option to send an empty string instead of the real url to the impala > client signalling that the debug web ui is not available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10224) Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC
[ https://issues.apache.org/jira/browse/IMPALA-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges resolved IMPALA-10224. --- Resolution: Fixed > Add startup flag not to expose debug web url via > PingImpalaService/PingImpalaHS2Service RPC > --- > > Key: IMPALA-10224 > URL: https://issues.apache.org/jira/browse/IMPALA-10224 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > > PingImpalaService/PingImpalaHS2Service RPC calls expose the coordinator's > debug web url to clients like impala shell. Since the debug web UI is not > something that end-users will necessarily have access to, we should have a > server option to send an empty string instead of the real url to the impala > client signalling that the debug web ui is not available. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IMPALA-8827) Create tests for Hive stats interop in ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges reassigned IMPALA-8827: Assignee: (was: Attila Jeges) > Create tests for Hive stats interop in ACID tables > -- > > Key: IMPALA-8827 > URL: https://issues.apache.org/jira/browse/IMPALA-8827 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Csaba Ringhofer >Priority: Critical > Labels: impala-acid > > Hive started to use stats to answer some queries in ACID tables. Some tests > should check what happens if Hive uses stats computed by Impala. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8827) Create tests for Hive stats interop in ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243961#comment-17243961 ] Attila Jeges edited comment on IMPALA-8827 at 12/4/20, 12:01 PM: - Unassigned it as I won't be able to work on this anytime soon. was (Author: attilaj): Unassigned it as I won't be able to work on this > Create tests for Hive stats interop in ACID tables > -- > > Key: IMPALA-8827 > URL: https://issues.apache.org/jira/browse/IMPALA-8827 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Csaba Ringhofer >Priority: Critical > Labels: impala-acid > > Hive started to use stats to answer some queries in ACID tables. Some tests > should check what happens if Hive uses stats computed by Impala. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8827) Create tests for Hive stats interop in ACID tables
[ https://issues.apache.org/jira/browse/IMPALA-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243961#comment-17243961 ] Attila Jeges commented on IMPALA-8827: -- Unassigned it as I won't be able to work on this > Create tests for Hive stats interop in ACID tables > -- > > Key: IMPALA-8827 > URL: https://issues.apache.org/jira/browse/IMPALA-8827 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Csaba Ringhofer >Priority: Critical > Labels: impala-acid > > Hive started to use stats to answer some queries in ACID tables. Some tests > should check what happens if Hive uses stats computed by Impala. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10376) Data loading of a functional-query ORC table fails with "Fail to get checksum"
[
https://issues.apache.org/jira/browse/IMPALA-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243901#comment-17243901
]
Zoltán Borók-Nagy commented on IMPALA-10376:
I'll bump the CDP build number once we have both HIVE-23825 and HIVE-24145 in
CDP Hive. These should fix these errors for good.
> Data loading of a functional-query ORC table fails with "Fail to get checksum"
> --
>
> Key: IMPALA-10376
> URL: https://issues.apache.org/jira/browse/IMPALA-10376
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: Aman Sinha
>Assignee: Zoltán Borók-Nagy
>Priority: Major
> Labels: broken-build, flaky
>
> The symptoms here are very similar to IMPALA-9923 (please see that JIRA for
> full context) except that this occurs for functional-query ORC table as shown
> below (seen in
> https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-erasure-coding/632/):
> {noformat}
> load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log:ERROR :
> FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to
> get checksum, since file /test-
> warehouse/managed/chars_medium_orc_def/base_001/_orc_acid_version is
> under construction.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10376) Data loading of a functional-query ORC table fails with "Fail to get checksum"
[
https://issues.apache.org/jira/browse/IMPALA-10376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy reassigned IMPALA-10376:
--
Assignee: Zoltán Borók-Nagy
> Data loading of a functional-query ORC table fails with "Fail to get checksum"
> --
>
> Key: IMPALA-10376
> URL: https://issues.apache.org/jira/browse/IMPALA-10376
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.4.0
>Reporter: Aman Sinha
>Assignee: Zoltán Borók-Nagy
>Priority: Major
> Labels: broken-build, flaky
>
> The symptoms here are very similar to IMPALA-9923 (please see that JIRA for
> full context) except that this occurs for functional-query ORC table as shown
> below (seen in
> https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-erasure-coding/632/):
> {noformat}
> load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log:ERROR :
> FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to
> get checksum, since file /test-
> warehouse/managed/chars_medium_orc_def/base_001/_orc_acid_version is
> under construction.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-9923) Data loading of TPC-DS ORC fails with "Fail to get checksum"
[
https://issues.apache.org/jira/browse/IMPALA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243896#comment-17243896
]
Zoltán Borók-Nagy edited comment on IMPALA-9923 at 12/4/20, 9:44 AM:
-
Thanks, Aman. It's interesting that it's always the orc_acid_version file that
causes this problem. HIVE-23825 turns off the generation of these files by
default, hopefully it'll resolve this issue. Also, HIVE-24145 should eliminate
the root cause. Neither of these are available in the current ASF builds, but
hopefully the next CDP build number bump will include them.
was (Author: boroknagyz):
Thanks, Aman. It's interesting that it's always the orc_acid_version file that
causes this problem. HIVE-23825 turns off the generation of these files by
default, hopefully it'll resolve this issue. Also, HIVE-24145 should eliminate
the root cause. Neither of these are available in the current ASF builds, but
hopefully the next CDP build number bump will include those.
> Data loading of TPC-DS ORC fails with "Fail to get checksum"
>
>
> Key: IMPALA-9923
> URL: https://issues.apache.org/jira/browse/IMPALA-9923
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Zoltán Borók-Nagy
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
> Attachments: load-tpcds-core-hive-generated-orc-def-block.sql,
> load-tpcds-core-hive-generated-orc-def-block.sql.log
>
>
> {noformat}
> INFO : Loading data to table tpcds_orc_def.store_sales partition
> (ss_sold_date_sk=null) from
> hdfs://localhost:20500/test-warehouse/managed/tpcds.store_sales_orc_def
> INFO :
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
> checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> INFO : Completed executing
> command(queryId=ubuntu_20200707055650_a1958916-1e85-4db5-b1bc-cc63d80b3537);
> Time taken: 14.512 seconds
> INFO : OK
> Error: Error while compiling statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to
> get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction. (state=08S01,code=1)
> java.sql.SQLException: Error while compiling statement: FAILED: Execution
> Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.io.IOException: Fail to get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1007)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
> at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1329)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1127)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11223/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9923) Data loading of TPC-DS ORC fails with "Fail to get checksum"
[
https://issues.apache.org/jira/browse/IMPALA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243896#comment-17243896
]
Zoltán Borók-Nagy commented on IMPALA-9923:
---
Thanks, Aman. It's interesting that it's always the orc_acid_version file that
causes this problem. HIVE-23825 turns off the generation of these files by
default, hopefully it'll resolve this issue. Also, HIVE-24145 should eliminate
the root cause. Neither of these are available in the current ASF builds, but
hopefully the next CDP build number bump will include those.
> Data loading of TPC-DS ORC fails with "Fail to get checksum"
>
>
> Key: IMPALA-9923
> URL: https://issues.apache.org/jira/browse/IMPALA-9923
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Zoltán Borók-Nagy
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
> Attachments: load-tpcds-core-hive-generated-orc-def-block.sql,
> load-tpcds-core-hive-generated-orc-def-block.sql.log
>
>
> {noformat}
> INFO : Loading data to table tpcds_orc_def.store_sales partition
> (ss_sold_date_sk=null) from
> hdfs://localhost:20500/test-warehouse/managed/tpcds.store_sales_orc_def
> INFO :
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
> checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> INFO : Completed executing
> command(queryId=ubuntu_20200707055650_a1958916-1e85-4db5-b1bc-cc63d80b3537);
> Time taken: 14.512 seconds
> INFO : OK
> Error: Error while compiling statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to
> get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction. (state=08S01,code=1)
> java.sql.SQLException: Error while compiling statement: FAILED: Execution
> Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.io.IOException: Fail to get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1007)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
> at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1329)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1127)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11223/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243882#comment-17243882 ] Yongle Zhang commented on IMPALA-10370: --- Hi [~tarmstrong], Thanks for the response. Is there a way that we can exclude the thrift definitions for the internal communications you mentioned, but only keep the result for those that matter (e.g., the TAggregateFunction/TFunction serialized to hive store)? Maybe our checker would provide more helpful results that way. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
