[jira] [Commented] (IMPALA-9549) Impalad startup fails to wait for catalogd to startup when using local catalog
[
https://issues.apache.org/jira/browse/IMPALA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068622#comment-17068622
]
ASF subversion and git services commented on IMPALA-9549:
-
Commit 1d63348b933b266f63d76b06eecbdf636cb45770 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1d63348 ]
IMPALA-9549: Handle catalogd startup delays when using local catalog
Impalads should be tolerant of delays in catalogd startup.
Currently, when running with the local catalog
(use_local_catalog=true), impalad startup can fail when catalogd
startup is delayed. What happens is that ImpalaServer's constructor
calls ImpalaServer::UpdateCatalogMetrics(), which maintains two
metrics counting the number of tables and databases. This is before
the code in ImpalaServer::Start() that waits for the catalogd to
start (added by IMPALA-4704), so there is no guarantee that catalogd
is running. The UpdateCatalogMetrics() call ends up calling getDbs()
in the frontend catalog. LocalCatalog::getDbs() tries to load the
databases (and thus contact catalogd), and this call will fail if
catalogd is not running. This fails startup.
use_local_catalog=false is immune to this only because it does not
contact catalogd in Catalog::getDbs().
This moves the UpdateCatalogMetrics() call from the ImpalaServer
constructor to ImpalaServer::Start() after the impalad has already
waited for the catalogd to start up. It also limits the call to
run only in coordinators.
Prior to this change, when using local catalog, the executors would
have catalog.num-databases and catalog.num-tables set to the right
values at startup. These values would not be kept up to date.
With this change, the executors do not have these values set.
Without local catalog, both before and after this change, executors
do not have accurate counts for catalog.num-databases or
catalog.num-tables.
Testing:
- Added a test to custom_cluster.test_catalog_wait to delay catalogd
start up by 60 seconds and verify that the impalads successfully
start up. This test fails prior to this change.
- Hand tested to verify that the metrics that are maintained by
UpdateCatalogMetrics() are not meaningfully changed for coordinators
and that executors do not have metrics set.
Change-Id: I1b5a94c59f25927a169dcb58f310ce6b1044
Reviewed-on: http://gerrit.cloudera.org:8080/15561
Reviewed-by: Vihang Karajgaonkar
Tested-by: Impala Public Jenkins
> Impalad startup fails to wait for catalogd to startup when using local catalog
> --
>
> Key: IMPALA-9549
> URL: https://issues.apache.org/jira/browse/IMPALA-9549
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Critical
>
> Since Impala coordinators and executors may be starting up at the same time
> as the catalogd, they should be tolerant of delays in the catalogd starting
> up. When using local catalog (use_local_catalog=true), the Impalads fail with
> the following error if the catalogd startup is delayed:
> {noformat}
> I0323 14:22:03.151849 29565 jni-util.cc:288]
> org.apache.impala.catalog.local.LocalCatalogException: Unable to load
> database names
> I0323 14:22:03.151849 29565 jni-util.cc:288]
> org.apache.impala.catalog.local.LocalCatalogException: Unable to load
> database names
> at org.apache.impala.catalog.local.LocalCatalog.loadDbs(LocalCatalog.java:94)
> at org.apache.impala.catalog.local.LocalCatalog.getDbs(LocalCatalog.java:83)
> at org.apache.impala.service.Frontend.getCatalogMetrics(Frontend.java:753)
> at
> org.apache.impala.service.JniFrontend.getCatalogMetrics(JniFrontend.java:220)
> Caused by: org.apache.thrift.TException:
> org.apache.impala.common.InternalException: Couldn't open transport for
> localhost:26000 (connect() failed: Connection refused)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:382)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.access$100(CatalogdMetaProvider.java:174)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider$1.call(CatalogdMetaProvider.java:583)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider$1.call(CatalogdMetaProvider.java:578)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:509)
> at
> org.apache.impala.catalog.local.CatalogdMetaProvider.loadDbList(CatalogdMetaProvider.java:577)
> at org.apache.impala.catalog.local.LocalCatalog.loadDbs(LocalCatalog.java:92)
> ... 3 more
> Caused by: org.apache.impala.common.InternalException: Couldn't open
> transport for localhost:26000 (connect()
[jira] [Commented] (IMPALA-8870) Bump guava version when building against Hive 3
[
https://issues.apache.org/jira/browse/IMPALA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068624#comment-17068624
]
ASF subversion and git services commented on IMPALA-8870:
-
Commit 0167c5b4242fcebf6be19aba5ecfb440204278ad in impala's branch
refs/heads/master from Fang-Yu Rao
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0167c5b ]
IMPALA-8870: Bump up Guava to 28.1-jre and set DISABLE_SENTRY to true
This patch bumps up the version of Guava libraries from 14.0.1 to
28.1-jre. Due to some changes in Guava's API's, we modify the call
sites accordingly.
Moreover, in order to instruct the Java classes under the directory of
$IMPALA_HOME/common/yarn-extras to use the new Guava libraries, we
explicitly added a dependency in the corresponding pom.xml file.
On the other hand, we set DISABLE_SENTRY to true regardless of
$USE_CDP_HIVE since Sentry's Guava version has not been bumped up yet
and thus run-sentry-service.sh cannot be successfully executed. Recall
that by setting DISABLE_SENTRY to true we also disable every
Sentry-related test, which is fine from now on since Impala 3.4 was
recently branched. The plan is to drop support for Sentry in the Impala
4 line.
Change-Id: I9690a926953a8d3c3872277680b4be0551546c68
Reviewed-on: http://gerrit.cloudera.org:8080/15214
Tested-by: Impala Public Jenkins
Reviewed-by: Csaba Ringhofer
> Bump guava version when building against Hive 3
> ---
>
> Key: IMPALA-8870
> URL: https://issues.apache.org/jira/browse/IMPALA-8870
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Fang-Yu Rao
>Priority: Blocker
>
> Guava is pinned to 14.01
> https://github.com/apache/impala/blob/8094811/impala-parent/pom.xml#L59
> {code}
>
> 14.0.1
> {code}
> I think this has likely changed in Hive 3 and we probably want to revisit
> this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4704) ImpalaD should not open 21000 and 21050 Ports till Catalog is Received
[
https://issues.apache.org/jira/browse/IMPALA-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068623#comment-17068623
]
ASF subversion and git services commented on IMPALA-4704:
-
Commit 1d63348b933b266f63d76b06eecbdf636cb45770 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1d63348 ]
IMPALA-9549: Handle catalogd startup delays when using local catalog
Impalads should be tolerant of delays in catalogd startup.
Currently, when running with the local catalog
(use_local_catalog=true), impalad startup can fail when catalogd
startup is delayed. What happens is that ImpalaServer's constructor
calls ImpalaServer::UpdateCatalogMetrics(), which maintains two
metrics counting the number of tables and databases. This is before
the code in ImpalaServer::Start() that waits for the catalogd to
start (added by IMPALA-4704), so there is no guarantee that catalogd
is running. The UpdateCatalogMetrics() call ends up calling getDbs()
in the frontend catalog. LocalCatalog::getDbs() tries to load the
databases (and thus contact catalogd), and this call will fail if
catalogd is not running. This fails startup.
use_local_catalog=false is immune to this only because it does not
contact catalogd in Catalog::getDbs().
This moves the UpdateCatalogMetrics() call from the ImpalaServer
constructor to ImpalaServer::Start() after the impalad has already
waited for the catalogd to start up. It also limits the call to
run only in coordinators.
Prior to this change, when using local catalog, the executors would
have catalog.num-databases and catalog.num-tables set to the right
values at startup. These values would not be kept up to date.
With this change, the executors do not have these values set.
Without local catalog, both before and after this change, executors
do not have accurate counts for catalog.num-databases or
catalog.num-tables.
Testing:
- Added a test to custom_cluster.test_catalog_wait to delay catalogd
start up by 60 seconds and verify that the impalads successfully
start up. This test fails prior to this change.
- Hand tested to verify that the metrics that are maintained by
UpdateCatalogMetrics() are not meaningfully changed for coordinators
and that executors do not have metrics set.
Change-Id: I1b5a94c59f25927a169dcb58f310ce6b1044
Reviewed-on: http://gerrit.cloudera.org:8080/15561
Reviewed-by: Vihang Karajgaonkar
Tested-by: Impala Public Jenkins
> ImpalaD should not open 21000 and 21050 Ports till Catalog is Received
> --
>
> Key: IMPALA-4704
> URL: https://issues.apache.org/jira/browse/IMPALA-4704
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0
>Reporter: Manish Maheshwari
>Assignee: Vuk Ercegovac
>Priority: Major
> Labels: impala, ramp-up
> Fix For: Impala 2.11.0
>
>
> Currently ImpalaD's open the frontend connections and this results in query
> failures. The preferred behaviour would be that the ports remain closed till
> the catalog is received and for any reason is the SS connectivity is not
> established after reasonable attempts and timeouts, then the impalad to
> simply shut down.
> {code}
> impalad.INFO:I1216 17:39:40.437333 10463 jni-util.cc:166]
> com.cloudera.impala.common.AnalysisException: This Impala daemon is not ready
> to accept user requests. Status: Waiting for catalog update from the
> StateStore.
> impalad.INFO:I1216 17:39:40.438743 10463 status.cc:112] AnalysisException:
> This Impala daemon is not ready to accept user requests. Status: Waiting for
> catalog update from the StateStore.
> impalad.INFO:I1216 17:39:40.918184 10464 jni-util.cc:166]
> com.cloudera.impala.common.AnalysisException: This Impala daemon is not ready
> to accept user requests. Status: Waiting for catalog update from the
> StateStore.
> impalad.INFO:I1216 17:39:40.918994 10464 status.cc:112] AnalysisException:
> This Impala daemon is not ready to accept user requests. Status: Waiting for
> catalog update from the StateStore.
> impalad.INFO:I1216 17:39:44.129482 10465 jni-util.cc:166]
> com.cloudera.impala.common.AnalysisException: This Impala daemon is not ready
> to accept user requests. Status: Waiting for catalog update from the
> StateStore.
> {code}
> This will help especially when we have multiple impala'd behind a LB and
> connections can be directed to daemons with the catalog when some
> servers/impala services are been restarted for any reason.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Updated] (IMPALA-9500) Allow returning complex types from a subselect
[
https://issues.apache.org/jira/browse/IMPALA-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Kaszab updated IMPALA-9500:
-
Description:
Once the rest of the tasks are implemented from the same epic there is a chance
that there won't be anything to do with this issue.
One sample query:
{code:java}
select id, nested_struct.a, a.item
from (
select id * 100 as id,
nested_struct,
int_array
from
functional_parquet.complextypestbl
) t, t.int_array a
{code}
was:Once the rest of the tasks are implemented from the same epic there is a
chance that there won't be anything to do with this issue.
> Allow returning complex types from a subselect
> --
>
> Key: IMPALA-9500
> URL: https://issues.apache.org/jira/browse/IMPALA-9500
> Project: IMPALA
> Issue Type: New Feature
>Reporter: Gabor Kaszab
>Priority: Major
> Labels: complextype
>
> Once the rest of the tasks are implemented from the same epic there is a
> chance that there won't be anything to do with this issue.
> One sample query:
> {code:java}
> select id, nested_struct.a, a.item
> from (
> select id * 100 as id,
> nested_struct,
> int_array
> from
> functional_parquet.complextypestbl
> ) t, t.int_array a
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9557) Implement to_json() for complex types
Gabor Kaszab created IMPALA-9557: Summary: Implement to_json() for complex types Key: IMPALA-9557 URL: https://issues.apache.org/jira/browse/IMPALA-9557 Project: IMPALA Issue Type: New Feature Reporter: Gabor Kaszab This built-in function should accept complex types as parameter and return a string containing that particular complex type in Json format. Check Hive for how each complex type looks like in Json. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9557) Implement to_json() for complex types
[ https://issues.apache.org/jira/browse/IMPALA-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Kaszab updated IMPALA-9557: - Labels: complextype (was: ) > Implement to_json() for complex types > - > > Key: IMPALA-9557 > URL: https://issues.apache.org/jira/browse/IMPALA-9557 > Project: IMPALA > Issue Type: New Feature >Reporter: Gabor Kaszab >Priority: Major > Labels: complextype > > This built-in function should accept complex types as parameter and return a > string containing that particular complex type in Json format. Check Hive for > how each complex type looks like in Json. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9558) Consider adding an m2 mountpoint to the Docker-based test infra
Laszlo Gaal created IMPALA-9558: --- Summary: Consider adding an m2 mountpoint to the Docker-based test infra Key: IMPALA-9558 URL: https://issues.apache.org/jira/browse/IMPALA-9558 Project: IMPALA Issue Type: Improvement Reporter: Laszlo Gaal This is similar to the ccache-directory mount point: the goal would be to accelerate subsequent Docker-based builds on the same host by keeping downloaded Java artifacts in a cache that can be shared between Docker runs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9559) Implement constructors for complex types
Gabor Kaszab created IMPALA-9559:
Summary: Implement constructors for complex types
Key: IMPALA-9559
URL: https://issues.apache.org/jira/browse/IMPALA-9559
Project: IMPALA
Issue Type: New Feature
Components: Backend, Frontend
Reporter: Gabor Kaszab
even if we can't write them to a file it makes sense to play around with
queries as:
{code:java}
WITH T as (SELECT ARRAY(1,2) A, STRUCT(1 as S1, 2 as S2) S) SELECT T.S.S1 from
T;
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8870) Bump guava version when building against Hive 3
[
https://issues.apache.org/jira/browse/IMPALA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fang-Yu Rao resolved IMPALA-8870.
-
Fix Version/s: Impala 4.0
Resolution: Fixed
> Bump guava version when building against Hive 3
> ---
>
> Key: IMPALA-8870
> URL: https://issues.apache.org/jira/browse/IMPALA-8870
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Fang-Yu Rao
>Priority: Blocker
> Fix For: Impala 4.0
>
>
> Guava is pinned to 14.01
> https://github.com/apache/impala/blob/8094811/impala-parent/pom.xml#L59
> {code}
>
> 14.0.1
> {code}
> I think this has likely changed in Hive 3 and we probably want to revisit
> this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9560) Changing version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE breaks TestStatsExtrapolation
Joe McDonnell created IMPALA-9560:
-
Summary: Changing version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE
breaks TestStatsExtrapolation
Key: IMPALA-9560
URL: https://issues.apache.org/jira/browse/IMPALA-9560
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.4.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell
When working on the Impala 3.4 release, we changed the version on branch-3.4.0
from 3.4.0-SNAPSHOT to 3.4.0-RELEASE.
metadata/test_stats_extrapolation.py::TestStatsExtrapolation::test_stats_extrapolation()
now fails with the following error:
{noformat}
metadata/test_stats_extrapolation.py:44: in test_stats_extrapolation
self.run_test_case('QueryTest/stats-extrapolation', vector, unique_database)
common/impala_test_suite.py:690: in run_test_case
self.__verify_results_and_errors(vector, test_section, result, use_db)
common/impala_test_suite.py:523: in __verify_results_and_errors
replace_filenames_with_placeholder)
common/test_result_verifier.py:456: in verify_raw_results
VERIFIER_MAP[verifier](expected, actual)
common/test_result_verifier.py:246: in verify_query_result_is_subset
assert expected_literal_strings <= actual_literal_strings
E assert Items in expected results not found in actual results:
E ' tuple-ids=0 row-size=4B cardinality=17.91K'
E Items in actual results:
E '| output exprs: id'
E ''
E ' table: rows=unavailable size=unavailable'
E ' stored statistics:'
E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=2'
E ' columns: unavailable'
E ' partitions: 0/24 rows=unavailable'
E '00:SCAN HDFS [test_stats_extrapolation_5c6bdfd.alltypes]'
E ' tuple-ids=0 row-size=4B cardinality=17.90K'
E '|'
E 'Analyzed query: SELECT id FROM test_stats_extrapolation_5c6bdfd.alltypes'
E 'F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1'
E ' HDFS partitions=24/24 files=36 size=281.43KB'
E 'test_stats_extrapolation_5c6bdfd.alltypes'
E 'PLAN-ROOT SINK'
E '| mem-estimate=0B mem-reservation=0B thread-reservation=0'
E '| Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB
thread-reservation=2'
E ' in pipelines: 00(GETNEXT)'
E ' extrapolated-rows=unavailable max-scan-range-rows=unavailable'
E 'Per-Host Resource Estimates: Memory=16MB'
E 'WARNING: The following tables are missing relevant table and/or column
statistics.'
E ' mem-estimate=16.00MB mem-reservation=8.00KB
thread-reservation=1'{noformat}
The output is expecting a cardinality of 17.91K, but instead the cardinality is
17.90K.
The RELEASE version has one character fewer than the SNAPSHOT version. The
version gets embedded in parquet files, so the parquet file is slightly smaller
than before. The test is estimating cardinality by looking at the size of the
parquet file. Apparently, this is right on the edge.
This test should tolerate this difference.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8005) Randomize partitioning exchanges destinations
[ https://issues.apache.org/jira/browse/IMPALA-8005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068920#comment-17068920 ] ASF subversion and git services commented on IMPALA-8005: - Commit df6196e064bc7453bee8c7e644bb591391ee3ce2 in impala's branch refs/heads/master from Anurag Mantripragada [ https://gitbox.apache.org/repos/asf?p=impala.git;h=df6196e ] IMPALA-8005: Randomize partitioning exchanges. Currently, we use the same hash seed for partitioning exchanges at the sender. For a table with skew in distribution in the shuffling keys, multiple queries using the same shuffling keys for exchanges will end up hashing to the same destination fragments running on a particular host and potentially overloading that host. This patch seeds the hash with query id. This will ensure that the partitioning exchanges do not always hash to the same destination with same shuffling keys. Testing: Added a test to data-stream-test to verify the data values at destination are different for different queries. Change-Id: I1936e6cc3e8d66420a5a9301f49221ca38f3e468 Reviewed-on: http://gerrit.cloudera.org:8080/15497 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins > Randomize partitioning exchanges destinations > - > > Key: IMPALA-8005 > URL: https://issues.apache.org/jira/browse/IMPALA-8005 > Project: IMPALA > Issue Type: Improvement > Components: Distributed Exec >Affects Versions: Impala 3.1.0 >Reporter: Michael Ho >Assignee: Anurag Mantripragada >Priority: Major > Labels: ramp-up > > Currently, we use the same hash seed for partitioning exchanges at the > sender. For a table with skew in distribution in the shuffling keys, multiple > queries using the same shuffling keys for exchanges will end up hashing to > the same destination fragments running on particular host and potentially > overloading that host. > We should consider using the query id or other query specific information to > seed the hashing function to randomize the destinations for different > queries. Thanks to [~tlipcon] for pointing this problem out. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9555) TestDateQueries.test_queries failing because Hive3 switched back to the hybrid Julian Gregorian calendar
[
https://issues.apache.org/jira/browse/IMPALA-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068921#comment-17068921
]
ASF subversion and git services commented on IMPALA-9555:
-
Commit 2cd7a2b77acfa04094e91efbc6803d11fabcc0e9 in impala's branch
refs/heads/master from Attila Jeges
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2cd7a2b ]
IMPALA-9555: [Hive3] Fix test failure introduced by HIVE-22589
With HIVE-22589 Hive3 switched back to using Julian Calendar for
historical dates by default which caused an Impala test failure
around Avro DATE values.
Change-Id: I51dd933867ea7877235e7f6e1f2b56711dca107e
Reviewed-on: http://gerrit.cloudera.org:8080/15564
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> TestDateQueries.test_queries failing because Hive3 switched back to the
> hybrid Julian Gregorian calendar
>
>
> Key: IMPALA-9555
> URL: https://issues.apache.org/jira/browse/IMPALA-9555
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Attila Jeges
>Assignee: Attila Jeges
>Priority: Critical
>
> TestDateQueries.test_queries is failing after upgrading the CDP GBN with the
> following error:
> {code}
> query_test.test_date_queries.TestDateQueries.test_queries[protocol: beeswax |
> exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen': 'true',
> 'batch_size': 1} | table_format: avro/snap/block] (from pytest)
> Error Message
> query_test/test_date_queries.py:60: in test_queries
> self.run_test_case('QueryTest/avro_date', vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 0,0001-01-01,0001-01-01 !=
> 10,1399-06-27,2017-11-28 E 1,0001-01-01,0001-12-31 != 11,1399-06-27,NULL
> E 10,1399-06-27,2017-11-28 != 12,1399-06-27,2018-12-31 E
> 11,1399-06-27,NULL != 20,2017-11-27,0001-06-19 E 12,1399-06-27,2018-12-31
> != 21,2017-11-27,0001-06-20 E 2,0001-01-01,0002-01-01 !=
> 22,2017-11-27,0001-06-21 E 20,2017-11-27,0001-06-21 !=
> 23,2017-11-27,0001-06-22 E 21,2017-11-27,0001-06-22 !=
> 24,2017-11-27,0001-06-23 E 22,2017-11-27,0001-06-23 !=
> 25,2017-11-27,0001-06-24 E 23,2017-11-27,0001-06-24 !=
> 26,2017-11-27,0001-06-25 E 24,2017-11-27,0001-06-25 !=
> 27,2017-11-27,0001-06-26 E 25,2017-11-27,0001-06-26 !=
> 28,2017-11-27,0001-06-27 E 26,2017-11-27,0001-06-27 !=
> 29,2017-11-27,2017-11-28 E 27,2017-11-27,0001-06-28 !=
> 30,-12-31,-12-01 E 28,2017-11-27,0001-06-29 !=
> 31,-12-31,-12-31 E 29,2017-11-27,2017-11-28 != None E
> 3,0001-01-01,1399-12-31 != None E 30,-12-31,-12-01 != None E
> 31,-12-31,-12-31 != None E 4,0001-01-01,2017-11-28 != None E
> 5,0001-01-01,-12-31 != None E 6,0001-01-01,NULL != None E Number
> of rows returned (expected vs actual): 22 != 15
> Stacktrace
> query_test/test_date_queries.py:60: in test_queries
> self.run_test_case('QueryTest/avro_date', vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 0,0001-01-01,0001-01-01 != 10,1399-06-27,2017-11-28
> E 1,0001-01-01,0001-12-31 != 11,1399-06-27,NULL
> E 10,1399-06-27,2017-11-28 != 12,1399-06-27,2018-12-31
> E 11,1399-06-27,NULL != 20,2017-11-27,0001-06-19
> E 12,1399-06-27,2018-12-31 != 21,2017-11-27,0001-06-20
> E 2,0001-01-01,0002-01-01 != 22,2017-11-27,0001-06-21
> E 20,2017-11-27,0001-06-21 != 23,2017-11-27,0001-06-22
> E 21,2017-11-27,0001-06-22 != 24,2017-11-27,0001-06-23
> E 22,2017-11-27,0001-06-23 != 25,2017-11-27,0001-06-24
> E 23,2017-11-27,0001-06-24 != 26,2017-11-27,0001-06-25
> E 24,2017-11-27,0001-06-25 != 27,2017-11-27,0001-06-26
> E 25,2017-11-27,0001-06-26 != 28,2017-11-27,0001-06-27
> E 26,
[jira] [Resolved] (IMPALA-9555) TestDateQueries.test_queries failing because Hive3 switched back to the hybrid Julian Gregorian calendar
[
https://issues.apache.org/jira/browse/IMPALA-9555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Jeges resolved IMPALA-9555.
--
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> TestDateQueries.test_queries failing because Hive3 switched back to the
> hybrid Julian Gregorian calendar
>
>
> Key: IMPALA-9555
> URL: https://issues.apache.org/jira/browse/IMPALA-9555
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Attila Jeges
>Assignee: Attila Jeges
>Priority: Critical
> Fix For: Impala 3.4.0
>
>
> TestDateQueries.test_queries is failing after upgrading the CDP GBN with the
> following error:
> {code}
> query_test.test_date_queries.TestDateQueries.test_queries[protocol: beeswax |
> exec_option: {'disable_codegen_rows_threshold': 0, 'disable_codegen': 'true',
> 'batch_size': 1} | table_format: avro/snap/block] (from pytest)
> Error Message
> query_test/test_date_queries.py:60: in test_queries
> self.run_test_case('QueryTest/avro_date', vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 0,0001-01-01,0001-01-01 !=
> 10,1399-06-27,2017-11-28 E 1,0001-01-01,0001-12-31 != 11,1399-06-27,NULL
> E 10,1399-06-27,2017-11-28 != 12,1399-06-27,2018-12-31 E
> 11,1399-06-27,NULL != 20,2017-11-27,0001-06-19 E 12,1399-06-27,2018-12-31
> != 21,2017-11-27,0001-06-20 E 2,0001-01-01,0002-01-01 !=
> 22,2017-11-27,0001-06-21 E 20,2017-11-27,0001-06-21 !=
> 23,2017-11-27,0001-06-22 E 21,2017-11-27,0001-06-22 !=
> 24,2017-11-27,0001-06-23 E 22,2017-11-27,0001-06-23 !=
> 25,2017-11-27,0001-06-24 E 23,2017-11-27,0001-06-24 !=
> 26,2017-11-27,0001-06-25 E 24,2017-11-27,0001-06-25 !=
> 27,2017-11-27,0001-06-26 E 25,2017-11-27,0001-06-26 !=
> 28,2017-11-27,0001-06-27 E 26,2017-11-27,0001-06-27 !=
> 29,2017-11-27,2017-11-28 E 27,2017-11-27,0001-06-28 !=
> 30,-12-31,-12-01 E 28,2017-11-27,0001-06-29 !=
> 31,-12-31,-12-31 E 29,2017-11-27,2017-11-28 != None E
> 3,0001-01-01,1399-12-31 != None E 30,-12-31,-12-01 != None E
> 31,-12-31,-12-31 != None E 4,0001-01-01,2017-11-28 != None E
> 5,0001-01-01,-12-31 != None E 6,0001-01-01,NULL != None E Number
> of rows returned (expected vs actual): 22 != 15
> Stacktrace
> query_test/test_date_queries.py:60: in test_queries
> self.run_test_case('QueryTest/avro_date', vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 0,0001-01-01,0001-01-01 != 10,1399-06-27,2017-11-28
> E 1,0001-01-01,0001-12-31 != 11,1399-06-27,NULL
> E 10,1399-06-27,2017-11-28 != 12,1399-06-27,2018-12-31
> E 11,1399-06-27,NULL != 20,2017-11-27,0001-06-19
> E 12,1399-06-27,2018-12-31 != 21,2017-11-27,0001-06-20
> E 2,0001-01-01,0002-01-01 != 22,2017-11-27,0001-06-21
> E 20,2017-11-27,0001-06-21 != 23,2017-11-27,0001-06-22
> E 21,2017-11-27,0001-06-22 != 24,2017-11-27,0001-06-23
> E 22,2017-11-27,0001-06-23 != 25,2017-11-27,0001-06-24
> E 23,2017-11-27,0001-06-24 != 26,2017-11-27,0001-06-25
> E 24,2017-11-27,0001-06-25 != 27,2017-11-27,0001-06-26
> E 25,2017-11-27,0001-06-26 != 28,2017-11-27,0001-06-27
> E 26,2017-11-27,0001-06-27 != 29,2017-11-27,2017-11-28
> E 27,2017-11-27,0001-06-28 != 30,-12-31,-12-01
> E 28,2017-11-27,0001-06-29 != 31,-12-31,-12-31
> E 29,2017-11-27,2017-11-28 != None
> E 3,0001-01-01,1399-12-31 != None
> E 30,-12-31,-12-01 != None
> E 31,-12-31,-12-31 != None
> E 4,0001-01-01,2017-11-28 != None
> E 5,0001-01-01,-12-31 != None
> E 6,0001-01-01,NULL != None
> E Number of rows returned (expected vs actual): 22 != 15
> Standard Error
> ERROR:test_configuration:Compa
[jira] [Created] (IMPALA-9561) Change hadoop-ozone-filesystem dependency to hadoop-ozone-filesystem-lib-current
Sahil Takiar created IMPALA-9561: Summary: Change hadoop-ozone-filesystem dependency to hadoop-ozone-filesystem-lib-current Key: IMPALA-9561 URL: https://issues.apache.org/jira/browse/IMPALA-9561 Project: IMPALA Issue Type: Sub-task Reporter: Sahil Takiar Assignee: Sahil Takiar Ozone publishes a client fat-jar that shades all client jar dependencies. This is nice because it doesn't pollute the client classpath with unnecessary jars / jar conflicts. Impala should use the fat-jar rather than the regular jar. It's also the recommended way for clients to interact with Ozone clusters. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069049#comment-17069049
]
ASF subversion and git services commented on IMPALA-9513:
-
Commit c9da86d00d5aba0aa91bfa59e47f2e28889b2a87 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c9da86d ]
IMPALA-9513: Fix TestKuduOperations.test_column_storage_attributes
When introducing Kudu date support, test_column_storage_attributes
was modified to add the date datatype. The test expects the date
to be represented in python as datetime.date. Instead, the date
is a string in python, so the test consistently fails.
This changes the test so that it expects a python string, and the
test now passes.
Change-Id: Ic198b72041fbe8fe7376c45356e484b304c6f16c
Reviewed-on: http://gerrit.cloudera.org:8080/15567
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Priority: Blocker
> Labels: build-failure
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9176) Make access to null-aware partition from PartitionedHashJoinNode read-only
[ https://issues.apache.org/jira/browse/IMPALA-9176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9176 started by Tim Armstrong. - > Make access to null-aware partition from PartitionedHashJoinNode read-only > -- > > Key: IMPALA-9176 > URL: https://issues.apache.org/jira/browse/IMPALA-9176 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: multithreading > > Currently the accesses to null_aware_partition() are logically read-only > (since the rows and other state is not mutated) and only accesses the build > row when pinned, but is implemented using the built-in read iterator of > BufferedTupleStream. This would prevent sharing of the build side for > null-aware anti-join. > We need to either allow multiple read iterators for a pinned stream, or build > an auxiliary structure, e.g. an array of Tuple ptrs or FlatRowPtr. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-9513:
-
Assignee: Joe McDonnell
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: build-failure
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-9513:
--
Affects Version/s: Impala 3.4.0
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Norbert Luksa
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: build-failure
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-9513:
--
Labels: broken-build (was: build-failure)
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Norbert Luksa
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9513.
---
Fix Version/s: Impala 4.0
Target Version: Impala 3.4.0
Resolution: Fixed
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Norbert Luksa
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9560) Changing version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE breaks TestStatsExtrapolation
[
https://issues.apache.org/jira/browse/IMPALA-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069116#comment-17069116
]
ASF subversion and git services commented on IMPALA-9560:
-
Commit e9dd5d3f8c1d533bc5ae94c7e0677820fcd851aa in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e9dd5d3 ]
IMPALA-9560: Fix TestStatsExtrapolation for release versions
When changing the Impala version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE,
TestStatsExtrapolation::test_stats_extrapolation started failing due
to a difference in the expected cardinality (expected: 17.91K,
actual 17.90K). This is because the Impala version gets embedded into
parquet files, and this causes a slight difference in file size, which
translates into a slight difference in expected cardinality.
This modifies TestStatsExtrapolation::test_stats_extrapolation to
allow any 17.9*K cardinality.
Testing:
- Tested on master and on branch-3.4.0
Change-Id: Iebe538936f23c095ef58c808e425cfb7b31edd94
Reviewed-on: http://gerrit.cloudera.org:8080/15569
Reviewed-by: Tim Armstrong
Tested-by: Impala Public Jenkins
> Changing version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE breaks
> TestStatsExtrapolation
> ---
>
> Key: IMPALA-9560
> URL: https://issues.apache.org/jira/browse/IMPALA-9560
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: broken-build
>
> When working on the Impala 3.4 release, we changed the version on
> branch-3.4.0 from 3.4.0-SNAPSHOT to 3.4.0-RELEASE.
> metadata/test_stats_extrapolation.py::TestStatsExtrapolation::test_stats_extrapolation()
> now fails with the following error:
> {noformat}
> metadata/test_stats_extrapolation.py:44: in test_stats_extrapolation
> self.run_test_case('QueryTest/stats-extrapolation', vector,
> unique_database)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:246: in verify_query_result_is_subset
> assert expected_literal_strings <= actual_literal_strings
> E assert Items in expected results not found in actual results:
> E ' tuple-ids=0 row-size=4B cardinality=17.91K'
> E Items in actual results:
> E '| output exprs: id'
> E ''
> E ' table: rows=unavailable size=unavailable'
> E ' stored statistics:'
> E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=2'
> E ' columns: unavailable'
> E ' partitions: 0/24 rows=unavailable'
> E '00:SCAN HDFS [test_stats_extrapolation_5c6bdfd.alltypes]'
> E ' tuple-ids=0 row-size=4B cardinality=17.90K'
> E '|'
> E 'Analyzed query: SELECT id FROM
> test_stats_extrapolation_5c6bdfd.alltypes'
> E 'F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1'
> E ' HDFS partitions=24/24 files=36 size=281.43KB'
> E 'test_stats_extrapolation_5c6bdfd.alltypes'
> E 'PLAN-ROOT SINK'
> E '| mem-estimate=0B mem-reservation=0B thread-reservation=0'
> E '| Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=2'
> E ' in pipelines: 00(GETNEXT)'
> E ' extrapolated-rows=unavailable max-scan-range-rows=unavailable'
> E 'Per-Host Resource Estimates: Memory=16MB'
> E 'WARNING: The following tables are missing relevant table and/or column
> statistics.'
> E ' mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=1'{noformat}
> The output is expecting a cardinality of 17.91K, but instead the cardinality
> is 17.90K.
> The RELEASE version has one character fewer than the SNAPSHOT version. The
> version gets embedded in parquet files, so the parquet file is slightly
> smaller than before. The test is estimating cardinality by looking at the
> size of the parquet file. Apparently, this is right on the edge.
> This test should tolerate this difference.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9560) Changing version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE breaks TestStatsExtrapolation
[
https://issues.apache.org/jira/browse/IMPALA-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-9560.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
> Changing version from 3.4.0-SNAPSHOT to 3.4.0-RELEASE breaks
> TestStatsExtrapolation
> ---
>
> Key: IMPALA-9560
> URL: https://issues.apache.org/jira/browse/IMPALA-9560
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> When working on the Impala 3.4 release, we changed the version on
> branch-3.4.0 from 3.4.0-SNAPSHOT to 3.4.0-RELEASE.
> metadata/test_stats_extrapolation.py::TestStatsExtrapolation::test_stats_extrapolation()
> now fails with the following error:
> {noformat}
> metadata/test_stats_extrapolation.py:44: in test_stats_extrapolation
> self.run_test_case('QueryTest/stats-extrapolation', vector,
> unique_database)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:246: in verify_query_result_is_subset
> assert expected_literal_strings <= actual_literal_strings
> E assert Items in expected results not found in actual results:
> E ' tuple-ids=0 row-size=4B cardinality=17.91K'
> E Items in actual results:
> E '| output exprs: id'
> E ''
> E ' table: rows=unavailable size=unavailable'
> E ' stored statistics:'
> E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=2'
> E ' columns: unavailable'
> E ' partitions: 0/24 rows=unavailable'
> E '00:SCAN HDFS [test_stats_extrapolation_5c6bdfd.alltypes]'
> E ' tuple-ids=0 row-size=4B cardinality=17.90K'
> E '|'
> E 'Analyzed query: SELECT id FROM
> test_stats_extrapolation_5c6bdfd.alltypes'
> E 'F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1'
> E ' HDFS partitions=24/24 files=36 size=281.43KB'
> E 'test_stats_extrapolation_5c6bdfd.alltypes'
> E 'PLAN-ROOT SINK'
> E '| mem-estimate=0B mem-reservation=0B thread-reservation=0'
> E '| Per-Host Resources: mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=2'
> E ' in pipelines: 00(GETNEXT)'
> E ' extrapolated-rows=unavailable max-scan-range-rows=unavailable'
> E 'Per-Host Resource Estimates: Memory=16MB'
> E 'WARNING: The following tables are missing relevant table and/or column
> statistics.'
> E ' mem-estimate=16.00MB mem-reservation=8.00KB
> thread-reservation=1'{noformat}
> The output is expecting a cardinality of 17.91K, but instead the cardinality
> is 17.90K.
> The RELEASE version has one character fewer than the SNAPSHOT version. The
> version gets embedded in parquet files, so the parquet file is slightly
> smaller than before. The test is estimating cardinality by looking at the
> size of the parquet file. Apparently, this is right on the edge.
> This test should tolerate this difference.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9443) [DOC] Update show table stats output
[
https://issues.apache.org/jira/browse/IMPALA-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kris Hahn resolved IMPALA-9443.
---
Resolution: Fixed
> [DOC] Update show table stats output
>
>
> Key: IMPALA-9443
> URL: https://issues.apache.org/jira/browse/IMPALA-9443
> Project: IMPALA
> Issue Type: Documentation
> Components: Docs
>Reporter: Tamas Mate
>Assignee: Kris Hahn
>Priority: Major
> Labels: future_release_doc, impala_user_docs_open
>
> The {{SHOW TABLE STATS}} output for HDFS tables is outdated on some doc sites
> or abbreviated with ellipsis. Additionally, some other tables can be broken
> as well, for example {{SHOW FILES IN}}. I am aware of the following pages:
> - [https://impala.apache.org/docs/build/html/topics/impala_show.html]
> - [https://impala.apache.org/docs/build/html/topics/impala_perf_stats.html]
> {code:java}
> Currently the output of an empty HDFS table looks like this:
> +---++--+--+---++---+-+
> | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format |
> Incremental stats | Location|
> +---++--+--+---++---+-+
> | -1| 15 | 149B | NOT CACHED | NOT CACHED| TEXT | false
> | hdfs://localhost:20500/test-warehouse/test |
> +---++--+--+---++---+-+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-3343) Impala-shell compatibility with python 3
[ https://issues.apache.org/jira/browse/IMPALA-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell updated IMPALA-3343: -- Target Version: Impala 4.0 (was: Impala 3.4.0) > Impala-shell compatibility with python 3 > > > Key: IMPALA-3343 > URL: https://issues.apache.org/jira/browse/IMPALA-3343 > Project: IMPALA > Issue Type: Bug > Components: Clients >Affects Versions: Impala 2.5.0 >Reporter: Peter Ebert >Assignee: David Knupp >Priority: Critical > > Installed Anaconda package and python 3, Impala shell has errors and will not > run in the python 3 environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9538) Bump up linux-syscall-support.h to newest version
[ https://issues.apache.org/jira/browse/IMPALA-9538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaorenhai resolved IMPALA-9538. Resolution: Fixed > Bump up linux-syscall-support.h to newest version > - > > Key: IMPALA-9538 > URL: https://issues.apache.org/jira/browse/IMPALA-9538 > Project: IMPALA > Issue Type: Sub-task >Reporter: zhaorenhai >Assignee: zhaorenhai >Priority: Major > > Bump up linux-syscall-support.h to newest version which support aarch64 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9562) Replace Intel's popcntq instruction with ARM's mechanism
zhaorenhai created IMPALA-9562: -- Summary: Replace Intel's popcntq instruction with ARM's mechanism Key: IMPALA-9562 URL: https://issues.apache.org/jira/browse/IMPALA-9562 Project: IMPALA Issue Type: Sub-task Reporter: zhaorenhai Assignee: zhaorenhai -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9563) Replace Intel's pcmpestri and pcmpestrm instructions with ARM mechanism
zhaorenhai created IMPALA-9563: -- Summary: Replace Intel's pcmpestri and pcmpestrm instructions with ARM mechanism Key: IMPALA-9563 URL: https://issues.apache.org/jira/browse/IMPALA-9563 Project: IMPALA Issue Type: Sub-task Reporter: zhaorenhai Assignee: zhaorenhai -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9564) Replace Intel's crc32 instructions with ARM's instructions
zhaorenhai created IMPALA-9564: -- Summary: Replace Intel's crc32 instructions with ARM's instructions Key: IMPALA-9564 URL: https://issues.apache.org/jira/browse/IMPALA-9564 Project: IMPALA Issue Type: Sub-task Reporter: zhaorenhai Assignee: zhaorenhai -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9565) Remove unused included file mm_malloc.h on ARM
zhaorenhai created IMPALA-9565: -- Summary: Remove unused included file mm_malloc.h on ARM Key: IMPALA-9565 URL: https://issues.apache.org/jira/browse/IMPALA-9565 Project: IMPALA Issue Type: Sub-task Reporter: zhaorenhai Assignee: zhaorenhai -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9565) Remove unused included file mm_malloc.h on ARM
[ https://issues.apache.org/jira/browse/IMPALA-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaorenhai updated IMPALA-9565: --- Description: ARM version gcc don't have mm_malloc.h file > Remove unused included file mm_malloc.h on ARM > -- > > Key: IMPALA-9565 > URL: https://issues.apache.org/jira/browse/IMPALA-9565 > Project: IMPALA > Issue Type: Sub-task >Reporter: zhaorenhai >Assignee: zhaorenhai >Priority: Major > > ARM version gcc don't have mm_malloc.h file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
