[jira] [Created] (IMPALA-10726) cdpd master compilation failing
Norbert Luksa created IMPALA-10726:
--
Summary: cdpd master compilation failing
Key: IMPALA-10726
URL: https://issues.apache.org/jira/browse/IMPALA-10726
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
We have compilation error in the latest cdpd build:
{code:java}
08:45:44 [ERROR]
/data0/jenkins/workspace/impala-cdpd-master-core/repos/Impala/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:[100,44]
cannot find symbol
08:45:44 [ERROR] symbol: class GetOpenTxnsRequest
08:45:44 [ERROR] location: package org.apache.hadoop.hive.metastore.api
08:45:44 [ERROR]
/data0/jenkins/workspace/impala-cdpd-master-core/repos/Impala/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:[2840,48]
cannot find symbol
08:45:44 [ERROR] symbol: class GetOpenTxnsRequest
08:45:44 [ERROR] location: class
org.apache.impala.catalog.metastore.MetastoreServiceHandler{code}
See job for more info:
https://master-03.jenkins.cloudera.com/job/impala-cdpd-master-core/55/testReport/junit/generate_junitxml/build/a0f544c01ed359a6c400cd6ffb6cba97/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10726) cdpd master FE compilation failing
[
https://issues.apache.org/jira/browse/IMPALA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa updated IMPALA-10726:
---
Summary: cdpd master FE compilation failing (was: cdpd master compilation
failing)
> cdpd master FE compilation failing
> --
>
> Key: IMPALA-10726
> URL: https://issues.apache.org/jira/browse/IMPALA-10726
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Priority: Blocker
> Labels: build-failure
>
> We have compilation error in the latest cdpd build:
> {code:java}
> 08:45:44 [ERROR]
> /data0/jenkins/workspace/impala-cdpd-master-core/repos/Impala/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:[100,44]
> cannot find symbol
> 08:45:44 [ERROR] symbol: class GetOpenTxnsRequest
> 08:45:44 [ERROR] location: package org.apache.hadoop.hive.metastore.api
> 08:45:44 [ERROR]
> /data0/jenkins/workspace/impala-cdpd-master-core/repos/Impala/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:[2840,48]
> cannot find symbol
> 08:45:44 [ERROR] symbol: class GetOpenTxnsRequest
> 08:45:44 [ERROR] location: class
> org.apache.impala.catalog.metastore.MetastoreServiceHandler{code}
> See job for more info:
> https://master-03.jenkins.cloudera.com/job/impala-cdpd-master-core/55/testReport/junit/generate_junitxml/build/a0f544c01ed359a6c400cd6ffb6cba97/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-10726) cdpd master FE compilation failing
[
https://issues.apache.org/jira/browse/IMPALA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa closed IMPALA-10726.
--
Resolution: Duplicate
> cdpd master FE compilation failing
> --
>
> Key: IMPALA-10726
> URL: https://issues.apache.org/jira/browse/IMPALA-10726
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Priority: Blocker
> Labels: build-failure
>
> We have compilation error in the latest cdpd build:
> {code:java}
> 08:45:44 [ERROR]
> /data0/jenkins/workspace/impala-cdpd-master-core/repos/Impala/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:[100,44]
> cannot find symbol
> 08:45:44 [ERROR] symbol: class GetOpenTxnsRequest
> 08:45:44 [ERROR] location: package org.apache.hadoop.hive.metastore.api
> 08:45:44 [ERROR]
> /data0/jenkins/workspace/impala-cdpd-master-core/repos/Impala/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:[2840,48]
> cannot find symbol
> 08:45:44 [ERROR] symbol: class GetOpenTxnsRequest
> 08:45:44 [ERROR] location: class
> org.apache.impala.catalog.metastore.MetastoreServiceHandler{code}
> See job for more info:
> https://master-03.jenkins.cloudera.com/job/impala-cdpd-master-core/55/testReport/junit/generate_junitxml/build/a0f544c01ed359a6c400cd6ffb6cba97/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10733) TestKuduOperations.test_replica_selection failing
Norbert Luksa created IMPALA-10733:
--
Summary: TestKuduOperations.test_replica_selection failing
Key: IMPALA-10733
URL: https://issues.apache.org/jira/browse/IMPALA-10733
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
Assignee: Wenzhe Zhou
Noticed in the recent builds these failures:
{code:java}
query_test/test_kudu.py:567: in test_replica_selection
assert cursor.fetchall() == [(100,)]
E assert [(85,)] == [(100,)]
E At index 0 diff: (85,) != (100,)
E Use -v to get the full diff
{code}
Job: https://master-03.jenkins.cloudera.com/job/impala-cdh-7.1-maint-core-s3/23/
[~wzhou] I'm assigning this to you, since you had relevant commits when this
failure appeared. Could you please check?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10734) Failure in webserver.test_web_pages.TestWebPage.test_query_stmt
Norbert Luksa created IMPALA-10734:
--
Summary: Failure in
webserver.test_web_pages.TestWebPage.test_query_stmt
Key: IMPALA-10734
URL: https://issues.apache.org/jira/browse/IMPALA-10734
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
Noticed exception in recent exhaustive run:
{code:java}
webserver/test_web_pages.py:469: in test_query_stmt
assert check_if_contains, "No matching statement found in the jsons."
E AssertionError: No matching statement found in the jsons.
E assert False
{code}
Job:
https://master-03.jenkins.cloudera.com/job/impala-cdpd-master-exhaustive/41/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10621) TestSpillingDebugActionDimensions.test_spilling_large_rows hanging
[
https://issues.apache.org/jira/browse/IMPALA-10621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358842#comment-17358842
]
Norbert Luksa commented on IMPALA-10621:
Not hanging, but the test failed now with the following exception. Maybe it's
relevant?
{code:java}
query_test/test_spilling.py:83: in test_spilling_large_rows
self.run_test_case('QueryTest/spilling-large-rows', vector, unique_database)
common/impala_test_suite.py:709: in run_test_case
self.__verify_results_and_errors(vector, test_section, result, use_db)
common/impala_test_suite.py:545: in __verify_results_and_errors
replace_filenames_with_placeholder)
common/test_result_verifier.py:469: in verify_raw_results
VERIFIER_MAP[verifier](expected, actual)
common/test_result_verifier.py:278: in verify_query_result_is_equal
assert expected_results == actual_results
E assert Comparing QueryTestResults (expected vs actual):
E '0',10 == '0',10
E '0',1000 == '0',1000
E '1',10 == '1',10
E '1',1000 == '1',1000
E '2',10 == '2',10
E '2',1000 == '2',1000
E '3',10 == '3',10
E '3',1000 == '3',1000
E '4',10 == '4',10
E '4',1000 == '4',1000
E '5',10 == '5',10
E '5',1000 != '6',10
E '6',10 != '6',1000
E '6',1000 != '7',10
E '7',10 != '8',10
E '7',1000 != '8',1000
E '8',10 != '9',10
E '8',1000 != '9',1000
E '9',10 != None
E '9',1000 != None
E Number of rows returned (expected vs actual): 20 != 18
{code}
Job:
https://master-03.jenkins.cloudera.com/job/impala-asf-master-exhaustive-release/43/
> TestSpillingDebugActionDimensions.test_spilling_large_rows hanging
> --
>
> Key: IMPALA-10621
> URL: https://issues.apache.org/jira/browse/IMPALA-10621
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Quanlong Huang
>Assignee: Quanlong Huang
>Priority: Critical
>
> Saw a possible haning in test_spilling_large_rows in a downstream exhaustive
> release build. The query didn't finish in 12mins:
> {code:java}
> query_test/test_spilling.py:83: in test_spilling_large_rows
> self.run_test_case('QueryTest/spilling-large-rows', vector,
> unique_database)
> common/impala_test_suite.py:665: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:603: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:923: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:205: in execute
> result = self.fetch_results(query_string, handle)
> beeswax/impala_beeswax.py:452: in fetch_results
> exec_result = self.__fetch_results(query_handle, max_rows)
> beeswax/impala_beeswax.py:463: in __fetch_results
> results = self.__do_rpc(lambda: self.imp_service.fetch(handle, False,
> fetch_rows))
> beeswax/impala_beeswax.py:520: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: Query b94d5cd2894cbe68:68acfd63 cancelled due to
> unresponsive backend: 127.0.0.1:27000 has not sent a report in 753530ms (max
> allowed lag is 72ms) {code}
> impalad.INFO indicates it's waiting for the coordinator fragment to finish:
> {code:java}
> I0325 23:54:16.146754 16521 Frontend.java:1600]
> b94d5cd2894cbe68:68acfd63] Analyzing query: select
> group_concat(string_col), length(bigstr) from bigstrs2
> group by bigstr db: test_spilling_large_rows_6df9f183
> I0325 23:54:16.147089 16521 BaseAuthorizationChecker.java:110]
> b94d5cd2894cbe68:68acfd63] Authorization check took 0 ms
> I0325 23:54:16.147109 16521 Frontend.java:1643]
> b94d5cd2894cbe68:68acfd63] Analysis and authorization finished.
> I0325 23:54:16.148593 20605 admission-controller.cc:1747]
> b94d5cd2894cbe68:68acfd63] Trying to admit
> id=b94d5cd2894cbe68:68acfd63 in pool_name=default-pool
> executor_group_name=default per_host_mem_estimate=309.98 MB
> dedicated_coord_mem_estimate=134.99 MB max_requests=-1 max_queued=200
> max_mem=-1.00 B
> I0325 23:54:16.148609 20605 admission-controller.cc:1755]
> b94d5cd2894cbe68:68acfd63] Stats: agg_num_running=0,
> agg_num_queued=0, agg_mem_reserved=1.00 GB, local_host(local_mem_admitted=0,
> num_admitted_running=0, num_queued=0, backend_mem_reserved=0,
> topN_query_stats: queries=[3e42ce0b9
[jira] [Assigned] (IMPALA-10922) test_orc_stats failing on exhaustive builds
[
https://issues.apache.org/jira/browse/IMPALA-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reassigned IMPALA-10922:
--
Assignee: Quanlong Huang (was: Norbert Luksa)
> test_orc_stats failing on exhaustive builds
> ---
>
> Key: IMPALA-10922
> URL: https://issues.apache.org/jira/browse/IMPALA-10922
> Project: IMPALA
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Quanlong Huang
>Priority: Blocker
> Labels: broken-build
>
> test_orc_stats.py is failing on certain exhaustive builds. The stack trace of
> the failure looks like below.
> {noformat}
> query_test/test_orc_stats.py:40: in test_orc_stats
> self.run_test_case('QueryTest/orc-stats', vector, use_db=unique_database)
> common/impala_test_suite.py:779: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:653: in verify_runtime_profile
> % (function, field, expected_value, actual_value, op, actual))
> E AssertionError: Aggregation of SUM over RowsRead did not match expected
> results.
> E EXPECTED VALUE:
> E 0
> E
> E
> E ACTUAL VALUE:
> E 10
> E
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10922) test_orc_stats failing on exhaustive builds
[
https://issues.apache.org/jira/browse/IMPALA-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418008#comment-17418008
]
Norbert Luksa commented on IMPALA-10922:
Reassigned to [~stigahuang] as he finished the work in IMPALA-6505.
> test_orc_stats failing on exhaustive builds
> ---
>
> Key: IMPALA-10922
> URL: https://issues.apache.org/jira/browse/IMPALA-10922
> Project: IMPALA
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Quanlong Huang
>Priority: Blocker
> Labels: broken-build
>
> test_orc_stats.py is failing on certain exhaustive builds. The stack trace of
> the failure looks like below.
> {noformat}
> query_test/test_orc_stats.py:40: in test_orc_stats
> self.run_test_case('QueryTest/orc-stats', vector, use_db=unique_database)
> common/impala_test_suite.py:779: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:653: in verify_runtime_profile
> % (function, field, expected_value, actual_value, op, actual))
> E AssertionError: Aggregation of SUM over RowsRead did not match expected
> results.
> E EXPECTED VALUE:
> E 0
> E
> E
> E ACTUAL VALUE:
> E 10
> E
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10822) Allow multiple group returns for LDAP search bind authentication group search
[ https://issues.apache.org/jira/browse/IMPALA-10822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463046#comment-17463046 ] Norbert Luksa commented on IMPALA-10822: Hi [~tmate], this can be closed now, right? > Allow multiple group returns for LDAP search bind authentication group search > - > > Key: IMPALA-10822 > URL: https://issues.apache.org/jira/browse/IMPALA-10822 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Security >Reporter: Tamas Mate >Assignee: Tamas Mate >Priority: Major > > Group search uses the same LdapSearch method that the user which has a > condition that only one result should be returned, otherwise the > authentication is ambiguous. However, this condition is too restrictive for > group searches and multiple search results should be allowed. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4741) ORDER BY behavior with UNION is incorrect
[
https://issues.apache.org/jira/browse/IMPALA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969143#comment-16969143
]
Norbert Luksa commented on IMPALA-4741:
---
Had a look at the parser, it looks like it's deliberately different from
MySQL's parsing, [as stated in a
comment|https://github.com/apache/impala/blob/f49f8d8a32d12128eafb4c76632ca2908d22fa28/fe/src/main/cup/sql-parser.cup#L2281].
Looks like the current parser is designed around UNION, and the binding of the
ORDER BY relies on its left precedence.
Tried to play around the precedence, but did not find a simple solution.
Changing the precedence for ORDER BY just for this scenario would introduce
conflicts at other parts of the parser which did not seem to be easy fixes.
Also, in another go I tried to define precedence between UNION and SELECT, but
that did not work either, breaking other things.
> ORDER BY behavior with UNION is incorrect
> -
>
> Key: IMPALA-4741
> URL: https://issues.apache.org/jira/browse/IMPALA-4741
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.8.0
>Reporter: Greg Rahn
>Assignee: Norbert Luksa
>Priority: Critical
> Labels: correctness, incompatibility, ramp-up, sql-language,
> tpc-ds
> Attachments: query36a.sql, query49.sql
>
>
> When a query uses the UNION, EXCEPT, or INTERSECT operators, the ORDER BY
> clause must be specified at the end of the statement and the results of the
> combined queries are sorted. ORDER BY clauses are not allowed in individual
> branches unless the branch is enclosed by parentheses.
> There are two bugs currently:
> # An ORDER BY is allowed in a branch of a UNION that is not enclosed in
> parentheses
> # The final ORDER BY of a UNION is attached to the nearest branch when it
> should be sorting the combined results of the UNION(s)
> For example, this is not valid syntax but is allowed in Impala
> {code}
> select * from t1 order by 1
> union all
> select * from t2
> {code}
> And for queries like this, the ORDER BY should order the unioned result, not
> just the nearest branch which is the current behavior.
> {code}
> select * from t1
> union all
> select * from t2
> order by 1
> {code}
> If one wants ordering within a branch, the query block must be enclosed by
> parentheses like such:
> {code}
> (select * from t1 order by 1)
> union all
> (select * from t2 order by 2)
> {code}
> Here is an example where incorrect results are returned.
> Impala
> {code}
> [impalad:21000] > select r_regionkey, r_name from region union all select
> r_regionkey, r_name from region order by 1 limit 2;
> +-+-+
> | r_regionkey | r_name |
> +-+-+
> | 0 | AFRICA |
> | 1 | AMERICA |
> | 2 | ASIA|
> | 3 | EUROPE |
> | 4 | MIDDLE EAST |
> | 0 | AFRICA |
> | 1 | AMERICA |
> +-+-+
> Fetched 7 row(s) in 0.12s
> {code}
> PostgreSQL
> {code}
> tpch=# select r_regionkey, r_name from region union all select r_regionkey,
> r_name from region order by 1 limit 2;
> r_regionkey | r_name
> -+---
>0 | AFRICA
>0 | AFRICA
> (2 rows)
> {code}
> see also https://cloud.google.com/spanner/docs/query-syntax#syntax_5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-4618) # hosts is inaccurate in exec summary when mt_dop > 0
[
https://issues.apache.org/jira/browse/IMPALA-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reassigned IMPALA-4618:
-
Assignee: Norbert Luksa
> # hosts is inaccurate in exec summary when mt_dop > 0
> -
>
> Key: IMPALA-4618
> URL: https://issues.apache.org/jira/browse/IMPALA-4618
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Tim Armstrong
>Assignee: Norbert Luksa
>Priority: Major
> Labels: multithreading, ramp-up, supportability
>
> I'm seeing this on a 3-node minicluster:
> {code}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> -
> 06:AGGREGATE1 212.433us 212.433us 0 164.00 B
> -1.00 B FINALIZE
> 05:EXCHANGE 10.000ns0.000ns 0 1 0
> -1.00 B UNPARTITIONED
> 02:AGGREGATE9 131.439us 167.821us 0 164.00 B
>10.00 MB
> 04:AGGREGATE90.000ns0.000ns 0 -1 183.42 MB
> 128.00 MB
> 03:EXCHANGE 9 707.426ms 823.681ms 20.97M -1 0
> 0 HASH(`_c0`)
> 01:AGGREGATE95s222ms5s355ms 22.89M -13.83 MB
> 128.00 MB STREAMING
> 00:SCAN HDFS92s623ms2s896ms 23.33M -1 492.02 KB
>88.00 MB default.big_uuids
> {code}
> It seems like it's reporting the # of fragment instances instead of the
> number of hosts.
> We should probably report both #Hosts and #Instances if mt_dop is enabled
> (and maybe in all cases)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-4618) # hosts is inaccurate in exec summary when mt_dop > 0
[
https://issues.apache.org/jira/browse/IMPALA-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-4618 started by Norbert Luksa.
-
> # hosts is inaccurate in exec summary when mt_dop > 0
> -
>
> Key: IMPALA-4618
> URL: https://issues.apache.org/jira/browse/IMPALA-4618
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Tim Armstrong
>Assignee: Norbert Luksa
>Priority: Major
> Labels: multithreading, ramp-up, supportability
>
> I'm seeing this on a 3-node minicluster:
> {code}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> -
> 06:AGGREGATE1 212.433us 212.433us 0 164.00 B
> -1.00 B FINALIZE
> 05:EXCHANGE 10.000ns0.000ns 0 1 0
> -1.00 B UNPARTITIONED
> 02:AGGREGATE9 131.439us 167.821us 0 164.00 B
>10.00 MB
> 04:AGGREGATE90.000ns0.000ns 0 -1 183.42 MB
> 128.00 MB
> 03:EXCHANGE 9 707.426ms 823.681ms 20.97M -1 0
> 0 HASH(`_c0`)
> 01:AGGREGATE95s222ms5s355ms 22.89M -13.83 MB
> 128.00 MB STREAMING
> 00:SCAN HDFS92s623ms2s896ms 23.33M -1 492.02 KB
>88.00 MB default.big_uuids
> {code}
> It seems like it's reporting the # of fragment instances instead of the
> number of hosts.
> We should probably report both #Hosts and #Instances if mt_dop is enabled
> (and maybe in all cases)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work stopped] (IMPALA-4741) ORDER BY behavior with UNION is incorrect
[
https://issues.apache.org/jira/browse/IMPALA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-4741 stopped by Norbert Luksa.
-
> ORDER BY behavior with UNION is incorrect
> -
>
> Key: IMPALA-4741
> URL: https://issues.apache.org/jira/browse/IMPALA-4741
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.8.0
>Reporter: Greg Rahn
>Assignee: Norbert Luksa
>Priority: Critical
> Labels: correctness, incompatibility, ramp-up, sql-language,
> tpc-ds
> Attachments: query36a.sql, query49.sql
>
>
> When a query uses the UNION, EXCEPT, or INTERSECT operators, the ORDER BY
> clause must be specified at the end of the statement and the results of the
> combined queries are sorted. ORDER BY clauses are not allowed in individual
> branches unless the branch is enclosed by parentheses.
> There are two bugs currently:
> # An ORDER BY is allowed in a branch of a UNION that is not enclosed in
> parentheses
> # The final ORDER BY of a UNION is attached to the nearest branch when it
> should be sorting the combined results of the UNION(s)
> For example, this is not valid syntax but is allowed in Impala
> {code}
> select * from t1 order by 1
> union all
> select * from t2
> {code}
> And for queries like this, the ORDER BY should order the unioned result, not
> just the nearest branch which is the current behavior.
> {code}
> select * from t1
> union all
> select * from t2
> order by 1
> {code}
> If one wants ordering within a branch, the query block must be enclosed by
> parentheses like such:
> {code}
> (select * from t1 order by 1)
> union all
> (select * from t2 order by 2)
> {code}
> Here is an example where incorrect results are returned.
> Impala
> {code}
> [impalad:21000] > select r_regionkey, r_name from region union all select
> r_regionkey, r_name from region order by 1 limit 2;
> +-+-+
> | r_regionkey | r_name |
> +-+-+
> | 0 | AFRICA |
> | 1 | AMERICA |
> | 2 | ASIA|
> | 3 | EUROPE |
> | 4 | MIDDLE EAST |
> | 0 | AFRICA |
> | 1 | AMERICA |
> +-+-+
> Fetched 7 row(s) in 0.12s
> {code}
> PostgreSQL
> {code}
> tpch=# select r_regionkey, r_name from region union all select r_regionkey,
> r_name from region order by 1 limit 2;
> r_regionkey | r_name
> -+---
>0 | AFRICA
>0 | AFRICA
> (2 rows)
> {code}
> see also https://cloud.google.com/spanner/docs/query-syntax#syntax_5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-4741) ORDER BY behavior with UNION is incorrect
[
https://issues.apache.org/jira/browse/IMPALA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reassigned IMPALA-4741:
-
Assignee: (was: Norbert Luksa)
> ORDER BY behavior with UNION is incorrect
> -
>
> Key: IMPALA-4741
> URL: https://issues.apache.org/jira/browse/IMPALA-4741
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.8.0
>Reporter: Greg Rahn
>Priority: Critical
> Labels: correctness, incompatibility, ramp-up, sql-language,
> tpc-ds
> Attachments: query36a.sql, query49.sql
>
>
> When a query uses the UNION, EXCEPT, or INTERSECT operators, the ORDER BY
> clause must be specified at the end of the statement and the results of the
> combined queries are sorted. ORDER BY clauses are not allowed in individual
> branches unless the branch is enclosed by parentheses.
> There are two bugs currently:
> # An ORDER BY is allowed in a branch of a UNION that is not enclosed in
> parentheses
> # The final ORDER BY of a UNION is attached to the nearest branch when it
> should be sorting the combined results of the UNION(s)
> For example, this is not valid syntax but is allowed in Impala
> {code}
> select * from t1 order by 1
> union all
> select * from t2
> {code}
> And for queries like this, the ORDER BY should order the unioned result, not
> just the nearest branch which is the current behavior.
> {code}
> select * from t1
> union all
> select * from t2
> order by 1
> {code}
> If one wants ordering within a branch, the query block must be enclosed by
> parentheses like such:
> {code}
> (select * from t1 order by 1)
> union all
> (select * from t2 order by 2)
> {code}
> Here is an example where incorrect results are returned.
> Impala
> {code}
> [impalad:21000] > select r_regionkey, r_name from region union all select
> r_regionkey, r_name from region order by 1 limit 2;
> +-+-+
> | r_regionkey | r_name |
> +-+-+
> | 0 | AFRICA |
> | 1 | AMERICA |
> | 2 | ASIA|
> | 3 | EUROPE |
> | 4 | MIDDLE EAST |
> | 0 | AFRICA |
> | 1 | AMERICA |
> +-+-+
> Fetched 7 row(s) in 0.12s
> {code}
> PostgreSQL
> {code}
> tpch=# select r_regionkey, r_name from region union all select r_regionkey,
> r_name from region order by 1 limit 2;
> r_regionkey | r_name
> -+---
>0 | AFRICA
>0 | AFRICA
> (2 rows)
> {code}
> see also https://cloud.google.com/spanner/docs/query-syntax#syntax_5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8046) Support CREATE TABLE from an ORC file
[ https://issues.apache.org/jira/browse/IMPALA-8046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa reassigned IMPALA-8046: - Assignee: Norbert Luksa > Support CREATE TABLE from an ORC file > - > > Key: IMPALA-8046 > URL: https://issues.apache.org/jira/browse/IMPALA-8046 > Project: IMPALA > Issue Type: New Feature >Reporter: Quanlong Huang >Assignee: Norbert Luksa >Priority: Major > > Impala supports creating a table using the schema of a file. However, only > parquet is supported currently. This ticket tracks adding support for > creating table from ORC files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9179) Enable reading from bucketed tables
[ https://issues.apache.org/jira/browse/IMPALA-9179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-9179. --- Resolution: Implemented [IMPALA-8593|https://issues.apache.org/jira/browse/IMPALA-8593] only prohibited writing to bucketed tables, while reading is left enabled. > Enable reading from bucketed tables > --- > > Key: IMPALA-9179 > URL: https://issues.apache.org/jira/browse/IMPALA-9179 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > > Reading from bucketed tables should just work out of the box, so we can just > enable it and add proper testing. INSERTs must still raise an error. > Later (in a separate jira) we can improve join and aggregation performance of > bucketed tables. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-9179) Enable reading from bucketed tables
[ https://issues.apache.org/jira/browse/IMPALA-9179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa closed IMPALA-9179. - > Enable reading from bucketed tables > --- > > Key: IMPALA-9179 > URL: https://issues.apache.org/jira/browse/IMPALA-9179 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > > Reading from bucketed tables should just work out of the box, so we can just > enable it and add proper testing. INSERTs must still raise an error. > Later (in a separate jira) we can improve join and aggregation performance of > bucketed tables. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8046) Support CREATE TABLE from an ORC file
[ https://issues.apache.org/jira/browse/IMPALA-8046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8046 started by Norbert Luksa. - > Support CREATE TABLE from an ORC file > - > > Key: IMPALA-8046 > URL: https://issues.apache.org/jira/browse/IMPALA-8046 > Project: IMPALA > Issue Type: New Feature >Reporter: Quanlong Huang >Assignee: Norbert Luksa >Priority: Major > > Impala supports creating a table using the schema of a file. However, only > parquet is supported currently. This ticket tracks adding support for > creating table from ORC files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6660) -0/+0 floating point do not compare as equal in hash table
[
https://issues.apache.org/jira/browse/IMPALA-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa resolved IMPALA-6660.
---
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> -0/+0 floating point do not compare as equal in hash table
> --
>
> Key: IMPALA-6660
> URL: https://issues.apache.org/jira/browse/IMPALA-6660
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0,
> Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Norbert Luksa
>Priority: Major
> Labels: correctness, ramp-up
> Fix For: Impala 3.4.0
>
>
> This can happen because we hash the binary representation of the numbers.
> -0/0 should be treated as equal for hash joins.
> {noformat}
> [localhost:21000] > select * from (select cast("-0" as float) c1) v1,
> (select cast("0" as float) c2) v2 where v1.c1 = v2.c2;
> Fetched 0 row(s) in 0.12s
> [localhost:21000] > select * from (select cast("0" as float) c1) v1, (select
> cast("0" as float) c2) v2 where v1.c1 = v2.c2;
> +++
> | c1 | c2 |
> +++
> | 0 | 0 |
> +++
> Fetched 1 row(s) in 0.11s
> [localhost:21000] > select * from (select cast("-0" as float) c1) v1,
> (select cast("-0" as float) c2) v2 where v1.c1 = v2.c2;
> +++
> | c1 | c2 |
> +++
> | -0 | -0 |
> +++
> Fetched 1 row(s) in 0.11s
> {noformat}
> With aggregations, we get separate groups. I could see the argument either
> way on whether this is the preferred behaviour for group by, since group by
> already handles equality of NULL differently. The behaviour here is tied to
> the behaviour in the join right now, so we should make sure to add a test for
> this case when fixing the join.
> {noformat}
> [localhost:21000] > select distinct * from (values(cast("-0" as float)),
> (cast("0" as float))) v;
> +-+
> | cast('-0' as float) |
> +-+
> | -0 |
> | 0 |
> +-+
> {noformat}
> *Workaround*
> Casting the floating point numbers to decimal fixes the problem.
> *Proposed solution*
> The frontend could wrap floating point expressions in the hash join or hash
> aggregation in a normalisation function that converts -0 to +0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-4618) # hosts is inaccurate in exec summary when mt_dop > 0
[
https://issues.apache.org/jira/browse/IMPALA-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984388#comment-16984388
]
Norbert Luksa commented on IMPALA-4618:
---
[~arodoni] should I create an issue for updating the docs/update them myself?
As Tim mentioned there are some places showing the result of SUMMARY.
> # hosts is inaccurate in exec summary when mt_dop > 0
> -
>
> Key: IMPALA-4618
> URL: https://issues.apache.org/jira/browse/IMPALA-4618
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Tim Armstrong
>Assignee: Norbert Luksa
>Priority: Major
> Labels: multithreading, ramp-up, supportability
>
> I'm seeing this on a 3-node minicluster:
> {code}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> -
> 06:AGGREGATE1 212.433us 212.433us 0 164.00 B
> -1.00 B FINALIZE
> 05:EXCHANGE 10.000ns0.000ns 0 1 0
> -1.00 B UNPARTITIONED
> 02:AGGREGATE9 131.439us 167.821us 0 164.00 B
>10.00 MB
> 04:AGGREGATE90.000ns0.000ns 0 -1 183.42 MB
> 128.00 MB
> 03:EXCHANGE 9 707.426ms 823.681ms 20.97M -1 0
> 0 HASH(`_c0`)
> 01:AGGREGATE95s222ms5s355ms 22.89M -13.83 MB
> 128.00 MB STREAMING
> 00:SCAN HDFS92s623ms2s896ms 23.33M -1 492.02 KB
>88.00 MB default.big_uuids
> {code}
> It seems like it's reporting the # of fragment instances instead of the
> number of hosts.
> We should probably report both #Hosts and #Instances if mt_dop is enabled
> (and maybe in all cases)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9207) Impala Doc: Document adding #Inst in exec summary
Norbert Luksa created IMPALA-9207: - Summary: Impala Doc: Document adding #Inst in exec summary Key: IMPALA-9207 URL: https://issues.apache.org/jira/browse/IMPALA-9207 Project: IMPALA Issue Type: Sub-task Reporter: Norbert Luksa Assignee: Alexandra Rodoni https://gerrit.cloudera.org/#/c/14715/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-4618) # hosts is inaccurate in exec summary when mt_dop > 0
[
https://issues.apache.org/jira/browse/IMPALA-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa resolved IMPALA-4618.
---
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> # hosts is inaccurate in exec summary when mt_dop > 0
> -
>
> Key: IMPALA-4618
> URL: https://issues.apache.org/jira/browse/IMPALA-4618
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.8.0
>Reporter: Tim Armstrong
>Assignee: Norbert Luksa
>Priority: Major
> Labels: multithreading, ramp-up, supportability
> Fix For: Impala 3.4.0
>
>
> I'm seeing this on a 3-node minicluster:
> {code}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
> Est. Peak Mem Detail
> -
> 06:AGGREGATE1 212.433us 212.433us 0 164.00 B
> -1.00 B FINALIZE
> 05:EXCHANGE 10.000ns0.000ns 0 1 0
> -1.00 B UNPARTITIONED
> 02:AGGREGATE9 131.439us 167.821us 0 164.00 B
>10.00 MB
> 04:AGGREGATE90.000ns0.000ns 0 -1 183.42 MB
> 128.00 MB
> 03:EXCHANGE 9 707.426ms 823.681ms 20.97M -1 0
> 0 HASH(`_c0`)
> 01:AGGREGATE95s222ms5s355ms 22.89M -13.83 MB
> 128.00 MB STREAMING
> 00:SCAN HDFS92s623ms2s896ms 23.33M -1 492.02 KB
>88.00 MB default.big_uuids
> {code}
> It seems like it's reporting the # of fragment instances instead of the
> number of hosts.
> We should probably report both #Hosts and #Instances if mt_dop is enabled
> (and maybe in all cases)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9175) Revisit the error handling logics in ORC scanner
[
https://issues.apache.org/jira/browse/IMPALA-9175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9175 started by Norbert Luksa.
-
> Revisit the error handling logics in ORC scanner
>
>
> Key: IMPALA-9175
> URL: https://issues.apache.org/jira/browse/IMPALA-9175
> Project: IMPALA
> Issue Type: Task
>Reporter: Quanlong Huang
>Assignee: Norbert Luksa
>Priority: Blocker
>
> This is a task to revisit all the corresponding error handling logics in the
> ORC scanner comparing to the Parquet scanner. For each kind of error handling
> in the parquet scanner, make sure we already handle it in the orc scanner,
> otherwise create separate JIRAs to handle them.
> Also need to make sure whether the exposed error messages are enough for
> debugging. For instance, one frequently encountered error when Impala has
> stale metadata of an ORC file is:
> {code:java}
> Encountered parse error in tail of ORC file
> hdfs://hadoop2cluster/user/hive-0.13.1/warehouse/bi_ucar.db/alliance_driver_stat_hour_api/dt=2019-08-09/part-6:
> Invalid ORC postscript length
> {code}
> It'd be better to also print the postscript length we read and the file size.
> So users can know whether the file is corrupt (so need data regeneration) or
> the metadata is stale (so need refresh). We may need changes in the ORC lib
> for these.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9226) Improve string allocations of the ORC scanner
[ https://issues.apache.org/jira/browse/IMPALA-9226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9226 started by Norbert Luksa. - > Improve string allocations of the ORC scanner > - > > Key: IMPALA-9226 > URL: https://issues.apache.org/jira/browse/IMPALA-9226 > Project: IMPALA > Issue Type: Improvement >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > Labels: orc > > Currently the ORC scanner allocates new memory for each string values (except > for fixed size strings): > [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/orc-column-readers.cc#L172] > Besides the too many allocations and copying it's also bad for memory > locality. > Since ORC-501 StringVectorBatch has a member named 'blob' that contains the > strings in the batch: > [https://github.com/apache/orc/blob/branch-1.6/c%2B%2B/include/orc/Vector.hh#L126] > 'blob' has type DataBuffer which is movable, so Impala might be able to get > ownership of it. Or, at least we could copy the whole blob array instead of > copying the strings one-by-one. > ORC-501 is included in ORC version 1.6, but Impala currently only uses ORC > 1.5.5. > ORC 1.6 also introduces a new string vector type, EncodedStringVectorBatch: > [https://github.com/apache/orc/blob/e40b9a7205d51995f11fe023c90769c0b7c4bb93/c%2B%2B/include/orc/Vector.hh#L153] > It uses dictionary encoding for storing the values. Impala could copy/move > the dictionary as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8046) Support CREATE TABLE from an ORC file
[ https://issues.apache.org/jira/browse/IMPALA-8046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-8046. --- Fix Version/s: Impala 3.4.0 Resolution: Implemented > Support CREATE TABLE from an ORC file > - > > Key: IMPALA-8046 > URL: https://issues.apache.org/jira/browse/IMPALA-8046 > Project: IMPALA > Issue Type: New Feature >Reporter: Quanlong Huang >Assignee: Norbert Luksa >Priority: Major > Fix For: Impala 3.4.0 > > > Impala supports creating a table using the schema of a file. However, only > parquet is supported currently. This ticket tracks adding support for > creating table from ORC files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9310) OrcStringColumnReader could move blob instead of copy
Norbert Luksa created IMPALA-9310: - Summary: OrcStringColumnReader could move blob instead of copy Key: IMPALA-9310 URL: https://issues.apache.org/jira/browse/IMPALA-9310 Project: IMPALA Issue Type: Bug Reporter: Norbert Luksa Since IMPALA-9226, in OrcStringColumnReader the blobs of a batch are copied to a freshly allocated space by Impala. As Zoltán pointed out in a [review|https://gerrit.cloudera.org/#/c/15051/1/be/src/exec/orc-column-readers.cc@158], a possible improvement is moving a blob by finding a way of getting hold of the memory allocated by the ORC library instead of copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9290) ORC scanner should support schema evolution between date and timestamp types
[
https://issues.apache.org/jira/browse/IMPALA-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reassigned IMPALA-9290:
-
Assignee: Norbert Luksa
> ORC scanner should support schema evolution between date and timestamp types
>
>
> Key: IMPALA-9290
> URL: https://issues.apache.org/jira/browse/IMPALA-9290
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Gabor Kaszab
>Assignee: Norbert Luksa
>Priority: Major
> Labels: orc
>
> *This is the desired use case:*
> 1. Create an ORC table TBL1 with a DATE column.
> 2. Create an ORC table TBL2 with a TIMESTAMP column that has the same
> location as TBL1.
> 3. Insert some DATE values into TBL1 and some TIMESTAMP values into TBL2.
> 4. select from TBL1 returns both DATE and TIMESTAMP values (converted to
> DATE).
> 5. select from TBL2 returns both DATE and TIMESTAMPS values. The DATE values
> are converted to TIMESTAMP.
> Without this feature Impala return an error:
> {code:java}
> ERROR: Type mismatch: table column DATE is map to column timestamp in ORC
> file 'hdfs://localhost:20500/test-warehouse/orc_date_tbl/00_0_copy_1'
> {code}
> *Note:*
> With https://issues.apache.org/jira/browse/IMPALA-8801 implementing Date type
> for ORC it is possible to read date values in ORC format. However, writing is
> still not supported and has to be done by Hive.
> *Let me copy-paste a code review comment from IMPALA-8801 as a suggestion for
> the implementation:*
> We can modify OrcTimestampReader to support reading orc::TimestampVectorBatch
> into Date type slots. In its constructor it knows which kind of slots
> (timestamp or date) it's writting to. So in ReadValue() it can have different
> behaviors based on different modes (timestamp values => timestamp slots /
> timestamp values => date slots). We can do the same on OrcDateColumnReader to
> let it support reading ORC Date values into Timestamp type slots.
> Note that the life cycle of a OrcColumnReader is within the life cycle of the
> HdfsOrcScanner which only reads a split of an ORC file, and an ORC file can't
> have two types for one column (e.g. column1 is timestamp in stripe1 and is
> date in stripe2). So we don't need to deal with different batch types in
> UpdateInputBatch().
> BTW, It'd be better to add test coverage for this type compactibility check
> in test_scanners.py (See TestOrc.test_type_conversions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9290) ORC scanner should support schema evolution between date and timestamp types
[
https://issues.apache.org/jira/browse/IMPALA-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9290 started by Norbert Luksa.
-
> ORC scanner should support schema evolution between date and timestamp types
>
>
> Key: IMPALA-9290
> URL: https://issues.apache.org/jira/browse/IMPALA-9290
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Gabor Kaszab
>Assignee: Norbert Luksa
>Priority: Major
> Labels: orc
>
> *This is the desired use case:*
> 1. Create an ORC table TBL1 with a DATE column.
> 2. Create an ORC table TBL2 with a TIMESTAMP column that has the same
> location as TBL1.
> 3. Insert some DATE values into TBL1 and some TIMESTAMP values into TBL2.
> 4. select from TBL1 returns both DATE and TIMESTAMP values (converted to
> DATE).
> 5. select from TBL2 returns both DATE and TIMESTAMPS values. The DATE values
> are converted to TIMESTAMP.
> Without this feature Impala return an error:
> {code:java}
> ERROR: Type mismatch: table column DATE is map to column timestamp in ORC
> file 'hdfs://localhost:20500/test-warehouse/orc_date_tbl/00_0_copy_1'
> {code}
> *Note:*
> With https://issues.apache.org/jira/browse/IMPALA-8801 implementing Date type
> for ORC it is possible to read date values in ORC format. However, writing is
> still not supported and has to be done by Hive.
> *Let me copy-paste a code review comment from IMPALA-8801 as a suggestion for
> the implementation:*
> We can modify OrcTimestampReader to support reading orc::TimestampVectorBatch
> into Date type slots. In its constructor it knows which kind of slots
> (timestamp or date) it's writting to. So in ReadValue() it can have different
> behaviors based on different modes (timestamp values => timestamp slots /
> timestamp values => date slots). We can do the same on OrcDateColumnReader to
> let it support reading ORC Date values into Timestamp type slots.
> Note that the life cycle of a OrcColumnReader is within the life cycle of the
> HdfsOrcScanner which only reads a split of an ORC file, and an ORC file can't
> have two types for one column (e.g. column1 is timestamp in stripe1 and is
> date in stripe2). So we don't need to deal with different batch types in
> UpdateInputBatch().
> BTW, It'd be better to add test coverage for this type compactibility check
> in test_scanners.py (See TestOrc.test_type_conversions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029728#comment-17029728
]
Norbert Luksa commented on IMPALA-9351:
---
Not sure what could be the problem there.
The data loading completed successfully and tests querying the same table
passed.
The test failed twice, then passed after. Let's see if this happens again?
https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-core/314/testReport/junit/org.apache.impala.analysis/AnalyzeDDLTest/TestCreateTableLikeFileOrc/history/
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagy
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036126#comment-17036126
]
Norbert Luksa commented on IMPALA-9351:
---
I don't think there is anything special, looks like this table is created like
any other table.
Maybe the location/filename can be different sometimes?
[~stigahuang], asking since you have worked more with ORC, have you experienced
anything like that with your ORC files?
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
Th
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038412#comment-17038412
]
Norbert Luksa commented on IMPALA-9351:
---
Thanks, [~joemcdonnell], sent in https://gerrit.cloudera.org/#/c/15234/
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additiona
[jira] [Assigned] (IMPALA-6505) Min-Max predicate push down in ORC scanner
[ https://issues.apache.org/jira/browse/IMPALA-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa reassigned IMPALA-6505: - Assignee: Norbert Luksa > Min-Max predicate push down in ORC scanner > -- > > Key: IMPALA-6505 > URL: https://issues.apache.org/jira/browse/IMPALA-6505 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Quanlong Huang >Assignee: Norbert Luksa >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041951#comment-17041951
]
Norbert Luksa commented on IMPALA-9351:
---
Thanks again for the help, Joe, sent in [https://gerrit.cloudera.org/#/c/15266/]
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
F
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042459#comment-17042459
]
Norbert Luksa commented on IMPALA-9351:
---
Thanks Tim!
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9420) test_scanners.TestOrc.test_type conversions fails after first run
Norbert Luksa created IMPALA-9420:
-
Summary: test_scanners.TestOrc.test_type conversions fails after
first run
Key: IMPALA-9420
URL: https://issues.apache.org/jira/browse/IMPALA-9420
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
Assignee: Gabor Kaszab
The mentioned test passes on the first run, but fails later on, finding more
rows than expected.
By running
{code:java}
hdfs dfs -ls -R / | grep union_comlextypes
{code}
we can find that the previously created files are not cleaned up, so Impala
will find and scan them.
The problem could be that the union_complextypes and ill_complextypes tables
are created as externals.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9420) test_scanners.TestOrc.test_type conversions fails after first run
[
https://issues.apache.org/jira/browse/IMPALA-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa updated IMPALA-9420:
--
Labels: orc ramp-up (was: )
> test_scanners.TestOrc.test_type conversions fails after first run
> -
>
> Key: IMPALA-9420
> URL: https://issues.apache.org/jira/browse/IMPALA-9420
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Assignee: Gabor Kaszab
>Priority: Major
> Labels: orc, ramp-up
>
> The mentioned test passes on the first run, but fails later on, finding more
> rows than expected.
> By running
> {code:java}
> hdfs dfs -ls -R / | grep union_comlextypes
> {code}
> we can find that the previously created files are not cleaned up, so Impala
> will find and scan them.
> The problem could be that the union_complextypes and ill_complextypes tables
> are created as externals.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9226) Improve string allocations of the ORC scanner
[ https://issues.apache.org/jira/browse/IMPALA-9226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-9226. --- Fix Version/s: Impala 4.0 Resolution: Fixed > Improve string allocations of the ORC scanner > - > > Key: IMPALA-9226 > URL: https://issues.apache.org/jira/browse/IMPALA-9226 > Project: IMPALA > Issue Type: Improvement >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > Labels: orc > Fix For: Impala 4.0 > > > Currently the ORC scanner allocates new memory for each string values (except > for fixed size strings): > [https://github.com/apache/impala/blob/85425b81f04c856d7d5ec375242303f78ec7964e/be/src/exec/orc-column-readers.cc#L172] > Besides the too many allocations and copying it's also bad for memory > locality. > Since ORC-501 StringVectorBatch has a member named 'blob' that contains the > strings in the batch: > [https://github.com/apache/orc/blob/branch-1.6/c%2B%2B/include/orc/Vector.hh#L126] > 'blob' has type DataBuffer which is movable, so Impala might be able to get > ownership of it. Or, at least we could copy the whole blob array instead of > copying the strings one-by-one. > ORC-501 is included in ORC version 1.6, but Impala currently only uses ORC > 1.5.5. > ORC 1.6 also introduces a new string vector type, EncodedStringVectorBatch: > [https://github.com/apache/orc/blob/e40b9a7205d51995f11fe023c90769c0b7c4bb93/c%2B%2B/include/orc/Vector.hh#L153] > It uses dictionary encoding for storing the values. Impala could copy/move > the dictionary as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048996#comment-17048996
]
Norbert Luksa commented on IMPALA-9351:
---
Looks like the test has not failed since the last fix. [~joemcdonnell], what do
you think, can we close this Jira now?
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues
[jira] [Work started] (IMPALA-9447) TupleRowCompareTest.DecimalTest crashes on release build
[
https://issues.apache.org/jira/browse/IMPALA-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9447 started by Norbert Luksa.
-
> TupleRowCompareTest.DecimalTest crashes on release build
>
>
> Key: IMPALA-9447
> URL: https://issues.apache.org/jira/browse/IMPALA-9447
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build
>
> TupleRowCompareTest.DecimalTest crashes on the release build with the
> following stack:
> {noformat}
> Operating system: Linux
> 0.0.0 Linux 4.15.0-64-generic #73~16.04.1-Ubuntu SMP Fri
> Sep 13 09:56:18 UTC 2019 x86_64
> CPU: amd64
> family 6 model 94 stepping 3
> 1 CPUGPU: UNKNOWNCrash reason: SIGSEGV
> Crash address: 0x0
> Process uptime: not availableThread 0 (crashed)
> 0 unifiedbetests!int
> impala::TupleRowCompareTest::DecimalDecimalTest
> >(long, long, long, long, int, int) [tuple-row-compare-test.cc : 121 + 0x0]
> rax = 0x0d2b2598 rdx = 0x0021
> rcx = 0x0e34e640 rbx = 0x0d1def00
> rsi = 0x05b9 rdi = 0x0d1df0d8
> rbp = 0x7ffcc5aff280 rsp = 0x7ffcc5aff150
> r8 = 0xr9 = 0x0064
> r10 = 0x r11 = 0x0001
> r12 = 0x0d1df0d8 r13 = 0x0002
> r14 = 0x7ffcc5aff101 r15 = 0x0d2b2580
> rip = 0x014b6981
> Found by: given as instruction pointer in context
> 1 unifiedbetests!impala::TupleRowCompareTest_DecimalTest_Test::TestBody()
> [tuple-row-compare-test.cc : 560 + 0x30]
> rbx = 0x7ffcc5aff2b0 rbp = 0x7ffcc5aff2f0
> rsp = 0x7ffcc5aff290 r12 = 0x7ffcc5aff2a0
> r13 = 0x7ffcc5aff2c0 r14 = 0x0d1def00
> r15 = 0x03399ef0 rip = 0x014a0c81
> Found by: call frame info
> 2 unifiedbetests!void
> testing::internal::HandleExceptionsInMethodIfSupported void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x33
> rbx = 0x014a01d0 rbp = 0x
> rsp = 0x7ffcc5aff300 r12 = 0x0d1def00
> r13 = 0x03576465 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033b4f03
> Found by: call frame info
> 3 unifiedbetests!testing::Test::Run() + 0xba
> rbx = 0x0d1def00 rbp = 0x03399ef0
> rsp = 0x7ffcc5aff340 r12 = 0x06aa2000
> r13 = 0x0d1def00 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033acb5a
> Found by: call frame info
> 4 unifiedbetests!testing::TestInfo::Run() + 0x118
> rbx = 0x06ad48f0 rbp = 0x06aa2000
> rsp = 0x7ffcc5aff360 r12 = 0x06a90de0
> r13 = 0x0d1def00 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033acca8
> Found by: call frame info
> 5 unifiedbetests!testing::TestCase::Run() + 0xb5
> rbx = 0x000b rbp = 0x06ac5500
> rsp = 0x7ffcc5aff3a0 r12 = 0x06aa2000
> r13 = 0x06a90de0 r14 = 0x03399ef0
> r15 = 0x01709c90b7de rip = 0x033acd85
> Found by: call frame info
> 6 unifiedbetests!testing::internal::UnitTestImpl::RunAllTests() + 0x258
> rbx = 0x06aa2000 rbp = 0x06a90de0
> rsp = 0x7ffcc5aff3f0 r12 = 0x006c
> r13 = 0x r14 = 0x01709c90b7de
> r15 = 0x0478d2d4 rip = 0x033ae008
> Found by: call frame info{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9447) TupleRowCompareTest.DecimalTest crashes on release build
[
https://issues.apache.org/jira/browse/IMPALA-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050030#comment-17050030
]
Norbert Luksa commented on IMPALA-9447:
---
This is test issue, caused by gcc generating unaligned instructions like
int128_t. They raises SegmentFault when addresses are not aligned to 16 bytes.
Sent in a fix: https://gerrit.cloudera.org/#/c/15347/
> TupleRowCompareTest.DecimalTest crashes on release build
>
>
> Key: IMPALA-9447
> URL: https://issues.apache.org/jira/browse/IMPALA-9447
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build
>
> TupleRowCompareTest.DecimalTest crashes on the release build with the
> following stack:
> {noformat}
> Operating system: Linux
> 0.0.0 Linux 4.15.0-64-generic #73~16.04.1-Ubuntu SMP Fri
> Sep 13 09:56:18 UTC 2019 x86_64
> CPU: amd64
> family 6 model 94 stepping 3
> 1 CPUGPU: UNKNOWNCrash reason: SIGSEGV
> Crash address: 0x0
> Process uptime: not availableThread 0 (crashed)
> 0 unifiedbetests!int
> impala::TupleRowCompareTest::DecimalDecimalTest
> >(long, long, long, long, int, int) [tuple-row-compare-test.cc : 121 + 0x0]
> rax = 0x0d2b2598 rdx = 0x0021
> rcx = 0x0e34e640 rbx = 0x0d1def00
> rsi = 0x05b9 rdi = 0x0d1df0d8
> rbp = 0x7ffcc5aff280 rsp = 0x7ffcc5aff150
> r8 = 0xr9 = 0x0064
> r10 = 0x r11 = 0x0001
> r12 = 0x0d1df0d8 r13 = 0x0002
> r14 = 0x7ffcc5aff101 r15 = 0x0d2b2580
> rip = 0x014b6981
> Found by: given as instruction pointer in context
> 1 unifiedbetests!impala::TupleRowCompareTest_DecimalTest_Test::TestBody()
> [tuple-row-compare-test.cc : 560 + 0x30]
> rbx = 0x7ffcc5aff2b0 rbp = 0x7ffcc5aff2f0
> rsp = 0x7ffcc5aff290 r12 = 0x7ffcc5aff2a0
> r13 = 0x7ffcc5aff2c0 r14 = 0x0d1def00
> r15 = 0x03399ef0 rip = 0x014a0c81
> Found by: call frame info
> 2 unifiedbetests!void
> testing::internal::HandleExceptionsInMethodIfSupported void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x33
> rbx = 0x014a01d0 rbp = 0x
> rsp = 0x7ffcc5aff300 r12 = 0x0d1def00
> r13 = 0x03576465 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033b4f03
> Found by: call frame info
> 3 unifiedbetests!testing::Test::Run() + 0xba
> rbx = 0x0d1def00 rbp = 0x03399ef0
> rsp = 0x7ffcc5aff340 r12 = 0x06aa2000
> r13 = 0x0d1def00 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033acb5a
> Found by: call frame info
> 4 unifiedbetests!testing::TestInfo::Run() + 0x118
> rbx = 0x06ad48f0 rbp = 0x06aa2000
> rsp = 0x7ffcc5aff360 r12 = 0x06a90de0
> r13 = 0x0d1def00 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033acca8
> Found by: call frame info
> 5 unifiedbetests!testing::TestCase::Run() + 0xb5
> rbx = 0x000b rbp = 0x06ac5500
> rsp = 0x7ffcc5aff3a0 r12 = 0x06aa2000
> r13 = 0x06a90de0 r14 = 0x03399ef0
> r15 = 0x01709c90b7de rip = 0x033acd85
> Found by: call frame info
> 6 unifiedbetests!testing::internal::UnitTestImpl::RunAllTests() + 0x258
> rbx = 0x06aa2000 rbp = 0x06a90de0
> rsp = 0x7ffcc5aff3f0 r12 = 0x006c
> r13 = 0x r14 = 0x01709c90b7de
> r15 = 0x0478d2d4 rip = 0x033ae008
> Found by: call frame info{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-9447) TupleRowCompareTest.DecimalTest crashes on release build
[
https://issues.apache.org/jira/browse/IMPALA-9447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050030#comment-17050030
]
Norbert Luksa edited comment on IMPALA-9447 at 3/3/20 9:16 AM:
---
This is test issue, caused by gcc generating unaligned instructions like movaps
for int128_t. They raises SegmentFault when addresses are not aligned to 16
bytes.
Sent in a fix: [https://gerrit.cloudera.org/#/c/15347/]
was (Author: norbertluksa):
This is test issue, caused by gcc generating unaligned instructions like
int128_t. They raises SegmentFault when addresses are not aligned to 16 bytes.
Sent in a fix: https://gerrit.cloudera.org/#/c/15347/
> TupleRowCompareTest.DecimalTest crashes on release build
>
>
> Key: IMPALA-9447
> URL: https://issues.apache.org/jira/browse/IMPALA-9447
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build
>
> TupleRowCompareTest.DecimalTest crashes on the release build with the
> following stack:
> {noformat}
> Operating system: Linux
> 0.0.0 Linux 4.15.0-64-generic #73~16.04.1-Ubuntu SMP Fri
> Sep 13 09:56:18 UTC 2019 x86_64
> CPU: amd64
> family 6 model 94 stepping 3
> 1 CPUGPU: UNKNOWNCrash reason: SIGSEGV
> Crash address: 0x0
> Process uptime: not availableThread 0 (crashed)
> 0 unifiedbetests!int
> impala::TupleRowCompareTest::DecimalDecimalTest
> >(long, long, long, long, int, int) [tuple-row-compare-test.cc : 121 + 0x0]
> rax = 0x0d2b2598 rdx = 0x0021
> rcx = 0x0e34e640 rbx = 0x0d1def00
> rsi = 0x05b9 rdi = 0x0d1df0d8
> rbp = 0x7ffcc5aff280 rsp = 0x7ffcc5aff150
> r8 = 0xr9 = 0x0064
> r10 = 0x r11 = 0x0001
> r12 = 0x0d1df0d8 r13 = 0x0002
> r14 = 0x7ffcc5aff101 r15 = 0x0d2b2580
> rip = 0x014b6981
> Found by: given as instruction pointer in context
> 1 unifiedbetests!impala::TupleRowCompareTest_DecimalTest_Test::TestBody()
> [tuple-row-compare-test.cc : 560 + 0x30]
> rbx = 0x7ffcc5aff2b0 rbp = 0x7ffcc5aff2f0
> rsp = 0x7ffcc5aff290 r12 = 0x7ffcc5aff2a0
> r13 = 0x7ffcc5aff2c0 r14 = 0x0d1def00
> r15 = 0x03399ef0 rip = 0x014a0c81
> Found by: call frame info
> 2 unifiedbetests!void
> testing::internal::HandleExceptionsInMethodIfSupported void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x33
> rbx = 0x014a01d0 rbp = 0x
> rsp = 0x7ffcc5aff300 r12 = 0x0d1def00
> r13 = 0x03576465 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033b4f03
> Found by: call frame info
> 3 unifiedbetests!testing::Test::Run() + 0xba
> rbx = 0x0d1def00 rbp = 0x03399ef0
> rsp = 0x7ffcc5aff340 r12 = 0x06aa2000
> r13 = 0x0d1def00 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033acb5a
> Found by: call frame info
> 4 unifiedbetests!testing::TestInfo::Run() + 0x118
> rbx = 0x06ad48f0 rbp = 0x06aa2000
> rsp = 0x7ffcc5aff360 r12 = 0x06a90de0
> r13 = 0x0d1def00 r14 = 0x01709c90bbdc
> r15 = 0x03399ef0 rip = 0x033acca8
> Found by: call frame info
> 5 unifiedbetests!testing::TestCase::Run() + 0xb5
> rbx = 0x000b rbp = 0x06ac5500
> rsp = 0x7ffcc5aff3a0 r12 = 0x06aa2000
> r13 = 0x06a90de0 r14 = 0x03399ef0
> r15 = 0x01709c90b7de rip = 0x033acd85
> Found by: call frame info
> 6 unifiedbetests!testing::internal::UnitTestImpl::RunAllTests() + 0x258
> rbx = 0x06aa2000 rbp = 0x06a90de0
> rsp = 0x7ffcc5aff3f0 r12 = 0x006c
> r13 = 0x r14 = 0x01709c90b7de
> r15 = 0x0478d2d4 rip = 0x033ae008
> Found by: call frame info{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-6505) Min-Max predicate push down in ORC scanner
[ https://issues.apache.org/jira/browse/IMPALA-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-6505 started by Norbert Luksa. - > Min-Max predicate push down in ORC scanner > -- > > Key: IMPALA-6505 > URL: https://issues.apache.org/jira/browse/IMPALA-6505 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Quanlong Huang >Assignee: Norbert Luksa >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa resolved IMPALA-9351.
---
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
> Fix For: Impala 3.4.0
>
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h.
[jira] [Work started] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-9351 started by Norbert Luksa.
-
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9464) Tuple::DeepCopyVarlenData crashes on memcpy
Norbert Luksa created IMPALA-9464:
-
Summary: Tuple::DeepCopyVarlenData crashes on memcpy
Key: IMPALA-9464
URL: https://issues.apache.org/jira/browse/IMPALA-9464
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
The stack trace is
{code:java}
Thread 1 (LWP 29440):
[...]
#6
#7 0x0037e4e896ce in memcpy () from ./lib64/libc.so.6
#8 0x00b3fcc6 in impala::Tuple::DeepCopyVarlenData
(this=0x7f318b494948, desc=..., pool=pool@entry=0x7f3486927468) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/tuple.cc:103
#9 0x00b3fdf4 in impala::Tuple::DeepCopy
(this=this@entry=0x7f31f0b200f6, dst=dst@entry=0x7f318b494948, desc=...,
pool=pool@entry=0x7f3486927468) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/tuple.cc:92
#10 0x00b3fe8f in impala::Tuple::DeepCopy (this=0x7f31f0b200f6,
desc=..., pool=0x7f3486927468) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/tuple.cc:83
#11 0x011e5811 in impala::DataStreamSender::Channel::AddRow
(this=0x7f349da09500, row=row@entry=0x191a7f80) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/data-stream-sender.cc:258
#12 0x011e63af in impala::DataStreamSender::Send (this=0x7f3485ca7860,
state=0x21b32e80, batch=0x262ae750) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/data-stream-sender.cc:475
#13 0x00b72124 in impala::FragmentInstanceState::ExecInternal
(this=this@entry=0x1a4ad680) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/fragment-instance-state.cc:275
#14 0x00b749f2 in impala::FragmentInstanceState::Exec
(this=this@entry=0x1a4ad680) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/fragment-instance-state.cc:89
#15 0x00b64038 in impala::QueryState::ExecFInstance (this=0x14347600,
fis=0x1a4ad680) at
/usr/src/debug/impala-2.11.0-cdh5.14.2/be/src/runtime/query-state.cc:382
[..]
{code}
Relevant part of Tuple::DeepCopyVarlenData
{code:c++}
char* string_copy = reinterpret_cast(pool->Allocate(string_v->len));
Ubsan::MemCpy(string_copy, string_v->ptr, string_v->len);
{code}
We could rewrite the function to return Status. This does not solve the root
cause, but at least we could avoid crashes if the allocation fails.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
Norbert Luksa created IMPALA-9513:
-
Summary:
query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on
exhaustive tests
Key: IMPALA-9513
URL: https://issues.apache.org/jira/browse/IMPALA-9513
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
Encountered the mentioned test failures in recent exhaustive tests. The failed
assertion is:
{code:java}
query_test/test_kudu.py:436: in test_column_storage_attributes assert
cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
datetime.date(2010, 1, 1)) E
{code}
Looks like it is caused by IMPALA-8800.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa updated IMPALA-9513:
--
Description:
Encountered the mentioned test failures in recent exhaustive tests. The failed
assertion is:
{code:java}
query_test/test_kudu.py:436: in test_column_storage_attributes assert
cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
datetime.date(2010, 1, 1)) E
{code}
Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
was:
Encountered the mentioned test failures in recent exhaustive tests. The failed
assertion is:
{code:java}
query_test/test_kudu.py:436: in test_column_storage_attributes assert
cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
datetime.date(2010, 1, 1)) E
{code}
Looks like it is caused by IMPALA-8800.
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Priority: Blocker
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060280#comment-17060280
]
Norbert Luksa commented on IMPALA-9513:
---
[~vladimir_committer] could you please take a look?
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Priority: Blocker
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-8857) test_kudu_col_not_null_changed may fail because client reads older timestamp
[
https://issues.apache.org/jira/browse/IMPALA-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reopened IMPALA-8857:
---
Encountered the issue recently in the following build:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/9713/
Not sure if it's relevant, but a similar test, test_kudu_col_changed failed due
to network error here:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/9710/
> test_kudu_col_not_null_changed may fail because client reads older timestamp
>
>
> Key: IMPALA-8857
> URL: https://issues.apache.org/jira/browse/IMPALA-8857
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.3.0
>Reporter: Tim Armstrong
>Assignee: Thomas Tauber-Marshall
>Priority: Critical
> Labels: flaky
>
> {noformat}
> uery_test/test_kudu.py:242: in test_kudu_col_not_null_changed
> assert len(cursor.fetchall()) == 100
> E assert 61 == 100
> E+ where 61 = len([(0, None), (2, None), (4, None), (11, None), (12,
> None), (19, None), ...])
> E+where [(0, None), (2, None), (4, None), (11, None), (12, None),
> (19, None), ...] = >()
> E+ where > =
> .fetchall
> {noformat}
> I believe this is a flaky tests, since there's no attempt to pass the
> timestamp from the kudu client that did the insert to the impala client
> that's doing the reading.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9463) TestInsertQueries.test_insert failed on S3
[
https://issues.apache.org/jira/browse/IMPALA-9463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060979#comment-17060979
]
Norbert Luksa commented on IMPALA-9463:
---
In a recent build, query_test.test_mt_dop.TestMtDopParquet.test_mt_dop_insert
also failed with a similar exception:
{code:java}
ImpalaBeeswaxException: ImpalaBeeswaxException:
Query aborted:InternalException: Error adding partitions CAUSED BY:
MetaException: java.io.IOException: Got exception:
java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
Requested resource not found (Service: AmazonDynamoDBv2;
Status Code: 400;
Error Code: ResourceNotFoundException;
Request ID:
0S5CT4S81MJ7HFQL60P9021D4VVV4KQNSO5AEMVJF66Q9ASUAAJG)
{code}
Stacktrace:
{code:java}
query_test/test_mt_dop.py:122: in test_mt_dop_insert
self.run_test_case('QueryTest/insert', vector)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:659:
in run_test_case
result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:594:
in __exec_in_impala
result = self.__execute_query(target_impalad_client, query, user=user)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:934:
in __execute_query
return impalad_client.execute(query, user=user)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_connection.py:205:
in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:187:
in execute
handle = self.__execute_query(query_string.strip(), user=user)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:365:
in __execute_query
self.wait_for_finished(handle)
/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:386:
in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
EQuery aborted:InternalException: Error adding partitions
E CAUSED BY: MetaException: java.io.IOException: Got exception:
java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ResourceNotFoundException; Request ID:
0S5CT4S81MJ7HFQL60P9021D4VVV4KQNSO5AEMVJF66Q9ASUAAJG)
{code}
> TestInsertQueries.test_insert failed on S3
> --
>
> Key: IMPALA-9463
> URL: https://issues.apache.org/jira/browse/IMPALA-9463
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: broken-build
>
> query_test.test_insert.TestInsertQueries.test_insert failed on S3 with the
> following error:
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> Query aborted:InternalException: Error adding partitions CAUSED BY:
> MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
> Requested resource not found (Service: AmazonDynamoDBv2;
> Status Code: 400;
> Error Code: ResourceNotFoundException;
> Request ID:
> 3NHON9GO4LOH94UN11B7KJ81KBVV4KQNSO5AEMVJF66Q9ASUAAJG){noformat}
> Stack trace of the test was:
> {noformat}
> query_test/test_insert.py:138: in test_insert
> multiple_impalad=vector.get_value('exec_option')['sync_ddl'] == 1)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:659:
> in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:594:
> in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:934:
> in __execute_query
> return impalad_client.execute(query, user=user)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_connection.py:205:
> in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> /data/jenkins/workspace/impala-asf-master-core
[jira] [Commented] (IMPALA-9463) TestInsertQueries.test_insert failed on S3
[
https://issues.apache.org/jira/browse/IMPALA-9463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060981#comment-17060981
]
Norbert Luksa commented on IMPALA-9463:
---
The failed build was:
[impala-asf-master-core-s3|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-s3/578/]
> TestInsertQueries.test_insert failed on S3
> --
>
> Key: IMPALA-9463
> URL: https://issues.apache.org/jira/browse/IMPALA-9463
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: broken-build
>
> query_test.test_insert.TestInsertQueries.test_insert failed on S3 with the
> following error:
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> Query aborted:InternalException: Error adding partitions CAUSED BY:
> MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
> Requested resource not found (Service: AmazonDynamoDBv2;
> Status Code: 400;
> Error Code: ResourceNotFoundException;
> Request ID:
> 3NHON9GO4LOH94UN11B7KJ81KBVV4KQNSO5AEMVJF66Q9ASUAAJG){noformat}
> Stack trace of the test was:
> {noformat}
> query_test/test_insert.py:138: in test_insert
> multiple_impalad=vector.get_value('exec_option')['sync_ddl'] == 1)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:659:
> in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:594:
> in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_test_suite.py:934:
> in __execute_query
> return impalad_client.execute(query, user=user)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_connection.py:205:
> in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:187:
> in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:365:
> in __execute_query
> self.wait_for_finished(handle)
> /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py:386:
> in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:InternalException: Error adding partitions
> E CAUSED BY: MetaException: java.io.IOException: Got exception:
> java.io.FileNotFoundException getVersionMarkerItem on ../VERSION:
> com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested
> resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
> ResourceNotFoundException; Request ID:
> 3NHON9GO4LOH94UN11B7KJ81KBVV4KQNSO5AEMVJF66Q9ASUAAJG){noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9446) query_test.test_compressed_formats.TestReadZtsdLibCompressedFile fails on S3
[
https://issues.apache.org/jira/browse/IMPALA-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060993#comment-17060993
]
Norbert Luksa commented on IMPALA-9446:
---
Looks like ASF Jira bot failed to copy the commit message, so here it is for
reference:
IMPALA-9446: Fix bug that impala failed to read zstd file on s3
S3 filesystem doesn't have a block concept, so scheduler splits each
file into a smaller scan ranges. Adding ZSTD as a case in the switch
statement to add support for zstd.
Testing done:
In mini-cluster, read external table located on s3 zstd file.
Run end to end test successfully on s3.
Change-Id: I13f8837fda7454ddb4bb47d20a675d6315a3462d
Reviewed-on: http://gerrit.cloudera.org:8080/15391
Reviewed-by: Impala Public Jenkins
Tested-by: Tim Armstrong
> query_test.test_compressed_formats.TestReadZtsdLibCompressedFile fails on S3
>
>
> Key: IMPALA-9446
> URL: https://issues.apache.org/jira/browse/IMPALA-9446
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.4.0
>Reporter: Joe McDonnell
>Assignee: Xiaomeng Zhang
>Priority: Blocker
> Labels: broken-build
>
> On the S3 configuration,
> query_test.test_compressed_formats.TestReadZtsdLibCompressedFile is
> encountering the following error:
>
> {noformat}
> query_test/test_compressed_formats.py:325: in test_query_large_file
> result = self.client.execute("select count(*) from %s" %
> self.COMPRESSED_TABLE_NAME)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:Rejected query from pool default-pool: Error during
> scheduling: Invalid file descriptor compression code: 10{noformat}
> The error comes from here:
>
> [https://github.com/apache/impala/blob/master/be/src/util/flat_buffer.cc#L63]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9430) Kerberos configs should be passed through to Kerberos libraries even if principal is not set
[ https://issues.apache.org/jira/browse/IMPALA-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060994#comment-17060994 ] Norbert Luksa commented on IMPALA-9430: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-9430: always pass through kerberos configs The behaviour of kerberos-related command line flags is changed so that their values are always passed through to underlying libraries, even if Kerberos isn't enabled for internal communication in Impala. This is good because: * Various libraries that communicate with external systems may use kerberos for outgoing connections, if *incoming* connections are not authenticated. e.g. it might just be enabled for HMS. Having them pick up different kerberos settings for outgoing connections if kerberos is disabled for incoming connections is a little weird. This is a safer default that reduces chances of inadvertant misconfigurations. * It matches the documentation of the flags. Some validations are still disabled when --principal is not set, e.g. we don't check the replay cache directory. This is to avoid any potential regressions or startup failures on non-kerberised clusters. Testing: Added unit tests for flag validation and env var setting on the code paths that I touched. Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 Reviewed-on: http://gerrit.cloudera.org:8080/15340 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Kerberos configs should be passed through to Kerberos libraries even if > principal is not set > > > Key: IMPALA-9430 > URL: https://issues.apache.org/jira/browse/IMPALA-9430 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: kerberos, security > Fix For: Impala 3.4.0 > > > InitKerberosEnv() configures native and JDK kerberos implementations based on > command-line flags: > https://github.com/apache/impala/blob/d1b42c836c3458a2ef3662c0b0b1fd8fbf8f2baf/be/src/rpc/authentication.cc#L866 > . It only does this when --principal is set. > It's possible that Impala can be set up to use kerberos to communicate with > some external services, e.g. HMS or Hive, even if --principal is not set, > since those clients read in config XML files that are independent of the > Impala flags. This isn't a recommended configuration and requires a fair bit > of expertise to get right, but I think it's very surprising that the configs > *don't* get passed through in the case. The documentation doesn't mention > this behaviour. > The suggested change here is to apply the config changes independent of the > value of --principal. It should be a noop if kerberos is not configured for > any services. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9359) Recover gracefully from corrupt kerberos credential cache
[ https://issues.apache.org/jira/browse/IMPALA-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060995#comment-17060995 ] Norbert Luksa commented on IMPALA-9359: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-9359: recover from corrupt kerberos ccache This is a clean cherry-pick of KUDU-3050. The original commit message is below. KUDU-3050: recover from corrupt kerberos ccache This handles two failure modes: * krb5_cc_start_seq_get() can fail if the kerberos credential cache gets corrupted on disk, e.g. is truncated. * the renewal can fail to find a credential in the credential cache, either if it is missing or the renewal thread hits an error while reading through credentials. Also add some additional logging and limit the max backoff time to make it easier to debug other kinds of renewal errors. The test triggers a pre-existing memory leak bug in some older Kerberos libraries. Added a suppression for leak sanitizer to ClientNegotiation::CheckGSSAPI() to suppress it. Test: Add a test that exercises the recovery logic after truncating the credential cache. The test failed before this change. Change-Id: I86567f16816d1c6729679398ce56296744cb30c9 Reviewed-on: http://gerrit.cloudera.org:8080/15407 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins > Recover gracefully from corrupt kerberos credential cache > - > > Key: IMPALA-9359 > URL: https://issues.apache.org/jira/browse/IMPALA-9359 > Project: IMPALA > Issue Type: Improvement > Components: Security >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: kerberos > Fix For: Impala 3.4.0 > > > # Start up a kerberized Impala cluster > # Corrupt the kerberos ticket cache used by impala /tmp/krb5cc_impala_internal > # Observe queries fail. The details depend a lot on timing, etc. I have seen > communication failures between impalads and with other systems, e.g. HDFS. > # The system will stay wedge in this state indefinitely > We have seen this happen once in production from /tmp filling up. > I prototyped a fix that amounts to re-running Kinit() to blow away the broken > credential cache. It needs more work to be production-ready -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9456) Allow disabling kerberos for incoming internal and external connections even if --principal is set
[ https://issues.apache.org/jira/browse/IMPALA-9456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060996#comment-17060996 ] Norbert Luksa commented on IMPALA-9456: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-9456: allow disabling kerberos selectively There are specific use cases where we need to talk to kerberized services (HMS, etc) and want to keep our TGT up to date using Impala's kinit infrastructure, but don't want to kerberize all connections. Adds --skip_internal_kerberos_auth and --skip_external_kerberos_auth to disable Kerberos authentication for incoming connections even if --principal is set. The daemon does a kinit and keeps tickets up-to-date with the background thread even if kerberos is disabled for all incoming connections. Behaviour only changes when those flags are toggled. The change required restructuring the code a bit, specifically pulling the call to InitKerberosForServer() out of SecureAuthProvider, which I think is a net improvement that makes the control flow clearer. Testing: Add unit tests to: * confirm that the kinit occurs even with auth disabled. * confirm that incoming KRPC connections do not require authentication. I would have liked to add automated tests for thrift interfaces but did not have the infrastructure to unit-test it. I think the changes I made are fairly low risk because they do not increase the number of code paths in AuthManager::Init() and can be verified by inspection. The tests I would have liked to add are: * confirm that incoming external connections do not require auth * confirm that internal thrift connections do not require auth. Manually started kerberized minicluster with internal/external kerberos disabled, e.g. with the command line: start-impala-cluster.py --impalad_args=--skip_external_kerberos_auth=true --impalad_args=--skip_internal_kerberos_auth=true --state_store_args=--skip_internal_kerberos_auth=true --catalogd_args=--skip_internal_kerberos_auth=true Confirmed that impala-shell connected without -k when --skip_external_kerberos_auth=true and requires -k otherwise. Confirmed that we could run queries against HDFS tables even with internal and external auth disabled. Checked logs to see that tickets were reacquired. I0303 08:33:49.319911 16079 init.cc:303] Successfully reacquired a new kerberos TGT Tested that a partially kerberised minicluster worked as expected, i.e. that impalad <-> catalog/statestore connections can have auth disabled so that the impalad can authenticate even if it does not have the right principal set. The first start-impala-cluster.py command below succeeds but the second and third fail because they cannot authenticate with the catalog and statestore respectively because the processes did not kinit as the impala principal. kinit -kt impala.keytab tarmstrong/localh...@example.com start-impala-cluster.py --impalad_args='--principal="" --be_principal="" --keytab_file="" --krb5_ccname="/tmp/krb5cc_1000"' --state_store_args=--skip_internal_kerberos_auth=true --catalogd_args=--skip_internal_kerberos_auth=true kinit -kt impala.keytab tarmstrong/localh...@example.com start-impala-cluster.py --impalad_args='--principal="" --be_principal="" --keytab_file="" --krb5_ccname="/tmp/krb5cc_1000"' --state_store_args=--skip_internal_kerberos_auth=false --catalogd_args=--skip_internal_kerberos_auth=true kinit -kt impala.keytab tarmstrong/localh...@example.com start-impala-cluster.py --impalad_args='--principal="" --be_principal="" --keytab_file="" --krb5_ccname="/tmp/krb5cc_1000"' --state_store_args=--skip_internal_kerberos_auth=false --catalogd_args=--skip_internal_kerberos_auth=true Change-Id: I3b1c641e05e588287e4d9d9cd8389d96fc71cf74 Reviewed-on: http://gerrit.cloudera.org:8080/15351 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins > Allow disabling kerberos for incoming internal and external connections even > if --principal is set > -- > > Key: IMPALA-9456 > URL: https://issues.apache.org/jira/browse/IMPALA-9456 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: kerberos, security > Fix For: Impala 3.4.0 > > > This would be useful in cases where we need to talk to kerberized services > (HMS, etc) and want to keep our TGT up to date using Impala's kinit > infrastructure, but don't want to kerberize all connections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e
[jira] [Commented] (IMPALA-8800) Add DATE type support to Kudu scanner
[ https://issues.apache.org/jira/browse/IMPALA-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060997#comment-17060997 ] Norbert Luksa commented on IMPALA-8800: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-8800: Added support of Kudu DATE type to Impala This patch supports reading and writing DATE values to Kudu tables. It does not add min-max filter runtime support, but there is followup JIRA IMPALA-9294. Corresponding Kudu JIRA is KUDU-2632. Change-Id: I91656749a58ac769b54c2a63bdd4f85c89520b32 Reviewed-on: http://gerrit.cloudera.org:8080/14705 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add DATE type support to Kudu scanner > - > > Key: IMPALA-8800 > URL: https://issues.apache.org/jira/browse/IMPALA-8800 > Project: IMPALA > Issue Type: Sub-task >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8800) Add DATE type support to Kudu scanner
[ https://issues.apache.org/jira/browse/IMPALA-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060999#comment-17060999 ] Norbert Luksa commented on IMPALA-8800: --- [~attilaj] could you have a look at IMPALA-9513? > Add DATE type support to Kudu scanner > - > > Key: IMPALA-8800 > URL: https://issues.apache.org/jira/browse/IMPALA-8800 > Project: IMPALA > Issue Type: Sub-task >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9490) DOC: Include brief statement of support for reading Apache Hudi optimized table
[ https://issues.apache.org/jira/browse/IMPALA-9490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061000#comment-17061000 ] Norbert Luksa commented on IMPALA-9490: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-9490 [DOCS] State support for reading Apache Hudi optimized table Added a row impala_file_format.xml, created impala_hudi.xml, added new file to ditamap, changed id of concept tag from orc to hudi. Change-Id: If210cd545a8deb059e66fd36e62e0df4402fc96c Reviewed-on: http://gerrit.cloudera.org:8080/15418 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong > DOC: Include brief statement of support for reading Apache Hudi optimized > table > --- > > Key: IMPALA-9490 > URL: https://issues.apache.org/jira/browse/IMPALA-9490 > Project: IMPALA > Issue Type: Documentation > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Kris Hahn >Assignee: Kris Hahn >Priority: Major > > Document experimental support for Apache Hudi Read Optimized Table. See > IMPALA-8778. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9369) Inserts on large tables could be very slow when event processing it turned on
[
https://issues.apache.org/jira/browse/IMPALA-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061001#comment-17061001
]
Norbert Luksa commented on IMPALA-9369:
---
Looks like ASF Jira bot failed to copy the commit message, so here it is for
reference:
IMPALA-9369: Make createInsertEvents() async.
This patch makes the createInsertEvents() method async to avoid
blocking the insert code path for long periods for tables with
large number of partitions and files.
Currently the createInsertEvents() method fires the HMS insert
event one partition at a time. This makes insert statements
with thousands of new files significantly slower. This change
makes the createInsertEvent() call asynchronous by making it
run in a separate thread.
Testing:
- Ran MetastoreEventsProcessorTest#testInsertEvents.
- Ran test_events_processing::test_insert_events.
Change-Id: I97802a5c03abc067fccf9e3a9d0047324626706e
Reviewed-on: http://gerrit.cloudera.org:8080/15263
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Inserts on large tables could be very slow when event processing it turned on
> -
>
> Key: IMPALA-9369
> URL: https://issues.apache.org/jira/browse/IMPALA-9369
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Reporter: Vihang Karajgaonkar
>Assignee: Anurag Mantripragada
>Priority: Critical
>
> In case where large number files are being inserted into a table, the
> {{createInsertEvents}} method fires insert events to HMS for each partition
> one take a time. This could be very slow for a insert statement which is
> added hundreds or thousands of files.
> We should see if we can fire the insert events asynchronously instead of
> blocking the query from returning to the user until all the insert events are
> fired.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9504) DOC: Remove "experimental" from description of ORC support.
[ https://issues.apache.org/jira/browse/IMPALA-9504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061003#comment-17061003 ] Norbert Luksa commented on IMPALA-9504: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-9446: Fix bug that impala failed to read zstd file on s3 S3 filesystem doesn't have a block concept, so scheduler splits each file into a smaller scan ranges. Adding ZSTD as a case in the switch statement to add support for zstd. Testing done: In mini-cluster, read external table located on s3 zstd file. Run end to end test successfully on s3. Change-Id: I13f8837fda7454ddb4bb47d20a675d6315a3462d Reviewed-on: http://gerrit.cloudera.org:8080/15391 Reviewed-by: Impala Public Jenkins Tested-by: Tim Armstrong > DOC: Remove "experimental" from description of ORC support. > --- > > Key: IMPALA-9504 > URL: https://issues.apache.org/jira/browse/IMPALA-9504 > Project: IMPALA > Issue Type: Documentation >Reporter: Kris Hahn >Assignee: Kris Hahn >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9513) query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails on exhaustive tests
[
https://issues.apache.org/jira/browse/IMPALA-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa updated IMPALA-9513:
--
Labels: build-failure (was: )
> query_test.test_kudu.TestKuduOperations.test_column_storage_attributes fails
> on exhaustive tests
>
>
> Key: IMPALA-9513
> URL: https://issues.apache.org/jira/browse/IMPALA-9513
> Project: IMPALA
> Issue Type: Bug
>Reporter: Norbert Luksa
>Priority: Blocker
> Labels: build-failure
>
> Encountered the mentioned test failures in recent exhaustive tests. The
> failed assertion is:
> {code:java}
> query_test/test_kudu.py:436: in test_column_storage_attributes assert
> cursor.fetchall() == \ E assert [(0, True, 0, 0, 0, 0, ...)] == [(0, True, 0,
> 0, 0, 0, ...)] E At index 0 diff: (0, True, 0, 0, 0, 0, 0.0, 0.0, '0',
> datetime.datetime(2009, 1, 1, 0, 0), Decimal('0'), '2010-01-01') != (0, True,
> 0, 0, 0, 0, 0.0, 0.0, '0', datetime.datetime(2009, 1, 1, 0, 0), 0,
> datetime.date(2010, 1, 1)) E
> {code}
> Looks like it is caused by https://gerrit.cloudera.org/#/c/14705/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9467) Impala Doc: Improve Impala shell usability by enabling live_progress in the interactive mode
[ https://issues.apache.org/jira/browse/IMPALA-9467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061112#comment-17061112 ] Norbert Luksa commented on IMPALA-9467: --- Looks like ASF Jira bot failed to copy the commit message, so here it is for reference: IMPALA-9467: [DOCS] live_progress enabled by default in interactive mode The following documents were impacted by the change: - impala_live_progress.xml, revised to explain new behavior - impala_shell_options.xml, added --disable_live_progress option Change-Id: I94e624b7bb916ecb5aeb4f007c0610807f7b18cf Reviewed-on: http://gerrit.cloudera.org:8080/15442 Tested-by: Impala Public Jenkins Reviewed-by: Alice Fan Reviewed-by: Joe McDonnell > Impala Doc: Improve Impala shell usability by enabling live_progress in the > interactive mode > > > Key: IMPALA-9467 > URL: https://issues.apache.org/jira/browse/IMPALA-9467 > Project: IMPALA > Issue Type: Documentation > Components: Clients >Reporter: Alice Fan >Assignee: Kris Hahn >Priority: Major > > https://gerrit.cloudera.org/#/c/15219/ > We enable shell option live_progress in interactive mode by default. As for > in the non-interactive mode, live reporting is not supported. Impala-shell > will disable live_progress if the mode is detected. Need to update the doc to > reflect the changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9114) TestKuduHMSIntegration failing: Kudu create table failing
[
https://issues.apache.org/jira/browse/IMPALA-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062590#comment-17062590
]
Norbert Luksa commented on IMPALA-9114:
---
Found a similar exception during a [data-loading
test|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-data-load/941/]:
{code:sql}
ERROR: CREATE TABLE functional_kudu.alltypes (
id INT PRIMARY KEY,
bool_col BOOLEAN,
tinyint_col TINYINT,
smallint_col SMALLINT,
int_col INT,
bigint_col BIGINT,
float_col FLOAT,
double_col DOUBLE,
date_string_col STRING,
string_col STRING,
timestamp_col TIMESTAMP,
year INT,
month INT
)
PARTITION BY HASH (id) PARTITIONS 3 STORED AS KUDU
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/bin/load-data.py",
line 208, in exec_impala_query_from_file
result = impala_client.execute(query)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 187, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 363, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 357, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 520, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION:
MESSAGE: AnalysisException: Cannot analyze Kudu table 'alltypes': Error
determining if Kudu's integration with the Hive Metastore is enabled: too many
attempts: KuduRpc(method=getHiveMetastoreConfig, tablet=null, attempt=101,
TimeoutTracker(timeout=18, elapsed=171248), Trace Summary(170894 ms):
Sent(0), Received(0), Delayed(100), MasterRefresh(0), AuthRefresh(0),
Truncated: false
Delayed: (UNKNOWN, [ getHiveMetastoreConfig, 100 ]))
{code}
> TestKuduHMSIntegration failing: Kudu create table failing
> -
>
> Key: IMPALA-9114
> URL: https://issues.apache.org/jira/browse/IMPALA-9114
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Bikramjeet Vig
>Assignee: Hao Hao
>Priority: Minor
> Labels: broken-build
>
> {noformat}
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Cannot
> analyze Kudu table 't': Error determining if Kudu's integration with the Hive
> Metastore is enabled: cannot complete before timeout:
> KuduRpc(method=getHiveMetastoreConfig, tablet=null, attempt=97,
> TimeoutTracker(timeout=18, elapsed=178723), Trace Summary(177842 ms):
> Sent(0), Received(0), Delayed(96), MasterRefresh(0), AuthRefresh(0),
> Truncated: false Delayed: (UNKNOWN, [ getHiveMetastoreConfig, 96 ]))
> Stacktrace
> custom_cluster/test_kudu.py:150: in test_create_managed_kudu_tables
> self.run_test_case('QueryTest/kudu_create', vector,
> use_db=unique_database)
> common/impala_test_suite.py:621: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:556: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:893: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:362: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:356: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:519: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Cannot analyze Kudu table 't': Error
> determining if Kudu's integration with the Hive Metastore is enabled: cannot
> complete before timeout: KuduRpc(method=getHiveMetastoreConfig, tablet=null,
> attempt=97, TimeoutTracker(timeout=18, elapsed=178723), Trace
> Summary(177842 ms): Sent(0), Received(0), Delayed(96), MasterRefresh(0),
> AuthRefresh(0), Truncated:
[jira] [Comment Edited] (IMPALA-9114) TestKuduHMSIntegration failing: Kudu create table failing
[
https://issues.apache.org/jira/browse/IMPALA-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062590#comment-17062590
]
Norbert Luksa edited comment on IMPALA-9114 at 3/19/20, 1:56 PM:
-
Found a similar exception during a [data-loading
test|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-data-load/941/]
in create-functional-query-exhaustive-impala-generated-kudu-none-none.sql:
{code:sql}
ERROR: CREATE TABLE functional_kudu.alltypes (
id INT PRIMARY KEY,
bool_col BOOLEAN,
tinyint_col TINYINT,
smallint_col SMALLINT,
int_col INT,
bigint_col BIGINT,
float_col FLOAT,
double_col DOUBLE,
date_string_col STRING,
string_col STRING,
timestamp_col TIMESTAMP,
year INT,
month INT
)
PARTITION BY HASH (id) PARTITIONS 3 STORED AS KUDU
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/bin/load-data.py",
line 208, in exec_impala_query_from_file
result = impala_client.execute(query)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 187, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 363, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 357, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 520, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION:
MESSAGE: AnalysisException: Cannot analyze Kudu table 'alltypes': Error
determining if Kudu's integration with the Hive Metastore is enabled: too many
attempts: KuduRpc(method=getHiveMetastoreConfig, tablet=null, attempt=101,
TimeoutTracker(timeout=18, elapsed=171248), Trace Summary(170894 ms):
Sent(0), Received(0), Delayed(100), MasterRefresh(0), AuthRefresh(0),
Truncated: false
Delayed: (UNKNOWN, [ getHiveMetastoreConfig, 100 ]))
{code}
was (Author: norbertluksa):
Found a similar exception during a [data-loading
test|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-data-load/941/]:
{code:sql}
ERROR: CREATE TABLE functional_kudu.alltypes (
id INT PRIMARY KEY,
bool_col BOOLEAN,
tinyint_col TINYINT,
smallint_col SMALLINT,
int_col INT,
bigint_col BIGINT,
float_col FLOAT,
double_col DOUBLE,
date_string_col STRING,
string_col STRING,
timestamp_col TIMESTAMP,
year INT,
month INT
)
PARTITION BY HASH (id) PARTITIONS 3 STORED AS KUDU
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/bin/load-data.py",
line 208, in exec_impala_query_from_file
result = impala_client.execute(query)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 187, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 363, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 357, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File
"/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/beeswax/impala_beeswax.py",
line 520, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION:
MESSAGE: AnalysisException: Cannot analyze Kudu table 'alltypes': Error
determining if Kudu's integration with the Hive Metastore is enabled: too many
attempts: KuduRpc(method=getHiveMetastoreConfig, tablet=null, attempt=101,
TimeoutTracker(timeout=18, elapsed=171248), Trace Summary(170894 ms):
Sent(0), Received(0), Delayed(100), MasterRefresh(0), AuthRefresh(0),
Truncated: false
Delayed: (UNKNOWN, [ getHiveMetastoreConfig, 100 ]))
{code}
> TestKuduHMSIntegration failing: Kudu create table failing
> -
>
> Key: IMPALA-9114
> URL: https://issues.apache.org/jira/browse/IMPALA-9114
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.4.0
>Reporter: Bikramjeet Vig
>Assignee: Hao Hao
>Priority: Minor
>
[jira] [Commented] (IMPALA-7181) Fix flaky test shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening
[
https://issues.apache.org/jira/browse/IMPALA-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062600#comment-17062600
]
Norbert Luksa commented on IMPALA-7181:
---
This happened again in
[impala-cdpd-master-core:|https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-core/393/]
{code:java}
/data/jenkins/workspace/impala-cdpd-master-core/repos/Impala/tests/shell/test_shell_commandline.py:822:
in test_socket_opening
self._validate_expected_socket_connected(vector, args2, s)
/data/jenkins/workspace/impala-cdpd-master-core/repos/Impala/tests/shell/test_shell_commandline.py:792:
in _validate_expected_socket_connected
connection, client_address = sock.accept()
/usr/lib64/python2.7/socket.py:202: in accept
sock, addr = self._sock.accept()
E error: [Errno 4] Interrupted system call
{code}
> Fix flaky test
> shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening
> -
>
> Key: IMPALA-7181
> URL: https://issues.apache.org/jira/browse/IMPALA-7181
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Vincent Tran
>Assignee: Vincent Tran
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 2.13.0, Impala 3.1.0
>
>
> shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening uses
> netcat to listen to an ephemeral port to verify impala-shell socket opening
> behavior.
> The port is hardcoded to 42000 which can fail the test if this port is used
> by an outgoing socket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7181) Fix flaky test shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening
[ https://issues.apache.org/jira/browse/IMPALA-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062604#comment-17062604 ] Norbert Luksa commented on IMPALA-7181: --- [~thundergun] could you take a look? > Fix flaky test > shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening > - > > Key: IMPALA-7181 > URL: https://issues.apache.org/jira/browse/IMPALA-7181 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Vincent Tran >Assignee: Vincent Tran >Priority: Blocker > Labels: broken-build > Fix For: Impala 2.13.0, Impala 3.1.0 > > > shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening uses > netcat to listen to an ephemeral port to verify impala-shell socket opening > behavior. > The port is hardcoded to 42000 which can fail the test if this port is used > by an outgoing socket. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-9572) Impalad crash when process decimal value
[ https://issues.apache.org/jira/browse/IMPALA-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa reassigned IMPALA-9572: - Assignee: Csaba Ringhofer (was: Norbert Luksa) > Impalad crash when process decimal value > > > Key: IMPALA-9572 > URL: https://issues.apache.org/jira/browse/IMPALA-9572 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 >Reporter: Yongzhi Chen >Assignee: Csaba Ringhofer >Priority: Blocker > Labels: broken-build, crash > Attachments: > impalad.impala-ec2-centos74-m5-4xlarge-ondemand-1a73.vpc.cloudera.com.jenkins.log.ERROR.20200328-031659.18222.gz > > > In impala-asf-master-exhaustive build, minidump shows impalad crashed: > Crash reason: SIGABRT > Crash address: 0x7d1472e > Process uptime: not available > Thread 396 (crashed) > 0 libc-2.17.so + 0x351f7 > rax = 0x rdx = 0x0006 > rcx = 0x rbx = 0x07246600 > rsi = 0x4273 rdi = 0x472e > rbp = 0x7f6e43ee7050 rsp = 0x7f6e43ee6ce8 > r8 = 0xr9 = 0x7f6e43ee6b60 > r10 = 0x0008 r11 = 0x0206 > r12 = 0x07246680 r13 = 0x0044 > r14 = 0x0724dfc4 r15 = 0x07246600 > rip = 0x7f6f2b1da1f7 > Found by: given as instruction pointer in context > 1 libc-2.17.so + 0x368e8 > rbp = 0x7f6e43ee7050 rsp = 0x7f6e43ee6cf0 > rip = 0x7f6f2b1db8e8 > Found by: stack scanning > 2 impalad!google_breakpad::ExceptionHandler::HandleSignal(int, siginfo_t*, > void*) + 0x1e0 > rbp = 0x7f6e43ee7050 rsp = 0x7f6e43ee6d78 > rip = 0x04ed0840 > Found by: stack scanning > 3 impalad!google::DumpStackTraceAndExit() + 0x24 > rbp = 0x7f6e43ee7050 rsp = 0x7f6e43ee6e20 > rip = 0x04ea2554 > Found by: stack scanning > 4 impalad!google::LogMessage::Fail() + 0xd > rbx = 0x07246600 rbp = 0x7f6e43ee7050 > rsp = 0x7f6e43ee6ed0 rip = 0x04e98fad > Found by: call frame info > 5 impalad!google::LogMessage::SendToLog() + 0x2b2 > rbx = 0x07246600 rbp = 0x7f6e43ee7050 > rsp = 0x7f6e43ee6ee0 rip = 0x04e9a852 > Found by: call frame info > 6 impalad!google::LogMessage::Flush() + 0x157 > rbx = 0x7f6e43ee7090 rbp = 0x7f6f2bd8a6a0 > rsp = 0x7f6e43ee7060 r12 = 0x7f6e43ee707f > r13 = 0x072554f8 r14 = 0x7f6e43ee7120 > r15 = 0x18954c50 rip = 0x04e98987 > Found by: call frame info > 7 impalad!google::LogMessageFatal::~LogMessageFatal() + 0xe > rbx = 0x7f6e43ee7120 rbp = 0x7f6e43ee7160 > rsp = 0x7f6e43ee70e0 r12 = 0x0001 > r13 = 0x072554f8 r14 = 0x000118ba > r15 = 0x18954c50 rip = 0x04e9bf4e > Found by: call frame info > 8 impalad!impala::StrideWriter::StrideWriter(long*, long) [mem-util.h > : 40 + 0xd] > rbx = 0x0001 rbp = 0x7f6e43ee7160 > rsp = 0x7f6e43ee7100 r12 = 0x0001 > r13 = 0x072554f8 r14 = 0x000118ba > r15 = 0x18954c50 rip = 0x02c07535 > Found by: call frame info > 9 > impalad!impala::BaseScalarColumnReader::FillPositionsInCandidateRange(int, > int, unsigned char*, int) [parquet-column-readers.cc : 1319 + 0x1f] > rbx = 0x17d4 rbp = 0x7f6e43ee7270 > rsp = 0x7f6e43ee7170 r12 = 0x > r13 = 0x0039 r14 = 0x000118ba > r15 = 0x18954c50 rip = 0x02c008d3 > Found by: call frame info > 10 impalad!impala::ScalarColumnReader, > (parquet::Type::type)7, true>::MaterializeValueBatchRepeatedDefLevel(int, > int, unsigned char*, int*) [parquet-column-readers.cc : 612 + 0x2b] > rbx = 0x0007 rbp = 0x7f6e43ee73a0 > rsp = 0x7f6e43ee7280 r12 = 0x > r13 = 0x0039 r14 = 0x000118ba > r15 = 0x18954c50 rip = 0x02c731f7 > Found by: call frame info > 11 impalad!bool impala::ScalarColumnReader, > (parquet::Type::type)7, true>::ReadValueBatch(int, int, unsigned char*, > int*) [parquet-column-readers.cc : 468 + 0x26] > rbx = 0x rbp = 0x7f6e43ee7520 > rsp = 0x7f6e43ee73b0 r12 = 0x0400 > r13 = 0x0039 r14 = 0x000118ba > r15 = 0x18954c50 rip = 0x02c54db1 > Found by: call frame info > 12 impalad!impala::ScalarColumnReader, > (parquet::Type::type)7, true>::ReadValueBatch(impala::MemPool*, int, int, > uns
[jira] [Commented] (IMPALA-8755) Implement Z-ordering for Impala
[ https://issues.apache.org/jira/browse/IMPALA-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110961#comment-17110961 ] Norbert Luksa commented on IMPALA-8755: --- Hi, I'll try to make some time to do this this week. > Implement Z-ordering for Impala > --- > > Key: IMPALA-8755 > URL: https://issues.apache.org/jira/browse/IMPALA-8755 > Project: IMPALA > Issue Type: New Feature >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > > Implement Z-ordering for Impala: [https://en.wikipedia.org/wiki/Z-order_curve] > A Z-order curve defines an ordering on multi-dimensional data. Data sorted > that way can be efficiently filtered by min/max statistics regarding to the > columns participating in the ordering. > Impala currently only supports lexicographic ordering via the SORT BY clause. > This strongly prefers the first column, i.e. given the "SORT BY A, B, C" > clause => A will be totally ordered (hence filtering on A will be very > efficient), but values belonging to B and C will be scattered throughout the > data set (hence filtering on B or C will barely do any good). > We could add a new clause, e.g. a "ZSORT BY" clause to Impala that writes the > data in Z-order. > "ZSORT BY A, B C" would cluster the rows in a way that filtering on A, B, or > C would be equally efficient. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8755) Implement Z-ordering for Impala
[ https://issues.apache.org/jira/browse/IMPALA-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-8755. --- Target Version: Impala 4.0 Resolution: Implemented > Implement Z-ordering for Impala > --- > > Key: IMPALA-8755 > URL: https://issues.apache.org/jira/browse/IMPALA-8755 > Project: IMPALA > Issue Type: New Feature >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > > Implement Z-ordering for Impala: [https://en.wikipedia.org/wiki/Z-order_curve] > A Z-order curve defines an ordering on multi-dimensional data. Data sorted > that way can be efficiently filtered by min/max statistics regarding to the > columns participating in the ordering. > Impala currently only supports lexicographic ordering via the SORT BY clause. > This strongly prefers the first column, i.e. given the "SORT BY A, B, C" > clause => A will be totally ordered (hence filtering on A will be very > efficient), but values belonging to B and C will be scattered throughout the > data set (hence filtering on B or C will barely do any good). > We could add a new clause, e.g. a "ZSORT BY" clause to Impala that writes the > data in Z-order. > "ZSORT BY A, B C" would cluster the rows in a way that filtering on A, B, or > C would be equally efficient. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-9912) Move unlock_zorder_sort flag to the graveyard
Norbert Luksa created IMPALA-9912: - Summary: Move unlock_zorder_sort flag to the graveyard Key: IMPALA-9912 URL: https://issues.apache.org/jira/browse/IMPALA-9912 Project: IMPALA Issue Type: Task Reporter: Norbert Luksa Assignee: Norbert Luksa -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-10054) test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reopened IMPALA-10054:
> test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
> ---
>
> Key: IMPALA-10054
> URL: https://issues.apache.org/jira/browse/IMPALA-10054
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Attila Jeges
>Assignee: Riza Suminto
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> test_multiple_sort_run_bytes_limits introduced in IMPALA-6692 seems to be
> flaky.
> Jenkins job that triggered the error:
> https://jenkins.impala.io/job/parallel-all-tests-nightly/1173
> Failing job:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2899/testReport/
> {code}
> Stacktrace
> query_test/test_sort.py:89: in test_multiple_sort_run_bytes_limits
> assert "SpilledRuns: " + spilled_runs in query_result.runtime_profile
> E assert ('SpilledRuns: ' + '3') in 'Query
> (id=404da0b1e56e7248:120789cd):\n DEBUG MODE WARNING: Query profile
> created while running a DEBUG buil... 27.999ms\n - WriteIoBytes:
> 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime:
> 0.000ns\n'
> E+ where 'Query (id=404da0b1e56e7248:120789cd):\n DEBUG MODE
> WARNING: Query profile created while running a DEBUG buil... 27.999ms\n
> - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n -
> WriteIoWaitTime: 0.000ns\n' =
> 0x7f51da77fb50>.runtime_profile
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10054) test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180362#comment-17180362
]
Norbert Luksa commented on IMPALA-10054:
[~rizaon] this has reoccurred in
[https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-core-asan/454/]
Could you please have a look at it?
> test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
> ---
>
> Key: IMPALA-10054
> URL: https://issues.apache.org/jira/browse/IMPALA-10054
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Attila Jeges
>Assignee: Riza Suminto
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> test_multiple_sort_run_bytes_limits introduced in IMPALA-6692 seems to be
> flaky.
> Jenkins job that triggered the error:
> https://jenkins.impala.io/job/parallel-all-tests-nightly/1173
> Failing job:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2899/testReport/
> {code}
> Stacktrace
> query_test/test_sort.py:89: in test_multiple_sort_run_bytes_limits
> assert "SpilledRuns: " + spilled_runs in query_result.runtime_profile
> E assert ('SpilledRuns: ' + '3') in 'Query
> (id=404da0b1e56e7248:120789cd):\n DEBUG MODE WARNING: Query profile
> created while running a DEBUG buil... 27.999ms\n - WriteIoBytes:
> 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime:
> 0.000ns\n'
> E+ where 'Query (id=404da0b1e56e7248:120789cd):\n DEBUG MODE
> WARNING: Query profile created while running a DEBUG buil... 27.999ms\n
> - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n -
> WriteIoWaitTime: 0.000ns\n' =
> 0x7f51da77fb50>.runtime_profile
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10094) TestResetMetadata.test_refresh_updated_partitions fails due to connection error
Norbert Luksa created IMPALA-10094:
--
Summary: TestResetMetadata.test_refresh_updated_partitions fails
due to connection error
Key: IMPALA-10094
URL: https://issues.apache.org/jira/browse/IMPALA-10094
Project: IMPALA
Issue Type: Bug
Reporter: Norbert Luksa
Assignee: Vihang Karajgaonkar
This has occurred in the last few builds in impala-cdpd-master-staging-core-s3.
Error message:
{code:java}
metadata/test_reset_metadata.py:49: in test_refresh_updated_partitions
"alter table {0} add partition (year=2020, month=8)".format(tbl))
common/impala_test_suite.py:983: in run_stmt_in_hive
raise RuntimeError(stderr)
E RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/apache-hive-3.1.3000.7.2.2.0-135-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/hadoop-3.1.1.7.2.2.0-135/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
E SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
E ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console. Set system property
'log4j2.debug' to show Log4j2 internal initialization logging.
E SLF4J: Class path contains multiple SLF4J bindings.
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/apache-hive-3.1.3000.7.2.2.0-135-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/hadoop-3.1.1.7.2.2.0-135/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
E SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
E Connecting to jdbc:hive2://localhost:11050
E 20/08/18 05:10:24 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:11050
E Could not open connection to the HS2 server. Please check the server URI
and if the URI is correct, then ask the administrator to check the server
status.
E Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:11050: java.net.ConnectException: Connection refused
(Connection refused) (state=08S01,code=0){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10094) TestResetMetadata.test_refresh_updated_partitions fails due to connection error
[
https://issues.apache.org/jira/browse/IMPALA-10094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa updated IMPALA-10094:
---
Description:
This has occurred in the last few builds in impala-cdpd-master-staging-core-s3:
[https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-staging-core-s3/14/]
Error message:
{code:java}
metadata/test_reset_metadata.py:49: in test_refresh_updated_partitions
"alter table {0} add partition (year=2020, month=8)".format(tbl))
common/impala_test_suite.py:983: in run_stmt_in_hive
raise RuntimeError(stderr)
E RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/apache-hive-3.1.3000.7.2.2.0-135-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/hadoop-3.1.1.7.2.2.0-135/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
E SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
E ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console. Set system property
'log4j2.debug' to show Log4j2 internal initialization logging.
E SLF4J: Class path contains multiple SLF4J bindings.
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/apache-hive-3.1.3000.7.2.2.0-135-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/hadoop-3.1.1.7.2.2.0-135/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
E SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
E Connecting to jdbc:hive2://localhost:11050
E 20/08/18 05:10:24 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:11050
E Could not open connection to the HS2 server. Please check the server URI
and if the URI is correct, then ask the administrator to check the server
status.
E Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:11050: java.net.ConnectException: Connection refused
(Connection refused) (state=08S01,code=0){code}
was:
This has occurred in the last few builds in impala-cdpd-master-staging-core-s3.
Error message:
{code:java}
metadata/test_reset_metadata.py:49: in test_refresh_updated_partitions
"alter table {0} add partition (year=2020, month=8)".format(tbl))
common/impala_test_suite.py:983: in run_stmt_in_hive
raise RuntimeError(stderr)
E RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/apache-hive-3.1.3000.7.2.2.0-135-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/hadoop-3.1.1.7.2.2.0-135/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
E SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
E ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console. Set system property
'log4j2.debug' to show Log4j2 internal initialization logging.
E SLF4J: Class path contains multiple SLF4J bindings.
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/apache-hive-3.1.3000.7.2.2.0-135-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-cdpd-master-staging-core-s3/Impala-Toolchain/cdp_components-5250295/hadoop-3.1.1.7.2.2.0-135/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
E SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
E SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
E Connecting to jdbc:hive2://localhost:11050
E 20/08/18 05:10:24 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:11050
E Could not open c
[jira] [Commented] (IMPALA-9923) Data loading of TPC-DS ORC fails with "Fail to get checksum"
[
https://issues.apache.org/jira/browse/IMPALA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180432#comment-17180432
]
Norbert Luksa commented on IMPALA-9923:
---
This also occurred in functional data loads:
[https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-exhaustive/454/]
The error from
load-functional-query-exhaustive-hive-generated-orc-def-block.sql.log:
{code:java}
INFO : Loading data to table functional_orc_def.complextypestbl_medium from
hdfs://localhost:20500/test-warehouse/managed/complextypestbl_medium_orc_defINFO
: Loading data to table functional_orc_def.complextypestbl_medium from
hdfs://localhost:20500/test-warehouse/managed/complextypestbl_medium_orc_defERROR
: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
checksum, since file
/test-warehouse/managed/complextypestbl_medium_orc_def/base_001/_orc_acid_version
is under construction.INFO : Completed executing
command(queryId=jenkins_20200816064113_1cafbc10-8d60-4acb-8812-567fb8e911c6);
Time taken: 4.04 secondsINFO : OKError: Error while compiling statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
checksum, since file
/test-warehouse/managed/complextypestbl_medium_orc_def/base_001/_orc_acid_version
is under construction. (state=08S01,code=1)java.sql.SQLException: Error while
compiling statement: FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
checksum, since file
/test-warehouse/managed/complextypestbl_medium_orc_def/base_001/_orc_acid_version
is under construction. at
org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266) at
org.apache.hive.beeline.Commands.executeInternal(Commands.java:1007) at
org.apache.hive.beeline.Commands.execute(Commands.java:1217) at
org.apache.hive.beeline.Commands.sql(Commands.java:1146) at
org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497) at
org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355) at
org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1329) at
org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1127) at
org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082) at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546) at
org.apache.hive.beeline.BeeLine.main(BeeLine.java:528) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.util.RunJar.run(RunJar.java:318) at
org.apache.hadoop.util.RunJar.main(RunJar.java:232)Closing: 0:
jdbc:hive2://localhost:11050/default;auth=none
{code}
> Data loading of TPC-DS ORC fails with "Fail to get checksum"
>
>
> Key: IMPALA-9923
> URL: https://issues.apache.org/jira/browse/IMPALA-9923
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Zoltán Borók-Nagy
>Priority: Critical
> Labels: broken-build, flaky
> Attachments: load-tpcds-core-hive-generated-orc-def-block.sql,
> load-tpcds-core-hive-generated-orc-def-block.sql.log
>
>
> {noformat}
> INFO : Loading data to table tpcds_orc_def.store_sales partition
> (ss_sold_date_sk=null) from
> hdfs://localhost:20500/test-warehouse/managed/tpcds.store_sales_orc_def
> INFO :
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
> checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> INFO : Completed executing
> command(queryId=ubuntu_20200707055650_a1958916-1e85-4db5-b1bc-cc63d80b3537);
> Time taken: 14.512 seconds
> INFO : OK
> Error: Error while compiling statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to
> get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction. (state=08S01,code=1)
> java.sql.SQLException: Error while compiling statement: FAILED: Execution
> Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.io.IOException: Fail to get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_aci
[jira] [Commented] (IMPALA-9920) BufferPoolTest.WriteErrorBlacklistHolepunch failed on FindPageInDir check
[ https://issues.apache.org/jira/browse/IMPALA-9920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182829#comment-17182829 ] Norbert Luksa commented on IMPALA-9920: --- This happened again in [https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-staging-core-ubsan/15/] Also, not sure if the same, but I saw similar errors in the log for [https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-ubsan/56/] Attached the logs. > BufferPoolTest.WriteErrorBlacklistHolepunch failed on FindPageInDir check > - > > Key: IMPALA-9920 > URL: https://issues.apache.org/jira/browse/IMPALA-9920 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 > Environment: BUILD_TAG > jenkins-impala-cdpd-master-core-ubsan-108 >Reporter: Wenzhe Zhou >Assignee: Tim Armstrong >Priority: Major > Labels: broken-build, build, test > Attachments: buffer-pool-test.ERROR > > > BufferPoolTest.WriteErrorBlacklistHolepunch failed with following error > messages: > Error Message > Value of: FindPageInDir(pages[NO_ERROR_QUERY], good_dir) != NULL > Actual: false > Expected: true > Stacktrace > /data/jenkins/workspace/impala-cdpd-master-core-ubsan/repos/Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1763 > Value of: FindPageInDir(pages[NO_ERROR_QUERY], good_dir) != NULL > Actual: false > Expected: true > > Saw following messages in buffer-pool-test.ERROR log file: > F0702 14:17:08.745453 8976 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.581707 12727 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.671219 12728 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.840062 12940 buffer-pool.cc:216] Check failed: > page_handle->is_pinned() > F0702 14:17:15.167520 13459 buffer-pool.cc:493] Check failed: > spilling_enabled() > F0702 14:17:15.326829 13946 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > E0702 14:17:17.119957 16180 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/impala-scratch/:_354de439-8418-4013-8ebe-55214f8396c5': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/impala-scratch/:_354de439-8418-4013-8ebe-55214f8396c5. > Access denied for the process' user errno=13 > E0702 14:17:17.270885 16570 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/buffer-pool-test.0/impala-scratch/:_21e8f7c1-2a63-44d5-8a5c-4ba78e9acb6e': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/buffer-pool-test.0/impala-scratch/:_21e8f7c1-2a63-44d5-8a5c-4ba78e9acb6e. > Access denied for the process' user errno=13 > E0702 14:17:17.436445 16964 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/buffer-pool-test.0/impala-scratch/:_78d2ef68-254a-4738-baf6-6e80b3b1ee2a': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/buffer-pool-test.0/impala-scratch/:_78d2ef68-254a-4738-baf6-6e80b3b1ee2a. > Access denied for the process' user errno=13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-9920) BufferPoolTest.WriteErrorBlacklistHolepunch failed on FindPageInDir check
[ https://issues.apache.org/jira/browse/IMPALA-9920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa reopened IMPALA-9920: --- > BufferPoolTest.WriteErrorBlacklistHolepunch failed on FindPageInDir check > - > > Key: IMPALA-9920 > URL: https://issues.apache.org/jira/browse/IMPALA-9920 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 > Environment: BUILD_TAG > jenkins-impala-cdpd-master-core-ubsan-108 >Reporter: Wenzhe Zhou >Assignee: Tim Armstrong >Priority: Major > Labels: broken-build, build, test > Attachments: buffer-pool-test-master-core-ubsan-56.ERROR, > buffer-pool-test.ERROR > > > BufferPoolTest.WriteErrorBlacklistHolepunch failed with following error > messages: > Error Message > Value of: FindPageInDir(pages[NO_ERROR_QUERY], good_dir) != NULL > Actual: false > Expected: true > Stacktrace > /data/jenkins/workspace/impala-cdpd-master-core-ubsan/repos/Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1763 > Value of: FindPageInDir(pages[NO_ERROR_QUERY], good_dir) != NULL > Actual: false > Expected: true > > Saw following messages in buffer-pool-test.ERROR log file: > F0702 14:17:08.745453 8976 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.581707 12727 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.671219 12728 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.840062 12940 buffer-pool.cc:216] Check failed: > page_handle->is_pinned() > F0702 14:17:15.167520 13459 buffer-pool.cc:493] Check failed: > spilling_enabled() > F0702 14:17:15.326829 13946 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > E0702 14:17:17.119957 16180 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/impala-scratch/:_354de439-8418-4013-8ebe-55214f8396c5': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/impala-scratch/:_354de439-8418-4013-8ebe-55214f8396c5. > Access denied for the process' user errno=13 > E0702 14:17:17.270885 16570 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/buffer-pool-test.0/impala-scratch/:_21e8f7c1-2a63-44d5-8a5c-4ba78e9acb6e': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/buffer-pool-test.0/impala-scratch/:_21e8f7c1-2a63-44d5-8a5c-4ba78e9acb6e. > Access denied for the process' user errno=13 > E0702 14:17:17.436445 16964 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/buffer-pool-test.0/impala-scratch/:_78d2ef68-254a-4738-baf6-6e80b3b1ee2a': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/buffer-pool-test.0/impala-scratch/:_78d2ef68-254a-4738-baf6-6e80b3b1ee2a. > Access denied for the process' user errno=13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9920) BufferPoolTest.WriteErrorBlacklistHolepunch failed on FindPageInDir check
[ https://issues.apache.org/jira/browse/IMPALA-9920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa updated IMPALA-9920: -- Attachment: buffer-pool-test-master-core-ubsan-56.ERROR > BufferPoolTest.WriteErrorBlacklistHolepunch failed on FindPageInDir check > - > > Key: IMPALA-9920 > URL: https://issues.apache.org/jira/browse/IMPALA-9920 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 4.0 > Environment: BUILD_TAG > jenkins-impala-cdpd-master-core-ubsan-108 >Reporter: Wenzhe Zhou >Assignee: Tim Armstrong >Priority: Major > Labels: broken-build, build, test > Attachments: buffer-pool-test-master-core-ubsan-56.ERROR, > buffer-pool-test.ERROR > > > BufferPoolTest.WriteErrorBlacklistHolepunch failed with following error > messages: > Error Message > Value of: FindPageInDir(pages[NO_ERROR_QUERY], good_dir) != NULL > Actual: false > Expected: true > Stacktrace > /data/jenkins/workspace/impala-cdpd-master-core-ubsan/repos/Impala/be/src/runtime/bufferpool/buffer-pool-test.cc:1763 > Value of: FindPageInDir(pages[NO_ERROR_QUERY], good_dir) != NULL > Actual: false > Expected: true > > Saw following messages in buffer-pool-test.ERROR log file: > F0702 14:17:08.745453 8976 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.581707 12727 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.671219 12728 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > F0702 14:17:14.840062 12940 buffer-pool.cc:216] Check failed: > page_handle->is_pinned() > F0702 14:17:15.167520 13459 buffer-pool.cc:493] Check failed: > spilling_enabled() > F0702 14:17:15.326829 13946 reservation-tracker.cc:389] Check failed: bytes > <= unused_reservation() (1024 vs. 0) > E0702 14:17:17.119957 16180 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/impala-scratch/:_354de439-8418-4013-8ebe-55214f8396c5': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/impala-scratch/:_354de439-8418-4013-8ebe-55214f8396c5. > Access denied for the process' user errno=13 > E0702 14:17:17.270885 16570 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/buffer-pool-test.0/impala-scratch/:_21e8f7c1-2a63-44d5-8a5c-4ba78e9acb6e': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/buffer-pool-test.0/impala-scratch/:_21e8f7c1-2a63-44d5-8a5c-4ba78e9acb6e. > Access denied for the process' user errno=13 > E0702 14:17:17.436445 16964 tmp-file-mgr.cc:334] Error for temporary file > '/tmp/buffer-pool-test.0/impala-scratch/:_78d2ef68-254a-4738-baf6-6e80b3b1ee2a': > Disk I/O error on > impala-ec2-centos74-m5-4xlarge-ondemand-1884.vpc.cloudera.com:22000: open() > failed for > /tmp/buffer-pool-test.0/impala-scratch/:_78d2ef68-254a-4738-baf6-6e80b3b1ee2a. > Access denied for the process' user errno=13 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9799) Flakiness in TestFetchFirst due to wrong results of get_num_in_flight_queries
[
https://issues.apache.org/jira/browse/IMPALA-9799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182831#comment-17182831
]
Norbert Luksa commented on IMPALA-9799:
---
Saw this again in
[https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-core-asan/456/]
Stacktrace:
{code:java}
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
in add_session lambda: fn(self))
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
in add_session_helper fn()
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
in lambda: fn(self))
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:120:
in test_query_stmts_v6_with_result_spooling
self.run_query_stmts_test({'spool_query_results': 'true'})
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
in run_query_stmts_test self.__test_invalid_result_caching("SELECT COUNT(*)
FROM functional.alltypes")
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
in __test_invalid_result_caching assert 0 ==
impalad.get_num_in_flight_queries() E assert 0 == 1 E + where 1 = >() E + where
> =
.get_num_in_flight_queries
{code}
> Flakiness in TestFetchFirst due to wrong results of get_num_in_flight_queries
> -
>
> Key: IMPALA-9799
> URL: https://issues.apache.org/jira/browse/IMPALA-9799
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Quanlong Huang
>Assignee: Sahil Takiar
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> Saw two failures for this test in different jenkins jobs:
> hs2.test_fetch_first.TestFetchFirst.test_query_stmts_v6 (from pytest)
> Stacktrace:
> {code:java}
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in add_session
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
> in add_session_helper
> fn()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:110:
> in test_query_stmts_v6
> self.run_query_stmts_test()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
> in run_query_stmts_test
> self.__test_invalid_result_caching("SELECT COUNT(*) FROM
> functional.alltypes")
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
> in __test_invalid_result_caching
> assert 0 == impalad.get_num_in_flight_queries()
> E assert 0 == 1
> E+ where 1 = >()
> E+where > =
> 0x6d25d10>.get_num_in_flight_queries{code}
> hs2.test_fetch_first.TestFetchFirst.test_query_stmts_v6_with_result_spooling
> (from pytest)
> Stacktrace:
> {code:java}
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in add_session
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
> in add_session_helper
> fn()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:120:
> in test_query_stmts_v6_with_result_spooling
> self.run_query_stmts_test({'spool_query_results': 'true'})
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
> in run_query_stmts_test
> self.__test_invalid_result_caching("SELECT COUNT(*) FROM
> functional.alltypes")
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
> in __test_invalid_result_caching
> assert 0 == impalad.get_num_in_flight_queries()
> E assert 0 == 1
> E+ where 1 = >()
> E+where > =
> 0x81d4990>.get_num_in_flight_queries{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-9799) Flakiness in TestFetchFirst due to wrong results of get_num_in_flight_queries
[
https://issues.apache.org/jira/browse/IMPALA-9799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182831#comment-17182831
]
Norbert Luksa edited comment on IMPALA-9799 at 8/23/20, 7:19 PM:
-
Saw this again in
[https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-core-asan/456/]
Stacktrace:
{code:java}
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
in add_session
lambda: fn(self))
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
in add_session_helper
fn()
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
in
lambda: fn(self))
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:120:
in test_query_stmts_v6_with_result_spooling
self.run_query_stmts_test({'spool_query_results': 'true'})
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
in run_query_stmts_test
self.__test_invalid_result_caching("SELECT COUNT(*) FROM
functional.alltypes")
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
in __test_invalid_result_caching
assert 0 == impalad.get_num_in_flight_queries()
E assert 0 == 1
E+ where 1 = >()
E+where > =
.get_num_in_flight_queries
{code}
was (Author: norbertluksa):
Saw this again in
[https://master-02.jenkins.cloudera.com/job/impala-cdpd-master-core-asan/456/]
Stacktrace:
{code:java}
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
in add_session lambda: fn(self))
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
in add_session_helper fn()
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
in lambda: fn(self))
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:120:
in test_query_stmts_v6_with_result_spooling
self.run_query_stmts_test({'spool_query_results': 'true'})
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
in run_query_stmts_test self.__test_invalid_result_caching("SELECT COUNT(*)
FROM functional.alltypes")
/data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
in __test_invalid_result_caching assert 0 ==
impalad.get_num_in_flight_queries() E assert 0 == 1 E + where 1 = >() E + where
> =
.get_num_in_flight_queries
{code}
> Flakiness in TestFetchFirst due to wrong results of get_num_in_flight_queries
> -
>
> Key: IMPALA-9799
> URL: https://issues.apache.org/jira/browse/IMPALA-9799
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Quanlong Huang
>Assignee: Sahil Takiar
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> Saw two failures for this test in different jenkins jobs:
> hs2.test_fetch_first.TestFetchFirst.test_query_stmts_v6 (from pytest)
> Stacktrace:
> {code:java}
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in add_session
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
> in add_session_helper
> fn()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:110:
> in test_query_stmts_v6
> self.run_query_stmts_test()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
> in run_query_stmts_test
> self.__test_invalid_result_caching("SELECT COUNT(*) FROM
> functional.alltypes")
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
> in __test_invalid_result_caching
> assert 0 == impalad.get_num_in_flight_queries()
> E assert 0 == 1
> E+ where 1 = >()
> E+where > =
> 0x6d25d10>.get_num_in_flight_queries{code}
> hs2.test_fetch_first.TestFetchFirst.test_query_stmts_v6_with_result_spooling
> (from pytest)
> Stacktrace:
> {code:java}
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in add_session
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
> in add_session_helper
> fn()
> /data/jenkins/workspace/impala-cdpd-master-core-asan
[jira] [Reopened] (IMPALA-9799) Flakiness in TestFetchFirst due to wrong results of get_num_in_flight_queries
[
https://issues.apache.org/jira/browse/IMPALA-9799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reopened IMPALA-9799:
---
> Flakiness in TestFetchFirst due to wrong results of get_num_in_flight_queries
> -
>
> Key: IMPALA-9799
> URL: https://issues.apache.org/jira/browse/IMPALA-9799
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Quanlong Huang
>Assignee: Sahil Takiar
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 4.0
>
>
> Saw two failures for this test in different jenkins jobs:
> hs2.test_fetch_first.TestFetchFirst.test_query_stmts_v6 (from pytest)
> Stacktrace:
> {code:java}
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in add_session
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
> in add_session_helper
> fn()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:110:
> in test_query_stmts_v6
> self.run_query_stmts_test()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
> in run_query_stmts_test
> self.__test_invalid_result_caching("SELECT COUNT(*) FROM
> functional.alltypes")
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
> in __test_invalid_result_caching
> assert 0 == impalad.get_num_in_flight_queries()
> E assert 0 == 1
> E+ where 1 = >()
> E+where > =
> 0x6d25d10>.get_num_in_flight_queries{code}
> hs2.test_fetch_first.TestFetchFirst.test_query_stmts_v6_with_result_spooling
> (from pytest)
> Stacktrace:
> {code:java}
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in add_session
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:44:
> in add_session_helper
> fn()
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/hs2_test_suite.py:63:
> in
> lambda: fn(self))
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:120:
> in test_query_stmts_v6_with_result_spooling
> self.run_query_stmts_test({'spool_query_results': 'true'})
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:181:
> in run_query_stmts_test
> self.__test_invalid_result_caching("SELECT COUNT(*) FROM
> functional.alltypes")
> /data/jenkins/workspace/impala-cdpd-master-core-asan/repos/Impala/tests/hs2/test_fetch_first.py:63:
> in __test_invalid_result_caching
> assert 0 == impalad.get_num_in_flight_queries()
> E assert 0 == 1
> E+ where 1 = >()
> E+where > =
> 0x81d4990>.get_num_in_flight_queries{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10054) test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182832#comment-17182832
]
Norbert Luksa commented on IMPALA-10054:
Right, that is the problem. Should we leave this Jira open until it gets
cherry-picked or we can close it now? (the current process is not entirely
clear to me)
> test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
> ---
>
> Key: IMPALA-10054
> URL: https://issues.apache.org/jira/browse/IMPALA-10054
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Attila Jeges
>Assignee: Riza Suminto
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> test_multiple_sort_run_bytes_limits introduced in IMPALA-6692 seems to be
> flaky.
> Jenkins job that triggered the error:
> https://jenkins.impala.io/job/parallel-all-tests-nightly/1173
> Failing job:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2899/testReport/
> {code}
> Stacktrace
> query_test/test_sort.py:89: in test_multiple_sort_run_bytes_limits
> assert "SpilledRuns: " + spilled_runs in query_result.runtime_profile
> E assert ('SpilledRuns: ' + '3') in 'Query
> (id=404da0b1e56e7248:120789cd):\n DEBUG MODE WARNING: Query profile
> created while running a DEBUG buil... 27.999ms\n - WriteIoBytes:
> 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime:
> 0.000ns\n'
> E+ where 'Query (id=404da0b1e56e7248:120789cd):\n DEBUG MODE
> WARNING: Query profile created while running a DEBUG buil... 27.999ms\n
> - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n -
> WriteIoWaitTime: 0.000ns\n' =
> 0x7f51da77fb50>.runtime_profile
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10054) test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188238#comment-17188238
]
Norbert Luksa commented on IMPALA-10054:
I see, thanks for clarifying!
> test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
> ---
>
> Key: IMPALA-10054
> URL: https://issues.apache.org/jira/browse/IMPALA-10054
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Attila Jeges
>Assignee: Riza Suminto
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> test_multiple_sort_run_bytes_limits introduced in IMPALA-6692 seems to be
> flaky.
> Jenkins job that triggered the error:
> https://jenkins.impala.io/job/parallel-all-tests-nightly/1173
> Failing job:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2899/testReport/
> {code}
> Stacktrace
> query_test/test_sort.py:89: in test_multiple_sort_run_bytes_limits
> assert "SpilledRuns: " + spilled_runs in query_result.runtime_profile
> E assert ('SpilledRuns: ' + '3') in 'Query
> (id=404da0b1e56e7248:120789cd):\n DEBUG MODE WARNING: Query profile
> created while running a DEBUG buil... 27.999ms\n - WriteIoBytes:
> 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime:
> 0.000ns\n'
> E+ where 'Query (id=404da0b1e56e7248:120789cd):\n DEBUG MODE
> WARNING: Query profile created while running a DEBUG buil... 27.999ms\n
> - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n -
> WriteIoWaitTime: 0.000ns\n' =
> 0x7f51da77fb50>.runtime_profile
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10054) test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
[
https://issues.apache.org/jira/browse/IMPALA-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa resolved IMPALA-10054.
Resolution: Fixed
> test_multiple_sort_run_bytes_limits fails in parallel-all-tests-nightly
> ---
>
> Key: IMPALA-10054
> URL: https://issues.apache.org/jira/browse/IMPALA-10054
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.0
>Reporter: Attila Jeges
>Assignee: Riza Suminto
>Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> test_multiple_sort_run_bytes_limits introduced in IMPALA-6692 seems to be
> flaky.
> Jenkins job that triggered the error:
> https://jenkins.impala.io/job/parallel-all-tests-nightly/1173
> Failing job:
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/2899/testReport/
> {code}
> Stacktrace
> query_test/test_sort.py:89: in test_multiple_sort_run_bytes_limits
> assert "SpilledRuns: " + spilled_runs in query_result.runtime_profile
> E assert ('SpilledRuns: ' + '3') in 'Query
> (id=404da0b1e56e7248:120789cd):\n DEBUG MODE WARNING: Query profile
> created while running a DEBUG buil... 27.999ms\n - WriteIoBytes:
> 0\n - WriteIoOps: 0 (0)\n - WriteIoWaitTime:
> 0.000ns\n'
> E+ where 'Query (id=404da0b1e56e7248:120789cd):\n DEBUG MODE
> WARNING: Query profile created while running a DEBUG buil... 27.999ms\n
> - WriteIoBytes: 0\n - WriteIoOps: 0 (0)\n -
> WriteIoWaitTime: 0.000ns\n' =
> 0x7f51da77fb50>.runtime_profile
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193532#comment-17193532
]
Norbert Luksa commented on IMPALA-9351:
---
Thanks [~stigahuang], I've reassigned the jira.
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Norbert Luksa
>Priority: Blocker
> Labels: broken-build, flaky-test
> Fix For: Impala 3.4.0
>
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
[jira] [Assigned] (IMPALA-9351) AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
[
https://issues.apache.org/jira/browse/IMPALA-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa reassigned IMPALA-9351:
-
Assignee: Quanlong Huang (was: Norbert Luksa)
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to non-existing path
> -
>
> Key: IMPALA-9351
> URL: https://issues.apache.org/jira/browse/IMPALA-9351
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Reporter: Fang-Yu Rao
>Assignee: Quanlong Huang
>Priority: Blocker
> Labels: broken-build, flaky-test
> Fix For: Impala 3.4.0
>
>
> AnalyzeDDLTest.TestCreateTableLikeFileOrc failed due to a non-existing path.
> Specifically, we see the following error message.
> {code:java}
> Error Message
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> {code}
> The stack trace is provided in the following.
> {code:java}
> Stacktrace
> java.lang.AssertionError:
> Error during analysis:
> org.apache.impala.common.AnalysisException: Cannot infer schema, path does
> not exist:
> hdfs://localhost:20500/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0
> sql:
> create table if not exists newtbl_DNE like orc
> '/test-warehouse/functional_orc_def.db/complextypes_fileformat/00_0'
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.impala.common.FrontendFixture.analyzeStmt(FrontendFixture.java:397)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:244)
> at
> org.apache.impala.common.FrontendTestBase.AnalyzesOk(FrontendTestBase.java:185)
> at
> org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLikeFileOrc(AnalyzeDDLTest.java:2045)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> {code}
> This test was recently added by [~norbertluksa], and [~boroknagyz] gave a +2,
> maybe [~boroknagyz] could provide some insight into this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h.
[jira] [Created] (IMPALA-10411) Build failed for impala-cdh-7.1-maint-exhaustive due to permission denied error
Norbert Luksa created IMPALA-10411:
--
Summary: Build failed for impala-cdh-7.1-maint-exhaustive due to
permission denied error
Key: IMPALA-10411
URL: https://issues.apache.org/jira/browse/IMPALA-10411
Project: IMPALA
Issue Type: Bug
Components: Jenkins
Reporter: Norbert Luksa
The error is:
{code:java}
19:39:47 [ERROR] Failed to execute goal on project impala-minimal-hive-exec:
Could not resolve dependencies for project
org.apache.impala:impala-minimal-hive-exec:jar:0.1-SNAPSHOT: Failed to collect
dependencies for [org.apache.hive:hive-exec:jar:3.1.3000.7.1.6.0-104
(compile)]: Failed to read artifact descriptor for
net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
net.minidev:json-smart:pom:2.3-SNAPSHOT from/to impala.cdh.repo
(https://native-toolchain.s3.amazonaws.com/build/cdh_components/1814051/maven):
Access denied to:
https://native-toolchain.s3.amazonaws.com/build/cdh_components/1814051/maven/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom,
ReasonPhrase:Forbidden. -> [Help 1] 19:39:47 [ERROR] 19:39:47 [ERROR] To see
the full stack trace of the errors, re-run Maven with the -e switch. 19:39:47
[ERROR] Re-run Maven using the -X switch to enable full debug logging. 19:39:47
[ERROR] 19:39:47 [ERROR] For more information about the errors and possible
solutions, please read the following articles: 19:39:47 [ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
19:39:47 mvn -U -s
/data/jenkins/workspace/impala-cdh-7.1-maint-exhaustive/repos/Impala-auxiliary-tests/jenkins/m2-settings.xml
-U -B install -DskipTests exited with code 0
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10411) Build failed for impala-cdh-7.1-maint-exhaustive due to access denied error
[
https://issues.apache.org/jira/browse/IMPALA-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Norbert Luksa updated IMPALA-10411:
---
Summary: Build failed for impala-cdh-7.1-maint-exhaustive due to access
denied error (was: Build failed for impala-cdh-7.1-maint-exhaustive due to
permission denied error)
> Build failed for impala-cdh-7.1-maint-exhaustive due to access denied error
> ---
>
> Key: IMPALA-10411
> URL: https://issues.apache.org/jira/browse/IMPALA-10411
> Project: IMPALA
> Issue Type: Bug
> Components: Jenkins
>Reporter: Norbert Luksa
>Priority: Major
> Labels: broken-build
>
> The error is:
> {code:java}
> 19:39:47 [ERROR] Failed to execute goal on project impala-minimal-hive-exec:
> Could not resolve dependencies for project
> org.apache.impala:impala-minimal-hive-exec:jar:0.1-SNAPSHOT: Failed to
> collect dependencies for [org.apache.hive:hive-exec:jar:3.1.3000.7.1.6.0-104
> (compile)]: Failed to read artifact descriptor for
> net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
> net.minidev:json-smart:pom:2.3-SNAPSHOT from/to impala.cdh.repo
> (https://native-toolchain.s3.amazonaws.com/build/cdh_components/1814051/maven):
> Access denied to:
> https://native-toolchain.s3.amazonaws.com/build/cdh_components/1814051/maven/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom,
> ReasonPhrase:Forbidden. -> [Help 1] 19:39:47 [ERROR] 19:39:47 [ERROR] To see
> the full stack trace of the errors, re-run Maven with the -e switch. 19:39:47
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> 19:39:47 [ERROR] 19:39:47 [ERROR] For more information about the errors and
> possible solutions, please read the following articles: 19:39:47 [ERROR]
> [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
> 19:39:47 mvn -U -s
> /data/jenkins/workspace/impala-cdh-7.1-maint-exhaustive/repos/Impala-auxiliary-tests/jenkins/m2-settings.xml
> -U -B install -DskipTests exited with code 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12403) Kerberos authentication fails when connecting with a proxy user that passes LDAP user and group filters but does not delegate another user
[ https://issues.apache.org/jira/browse/IMPALA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-12403. Resolution: Fixed > Kerberos authentication fails when connecting with a proxy user that passes > LDAP user and group filters but does not delegate another user > -- > > Key: IMPALA-12403 > URL: https://issues.apache.org/jira/browse/IMPALA-12403 > Project: IMPALA > Issue Type: Bug > Components: be >Reporter: Gergely Farkas >Assignee: Gergely Farkas >Priority: Major > > When connecting with a proxy user without _doAs_ request parameter or > _impala.doas.user_ connection config then the filters are executed with the > authenticated user itself, however, in case of Kerberos auth, the > authenticated user is a Kerberos user principal which will definitely not > pass the LDAP checks, because LDAP filters here need to be checked with a > short username (that needs to be extracted from the Kerberos user principal). > During the Kerberos authentication process, the short username is checked ( > see > [https://github.com/apache/impala/blob/master/be/src/rpc/authentication.cc#L757-L764]), > , the only point where it doesn't work like that is this: > [https://github.com/apache/impala/blob/master/be/src/service/impala-hs2-server.cc#L394-L403] > [https://github.com/apache/impala/blob/master/be/src/util/auth-util.cc#L43-L52] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12403) Kerberos authentication fails when connecting with a proxy user that passes LDAP user and group filters but does not delegate another user
[ https://issues.apache.org/jira/browse/IMPALA-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17765482#comment-17765482 ] Norbert Luksa commented on IMPALA-12403: Change is merged: https://gerrit.cloudera.org/#/c/20421/ > Kerberos authentication fails when connecting with a proxy user that passes > LDAP user and group filters but does not delegate another user > -- > > Key: IMPALA-12403 > URL: https://issues.apache.org/jira/browse/IMPALA-12403 > Project: IMPALA > Issue Type: Bug > Components: be >Reporter: Gergely Farkas >Assignee: Gergely Farkas >Priority: Major > > When connecting with a proxy user without _doAs_ request parameter or > _impala.doas.user_ connection config then the filters are executed with the > authenticated user itself, however, in case of Kerberos auth, the > authenticated user is a Kerberos user principal which will definitely not > pass the LDAP checks, because LDAP filters here need to be checked with a > short username (that needs to be extracted from the Kerberos user principal). > During the Kerberos authentication process, the short username is checked ( > see > [https://github.com/apache/impala/blob/master/be/src/rpc/authentication.cc#L757-L764]), > , the only point where it doesn't work like that is this: > [https://github.com/apache/impala/blob/master/be/src/service/impala-hs2-server.cc#L394-L403] > [https://github.com/apache/impala/blob/master/be/src/util/auth-util.cc#L43-L52] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8752) Add Jaro-winkler edit distance and similarity built-in function
Norbert Luksa created IMPALA-8752: - Summary: Add Jaro-winkler edit distance and similarity built-in function Key: IMPALA-8752 URL: https://issues.apache.org/jira/browse/IMPALA-8752 Project: IMPALA Issue Type: New Feature Reporter: Norbert Luksa Assignee: Norbert Luksa References: * [Apache commons - JaroWinklerDistance |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerDistance.html]] * [Apache commons - JaroWinklerSimilarity |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerSimilarity.html]] * [Oracle - JARO_WINKLER[_SIMILARITY]|[https://oracle-base.com/articles/11g/utl_match-string-matching-in-oracle]] Notable difference: * With similarity, the Oracle version returns a normalized result ranging from 0 to 100. * In the Appache version, null values result in exceptions. * Apache rounds the values to two digits The scaling factor of the algorithm can be added as an extra/default argument. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8755) Implement Z-ordering for Impala
[ https://issues.apache.org/jira/browse/IMPALA-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8755 started by Norbert Luksa. - > Implement Z-ordering for Impala > --- > > Key: IMPALA-8755 > URL: https://issues.apache.org/jira/browse/IMPALA-8755 > Project: IMPALA > Issue Type: New Feature >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > > Implement Z-ordering for Impala: [https://en.wikipedia.org/wiki/Z-order_curve] > A Z-order curve defines an ordering on multi-dimensional data. Data sorted > that way can be efficiently filtered by min/max statistics regarding to the > columns participating in the ordering. > Impala currently only supports lexicographic ordering via the SORT BY clause. > This strongly prefers the first column, i.e. given the "SORT BY A, B, C" > clause => A will be totally ordered (hence filtering on A will be very > efficient), but values belonging to B and C will be scattered throughout the > data set (hence filtering on B or C will barely do any good). > We could add a new clause, e.g. a "ZSORT BY" clause to Impala that writes the > data in Z-order. > "ZSORT BY A, B C" would cluster the rows in a way that filtering on A, B, or > C would be equally efficient. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8752) Add Jaro-winkler edit distance and similarity built-in function
[ https://issues.apache.org/jira/browse/IMPALA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8752 started by Norbert Luksa. - > Add Jaro-winkler edit distance and similarity built-in function > --- > > Key: IMPALA-8752 > URL: https://issues.apache.org/jira/browse/IMPALA-8752 > Project: IMPALA > Issue Type: New Feature >Reporter: Norbert Luksa >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > References: > * [Apache commons - JaroWinklerDistance > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerDistance.html]] > * [Apache commons - JaroWinklerSimilarity > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerSimilarity.html]] > * [Oracle - > JARO_WINKLER[_SIMILARITY]|[https://oracle-base.com/articles/11g/utl_match-string-matching-in-oracle]] > Notable difference: > * With similarity, the Oracle version returns a normalized result ranging > from 0 to 100. > * In the Appache version, null values result in exceptions. > * Apache rounds the values to two digits > The scaling factor of the algorithm can be added as an extra/default argument. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7770) SPLIT_PART to support negative indexes
[ https://issues.apache.org/jira/browse/IMPALA-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa reassigned IMPALA-7770: - Assignee: Norbert Luksa > SPLIT_PART to support negative indexes > -- > > Key: IMPALA-7770 > URL: https://issues.apache.org/jira/browse/IMPALA-7770 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tristan Stevens >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > Request is for SPLIT_PART to support negative indexes i.e. support right to > left searching. > See Snowflake documentation for details: > [https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html|https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html] > partNr: Requested part of the split (1-based). 0 is treated as 1. If the > value is negative, the parts are counted from the right side of the string. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7770) SPLIT_PART to support negative indexes
[ https://issues.apache.org/jira/browse/IMPALA-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896104#comment-16896104 ] Norbert Luksa commented on IMPALA-7770: --- Feel free to review: https://gerrit.cloudera.org/#/c/13880/ > SPLIT_PART to support negative indexes > -- > > Key: IMPALA-7770 > URL: https://issues.apache.org/jira/browse/IMPALA-7770 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tristan Stevens >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > Request is for SPLIT_PART to support negative indexes i.e. support right to > left searching. > See Snowflake documentation for details: > [https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html|https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html] > partNr: Requested part of the split (1-based). 0 is treated as 1. If the > value is negative, the parts are counted from the right side of the string. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8752) Add Jaro-winkler edit distance and similarity built-in function
[ https://issues.apache.org/jira/browse/IMPALA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896106#comment-16896106 ] Norbert Luksa commented on IMPALA-8752: --- https://gerrit.cloudera.org/#/c/13870/ > Add Jaro-winkler edit distance and similarity built-in function > --- > > Key: IMPALA-8752 > URL: https://issues.apache.org/jira/browse/IMPALA-8752 > Project: IMPALA > Issue Type: New Feature >Reporter: Norbert Luksa >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > References: > * [Apache commons - JaroWinklerDistance > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerDistance.html]] > * [Apache commons - JaroWinklerSimilarity > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerSimilarity.html]] > * [Oracle - > JARO_WINKLER[_SIMILARITY]|[https://oracle-base.com/articles/11g/utl_match-string-matching-in-oracle]] > Notable difference: > * With similarity, the Oracle version returns a normalized result ranging > from 0 to 100. > * In the Appache version, null values result in exceptions. > * Apache rounds the values to two digits > The scaling factor of the algorithm can be added as an extra/default argument. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7770) SPLIT_PART to support negative indexes
[ https://issues.apache.org/jira/browse/IMPALA-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896930#comment-16896930 ] Norbert Luksa commented on IMPALA-7770: --- Yes, it's only a small change. > SPLIT_PART to support negative indexes > -- > > Key: IMPALA-7770 > URL: https://issues.apache.org/jira/browse/IMPALA-7770 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tristan Stevens >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > > Request is for SPLIT_PART to support negative indexes i.e. support right to > left searching. > See Snowflake documentation for details: > [https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html|https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html] > partNr: Requested part of the split (1-based). 0 is treated as 1. If the > value is negative, the parts are counted from the right side of the string. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8752) Add Jaro-winkler edit distance and similarity built-in function
[ https://issues.apache.org/jira/browse/IMPALA-8752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-8752. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Add Jaro-winkler edit distance and similarity built-in function > --- > > Key: IMPALA-8752 > URL: https://issues.apache.org/jira/browse/IMPALA-8752 > Project: IMPALA > Issue Type: New Feature >Reporter: Norbert Luksa >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > Fix For: Impala 3.3.0 > > > References: > * [Apache commons - JaroWinklerDistance > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerDistance.html]] > * [Apache commons - JaroWinklerSimilarity > |[https://commons.apache.org/proper/commons-text/apidocs/org/apache/commons/text/similarity/JaroWinklerSimilarity.html]] > * [Oracle - > JARO_WINKLER[_SIMILARITY]|[https://oracle-base.com/articles/11g/utl_match-string-matching-in-oracle]] > Notable difference: > * With similarity, the Oracle version returns a normalized result ranging > from 0 to 100. > * In the Appache version, null values result in exceptions. > * Apache rounds the values to two digits > The scaling factor of the algorithm can be added as an extra/default argument. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7770) SPLIT_PART to support negative indexes
[ https://issues.apache.org/jira/browse/IMPALA-7770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Norbert Luksa resolved IMPALA-7770. --- Resolution: Implemented Fix Version/s: Impala 3.3.0 > SPLIT_PART to support negative indexes > -- > > Key: IMPALA-7770 > URL: https://issues.apache.org/jira/browse/IMPALA-7770 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Tristan Stevens >Assignee: Norbert Luksa >Priority: Major > Labels: built-in-function > Fix For: Impala 3.3.0 > > > Request is for SPLIT_PART to support negative indexes i.e. support right to > left searching. > See Snowflake documentation for details: > [https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html|https://docs.snowflake.net/manuals/sql-reference/functions/split_part.html] > partNr: Requested part of the split (1-based). 0 is treated as 1. If the > value is negative, the parts are counted from the right side of the string. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8498) Write column index for floating types when NaN is not present
[ https://issues.apache.org/jira/browse/IMPALA-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8498 started by Norbert Luksa. - > Write column index for floating types when NaN is not present > - > > Key: IMPALA-8498 > URL: https://issues.apache.org/jira/browse/IMPALA-8498 > Project: IMPALA > Issue Type: Bug >Reporter: Zoltán Borók-Nagy >Assignee: Norbert Luksa >Priority: Major > Labels: ramp-up > > IMPALA-7304 disabled column index writing for floating point columns until > PARQUET-1222 is resolved. > PARQUET-1222 is responsible for defining a total order for floating values, > but the problematic values are only the NaNs. Therefore we can write the > column index if NaNs are not present in the data. Parquet-MR also does this, > following the principles in > [https://github.com/apache/parquet-format/blob/75eb7a7b84e6e62bfb09668b6d8d40b12597456e/src/main/thrift/parquet.thrift#L827-L834] > > Impala should follow this behavior, and also when storing zeroes, it should > store -0.0 as minimum and +0.0 as maximum. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
