[jira] [Updated] (IMPALA-10365) test_page_index failed in exhaustive release build
[
https://issues.apache.org/jira/browse/IMPALA-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-10365:
---
Labels: broken-build correctness (was: correctness)
> test_page_index failed in exhaustive release build
> --
>
> Key: IMPALA-10365
> URL: https://issues.apache.org/jira/browse/IMPALA-10365
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Zoltán Borók-Nagy
>Priority: Major
> Labels: broken-build, correctness
>
> query_test.test_parquet_stats.TestParquetStats.test_page_index failed in
> exhaustive release build.
> *Stack Trace*
> {noformat}
> query_test/test_parquet_stats.py:98: in test_page_index
> unique_database)
> common/impala_test_suite.py:693: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:529: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 'AIR','1-URGENT',4 != 'AIR','1-URGENT',3
> E 'AIR','2-HIGH',3 != 'AIR','2-HIGH',2
> E 'AIR','3-MEDIUM',4 == 'AIR','3-MEDIUM',4
> E 'AIR','4-NOT SPECIFIED',2 == 'AIR','4-NOT SPECIFIED',2
> E 'AIR','5-LOW',1 == 'AIR','5-LOW',1
> E 'FOB','1-URGENT',3 == 'FOB','1-URGENT',3
> E 'FOB','2-HIGH',1 != 'FOB','3-MEDIUM',2
> E 'FOB','3-MEDIUM',2 != 'FOB','4-NOT SPECIFIED',3
> E 'FOB','4-NOT SPECIFIED',3 != 'FOB','5-LOW',1
> E 'FOB','5-LOW',2 != 'MAIL','2-HIGH',2
> E 'MAIL','2-HIGH',3 != 'MAIL','3-MEDIUM',3
> E 'MAIL','3-MEDIUM',3 != 'MAIL','4-NOT SPECIFIED',2
> E 'MAIL','4-NOT SPECIFIED',3 != 'MAIL','5-LOW',1
> E 'MAIL','5-LOW',1 != 'RAIL','2-HIGH',1
> E 'RAIL','2-HIGH',1 != 'RAIL','3-MEDIUM',2
> E 'RAIL','3-MEDIUM',2 != 'RAIL','4-NOT SPECIFIED',1
> E 'RAIL','4-NOT SPECIFIED',1 != 'RAIL','5-LOW',1
> E 'RAIL','5-LOW',1 != 'REG AIR','2-HIGH',3
> E 'REG AIR','2-HIGH',4 != 'REG AIR','3-MEDIUM',1
> E 'REG AIR','3-MEDIUM',2 != 'REG AIR','5-LOW',2
> E 'REG AIR','5-LOW',2 != 'SHIP','1-URGENT',3
> E 'SHIP','1-URGENT',4 != 'SHIP','2-HIGH',2
> E 'SHIP','2-HIGH',2 != 'SHIP','4-NOT SPECIFIED',2
> E 'SHIP','4-NOT SPECIFIED',2 != 'TRUCK','3-MEDIUM',2
> E 'TRUCK','3-MEDIUM',2 != 'TRUCK','5-LOW',2
> E 'TRUCK','5-LOW',3 != None
> E Number of rows returned (expected vs actual): 26 != 25{noformat}
> After adjusting the lines:
> {noformat}
> E 'AIR','1-URGENT',4 != 'AIR','1-URGENT',3
> E 'AIR','2-HIGH',3 != 'AIR','2-HIGH',2
> E 'AIR','3-MEDIUM',4 == 'AIR','3-MEDIUM',4
> E 'AIR','4-NOT SPECIFIED',2 == 'AIR','4-NOT SPECIFIED',2
> E 'AIR','5-LOW',1 == 'AIR','5-LOW',1
> E 'FOB','1-URGENT',3 == 'FOB','1-URGENT',3
> E 'FOB','2-HIGH',1 != None
> E 'FOB','3-MEDIUM',2 == 'FOB','3-MEDIUM',2
> E 'FOB','4-NOT SPECIFIED',3 == 'FOB','4-NOT SPECIFIED',3
> E 'FOB','5-LOW',2 != 'FOB','5-LOW',1
> E 'MAIL','2-HIGH',3 != 'MAIL','2-HIGH',2
> E 'MAIL','3-MEDIUM',3 == 'MAIL','3-MEDIUM',3
> E 'MAIL','4-NOT SPECIFIED',3 != 'MAIL','4-NOT SPECIFIED',2
> E 'MAIL','5-LOW',1 == 'MAIL','5-LOW',1
> E 'RAIL','2-HIGH',1 == 'RAIL','2-HIGH',1
> E 'RAIL','3-MEDIUM',2 == 'RAIL','3-MEDIUM',2
> E 'RAIL','4-NOT SPECIFIED',1 == 'RAIL','4-NOT SPECIFIED',1
> E 'RAIL','5-LOW',1 == 'RAIL','5-LOW',1
> E 'REG AIR','2-HIGH',4 != 'REG AIR','2-HIGH',3
> E 'REG AIR','3-MEDIUM',2 != 'REG AIR','3-MEDIUM',1
> E 'REG AIR','5-LOW',2 == 'REG AIR','5-LOW',2
> E 'SHIP','1-URGENT',4 != 'SHIP','1-URGENT',3
> E 'SHIP','2-HIGH',2 == 'SHIP','2-HIGH',2
> E 'SHIP','4-NOT SPECIFIED',2 == 'SHIP','4-NOT SPECIFIED',2
> E 'TRUCK','3-MEDIUM',2 == 'TRUCK','3-MEDIUM',2
> E 'TRUCK','5-LOW',3 != 'TRUCK','5-LOW',2{noformat}
> The query was:
> {noformat}
> select l_shipmode, o_orderpriority, count(*)
> from tpch_nested_parquet.customer.c_orders o, o.o_lineitems l
> where l_receiptdate < '1992-01-10'
> group by l_shipmode, o_orderpriority{noformat}
> Nested tables are written by Hive, so it must have been a reader-side problem
> that also passed
> HdfsParquetScanner::CheckPageFiltering().
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10368) Support required/optional property when creating Iceberg table by Impala
[ https://issues.apache.org/jira/browse/IMPALA-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241434#comment-17241434 ] Zoltán Borók-Nagy commented on IMPALA-10368: I like the idea. I think it would be nice to synchronize across the engines. On the Iceberg dev mailing list there's an ongoing discussion under the subject "Iceberg - Hive schema synchronization". I think it would be good to coin this idea in that thread. > Support required/optional property when creating Iceberg table by Impala > > > Key: IMPALA-10368 > URL: https://issues.apache.org/jira/browse/IMPALA-10368 > Project: IMPALA > Issue Type: Improvement >Reporter: Sheng Wang >Assignee: Sheng Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10362) FE test testShellLdapAuth() seems to be flaky
[
https://issues.apache.org/jira/browse/IMPALA-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Jeges resolved IMPALA-10362.
---
Resolution: Fixed
> FE test testShellLdapAuth() seems to be flaky
> -
>
> Key: IMPALA-10362
> URL: https://issues.apache.org/jira/browse/IMPALA-10362
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 4.0
>Reporter: Fang-Yu Rao
>Assignee: Attila Jeges
>Priority: Major
> Labels: broken-build, flaky-test
> Attachments: org.apache.impala.customcluster.LdapImpalaShellTest.txt
>
>
> We found that sometimes {{LdapImpalaShellTest#testShellLdapAuth()}} failed at
> [https://github.com/apache/impala/blob/1c72c5a8f93ae3aa7e30b8bb59e0a18a654d0d34/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java#L146]
> due to hitting an {{assertTrue()}} at
> [https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/customcluster/RunShellCommand.java#L50].
> This test was slightly revised in IMPALA-10234. Maybe [~attilaj] could offer
> some insight into it.
> The corresponding log file is also provided. Let me know if other log files
> are needed as well. Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10362) FE test testShellLdapAuth() seems to be flaky
[
https://issues.apache.org/jira/browse/IMPALA-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Jeges updated IMPALA-10362:
--
Fix Version/s: Impala 4.0
> FE test testShellLdapAuth() seems to be flaky
> -
>
> Key: IMPALA-10362
> URL: https://issues.apache.org/jira/browse/IMPALA-10362
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 4.0
>Reporter: Fang-Yu Rao
>Assignee: Attila Jeges
>Priority: Major
> Labels: broken-build, flaky-test
> Fix For: Impala 4.0
>
> Attachments: org.apache.impala.customcluster.LdapImpalaShellTest.txt
>
>
> We found that sometimes {{LdapImpalaShellTest#testShellLdapAuth()}} failed at
> [https://github.com/apache/impala/blob/1c72c5a8f93ae3aa7e30b8bb59e0a18a654d0d34/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java#L146]
> due to hitting an {{assertTrue()}} at
> [https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/customcluster/RunShellCommand.java#L50].
> This test was slightly revised in IMPALA-10234. Maybe [~attilaj] could offer
> some insight into it.
> The corresponding log file is also provided. Let me know if other log files
> are needed as well. Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10280) TestConcurrentDdls.test_concurrent_invalidate_metadata hits INVALIDATE METADATA timeout on s3
[
https://issues.apache.org/jira/browse/IMPALA-10280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241471#comment-17241471
]
Quanlong Huang commented on IMPALA-10280:
-
Thank [~joemcdonnell]! Uploaded a patch for dumping the stacktraces:
[https://gerrit.cloudera.org/c/16800/]
> TestConcurrentDdls.test_concurrent_invalidate_metadata hits INVALIDATE
> METADATA timeout on s3
> -
>
> Key: IMPALA-10280
> URL: https://issues.apache.org/jira/browse/IMPALA-10280
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 4.0
>Reporter: Joe McDonnell
>Assignee: Quanlong Huang
>Priority: Critical
> Labels: broken-build, flaky
> Attachments:
> catalogd.impala-ec2-centos74-m5-4xlarge-ondemand-159c.vpc.cloudera.com.jenkins.log.INFO.20201023-072544.3234,
> hive-metastore.log.gz,
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-159c.vpc.cloudera.com.jenkins.log.INFO.20201023-072544.3301,
>
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-159c.vpc.cloudera.com.jenkins.log.INFO.20201023-072544.3304,
>
> impalad.impala-ec2-centos74-m5-4xlarge-ondemand-159c.vpc.cloudera.com.jenkins.log.INFO.20201023-072544.3313,
>
> statestored.impala-ec2-centos74-m5-4xlarge-ondemand-159c.vpc.cloudera.com.jenkins.log.INFO.20201023-072544.3218
>
>
> A recent test run on s3 hit the following test failure:
> {noformat}
> custom_cluster/test_concurrent_ddls.py:194: in
> test_concurrent_invalidate_metadata
> assert False, "INVALIDATE METADATA timeout in 60s!"
> E AssertionError: INVALIDATE METADATA timeout in 60s!
> E assert False{noformat}
> This seems to be distinct from IMPALA-10077, as this is an actual timeout on
> the INVALIDATE METADATA, not the test as a whole.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10152) Add support for Iceberg HiveCatalog
[ https://issues.apache.org/jira/browse/IMPALA-10152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved IMPALA-10152. Fix Version/s: Impala 4.0 Resolution: Fixed > Add support for Iceberg HiveCatalog > --- > > Key: IMPALA-10152 > URL: https://issues.apache.org/jira/browse/IMPALA-10152 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Labels: impala-iceberg > Fix For: Impala 4.0 > > > HadoopTables and HadoopCatalog only work on filesystems that support atomic > rename. Therefore they are not safe on S3. > HiveCatalog uses the Hive Metastore to keep track of Iceberg tables, so we > should use this interface when we store our tables in object stores. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9923) Data loading of TPC-DS ORC fails with "Fail to get checksum"
[
https://issues.apache.org/jira/browse/IMPALA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy resolved IMPALA-9923.
---
Fix Version/s: Impala 4.0
Resolution: Fixed
Haven't seen this since the workaround, closing this.
> Data loading of TPC-DS ORC fails with "Fail to get checksum"
>
>
> Key: IMPALA-9923
> URL: https://issues.apache.org/jira/browse/IMPALA-9923
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Zoltán Borók-Nagy
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
> Attachments: load-tpcds-core-hive-generated-orc-def-block.sql,
> load-tpcds-core-hive-generated-orc-def-block.sql.log
>
>
> {noformat}
> INFO : Loading data to table tpcds_orc_def.store_sales partition
> (ss_sold_date_sk=null) from
> hdfs://localhost:20500/test-warehouse/managed/tpcds.store_sales_orc_def
> INFO :
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get
> checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> INFO : Completed executing
> command(queryId=ubuntu_20200707055650_a1958916-1e85-4db5-b1bc-cc63d80b3537);
> Time taken: 14.512 seconds
> INFO : OK
> Error: Error while compiling statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to
> get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction. (state=08S01,code=1)
> java.sql.SQLException: Error while compiling statement: FAILED: Execution
> Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
> java.io.IOException: Fail to get checksum, since file
> /test-warehouse/managed/tpcds.store_sales_orc_def/ss_sold_date_sk=2451646/base_003/_orc_acid_version
> is under construction.
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1007)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1217)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1146)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1497)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1355)
> at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1329)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1127)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
> at
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
> {noformat}
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11223/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10243) ConcurrentModificationException during parallel INSERTs
[ https://issues.apache.org/jira/browse/IMPALA-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved IMPALA-10243. Fix Version/s: Impala 4.0 Resolution: Fixed > ConcurrentModificationException during parallel INSERTs > --- > > Key: IMPALA-10243 > URL: https://issues.apache.org/jira/browse/IMPALA-10243 > Project: IMPALA > Issue Type: Bug >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Fix For: Impala 4.0 > > > Impala might throw a ConcurrentModificationException during a high load of > INSERTs to the same table. > The exception happens during thrift serialization of TUpdateCatalogResponse > which have a reference to the metastore table. The serialization happens > without a lock, so another thread might modify the metastore table object in > the meantime. This can potentially happen in > CatalogOpExecutor.updateCatalog() which updates the catalog version and > unsets table column statistics. A high load of INSERT statements increase the > probability of the concurrent modification. > I think the problem is that in Table.toThrift() we set a reference to the > metastore table object instead of deep copying it: > [https://github.com/apache/impala/blob/481ea4ab0d476a4aa491f99c2a4e376faddc0b03/fe/src/main/java/org/apache/impala/catalog/Table.java#L505] > The stack trace looks like the following: > [1] java.util.HashMap$HashIterator.nextNode (HashMap.java:1,445) > [2] java.util.HashMap$EntryIterator.next (HashMap.java:1,479) > [3] java.util.HashMap$EntryIterator.next (HashMap.java:1,477) > [4] org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.write > (Table.java:2,641) > [5] org.apache.hadoop.hive.metastore.api.Table$TableStandardScheme.write > (Table.java:2,324) > [6] org.apache.hadoop.hive.metastore.api.Table.write (Table.java:2,082) > [7] org.apache.impala.thrift.TTable$TTableStandardScheme.write > (TTable.java:1,829) > [8] org.apache.impala.thrift.TTable$TTableStandardScheme.write > (TTable.java:1,569) > [9] org.apache.impala.thrift.TTable.write (TTable.java:1,357) > [10] > org.apache.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write > (TCatalogObject.java:1,433) > [11] > org.apache.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write > (TCatalogObject.java:1,272) > [12] org.apache.impala.thrift.TCatalogObject.write > (TCatalogObject.java:1,086) > [13] > org.apache.impala.thrift.TCatalogUpdateResult$TCatalogUpdateResultStandardScheme.write > (TCatalogUpdateResult.java:908) > [14] > org.apache.impala.thrift.TCatalogUpdateResult$TCatalogUpdateResultStandardScheme.write > (TCatalogUpdateResult.java:780) > [15] org.apache.impala.thrift.TCatalogUpdateResult.write > (TCatalogUpdateResult.java:682) > [16] > org.apache.impala.thrift.TUpdateCatalogResponse$TUpdateCatalogResponseStandardScheme.write > (TUpdateCatalogResponse.java:363) > [17] > org.apache.impala.thrift.TUpdateCatalogResponse$TUpdateCatalogResponseStandardScheme.write > (TUpdateCatalogResponse.java:325) > [18] org.apache.impala.thrift.TUpdateCatalogResponse.write > (TUpdateCatalogResponse.java:273) > [19] org.apache.thrift.TSerializer.serialize (TSerializer.java:79) > [20] org.apache.impala.service.JniCatalog.updateCatalog (JniCatalog.java:314) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10209) Iceberg table doesn't set object_catalog_version for the response object in doGetPartialCatalogObject
[ https://issues.apache.org/jira/browse/IMPALA-10209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved IMPALA-10209. Resolution: Fixed Fixed by IMPALA-10215. > Iceberg table doesn't set object_catalog_version for the response object in > doGetPartialCatalogObject > - > > Key: IMPALA-10209 > URL: https://issues.apache.org/jira/browse/IMPALA-10209 > Project: IMPALA > Issue Type: Bug >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Labels: impala-iceberg > > Iceberg tables have an underlying hdfs table member. When the IcebergTable > object gets a new catalog version it doesn't set the version number for the > underlying HdfsTable. > We use the HdfsTable member to answer > CatalogServiceCatalog.doGetPartialCatalogObject() for Iceberg tables, so the > response object will always have object_version_number == 0. This leads to > various errors and strange behavior as the cache invalidation in local > catalog depends on this value. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-10366) TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
[
https://issues.apache.org/jira/browse/IMPALA-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10366.
Fix Version/s: Impala 4.0
Resolution: Fixed
> TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
>
>
> Key: IMPALA-10366
> URL: https://issues.apache.org/jira/browse/IMPALA-10366
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.4.0
>Reporter: Aman Sinha
>Assignee: Tim Armstrong
>Priority: Major
> Fix For: Impala 4.0
>
>
> Test failure on an impala-asf-master-core-erasure-coding build:
> {noformat}
> custom_cluster.test_runtime_profile.TestRuntimeProfile.test_runtime_profile_aggregated[protocol:
> beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none
> {noformat}
> The output is very verbose. Here's the first part of the Stacktrace:
> {noformat}
> custom_cluster/test_runtime_profile.py:46: in test_runtime_profile_aggregated
> self.run_test_case('runtime-profile-aggregated', vector)
> common/impala_test_suite.py:718: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:612: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*NumBackends: 4 .*
> E row_regex: .*Fragment F00 \[4 instances\].*
> E row_regex: .*Table Name\[0-3\]: tpch.*lineitem.*
> E
> E ACTUAL PROFILE:
> E Query (id=7b4476dc2da1ab29:ca76ce49):
> E DEBUG MODE WARNING: Query profile created while running a DEBUG build
> of Impala. Use RELEASE builds to measure query performance.
> E - InactiveTotalTime: 0.000ns
> E - TotalTime: 0.000ns
> E Summary:
> E Session ID: d94934f8d0ba24cb:79a237b3d0ef5a9a
> E Session Type: BEESWAX
> E Start Time: 2020-11-28 21:44:31.917844000
> E End Time: 2020-11-28 21:44:36.293303000
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.0.0-SNAPSHOT DEBUG (build
> a4cf449c88ef3fe08db9abbad82664b99382014c)
> E User: jenkins
> E Connected User: jenkins
> E Delegated User:
> E Network Address: ::1:45158
> E Default Db: tpch
> E Sql Statement: select STRAIGHT_JOIN count(distinct l_partkey)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10366) TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
[
https://issues.apache.org/jira/browse/IMPALA-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241682#comment-17241682
]
ASF subversion and git services commented on IMPALA-10366:
--
Commit 62c19e63396c1e783e93b4e7fb71804b70955e89 in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=62c19e6 ]
IMPALA-10366: skip test_runtime_profile_aggregated for EC
The schedule for erasure coded data results in 3 instead
of 4 instances of the fragment with the scan. Skip the
test - we don't need special coverage for erasure coding.
Change-Id: I2bb47d89f6d6c59242f2632c481f26d93e28e33e
Reviewed-on: http://gerrit.cloudera.org:8080/16799
Reviewed-by: Aman Sinha
Tested-by: Impala Public Jenkins
> TestRuntimeProfile.test_runtime_profile_aggregated failed on master core run
>
>
> Key: IMPALA-10366
> URL: https://issues.apache.org/jira/browse/IMPALA-10366
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
>Affects Versions: Impala 3.4.0
>Reporter: Aman Sinha
>Assignee: Tim Armstrong
>Priority: Major
>
> Test failure on an impala-asf-master-core-erasure-coding build:
> {noformat}
> custom_cluster.test_runtime_profile.TestRuntimeProfile.test_runtime_profile_aggregated[protocol:
> beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none
> {noformat}
> The output is very verbose. Here's the first part of the Stacktrace:
> {noformat}
> custom_cluster/test_runtime_profile.py:46: in test_runtime_profile_aggregated
> self.run_test_case('runtime-profile-aggregated', vector)
> common/impala_test_suite.py:718: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:612: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*NumBackends: 4 .*
> E row_regex: .*Fragment F00 \[4 instances\].*
> E row_regex: .*Table Name\[0-3\]: tpch.*lineitem.*
> E
> E ACTUAL PROFILE:
> E Query (id=7b4476dc2da1ab29:ca76ce49):
> E DEBUG MODE WARNING: Query profile created while running a DEBUG build
> of Impala. Use RELEASE builds to measure query performance.
> E - InactiveTotalTime: 0.000ns
> E - TotalTime: 0.000ns
> E Summary:
> E Session ID: d94934f8d0ba24cb:79a237b3d0ef5a9a
> E Session Type: BEESWAX
> E Start Time: 2020-11-28 21:44:31.917844000
> E End Time: 2020-11-28 21:44:36.293303000
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.0.0-SNAPSHOT DEBUG (build
> a4cf449c88ef3fe08db9abbad82664b99382014c)
> E User: jenkins
> E Connected User: jenkins
> E Delegated User:
> E Network Address: ::1:45158
> E Default Db: tpch
> E Sql Statement: select STRAIGHT_JOIN count(distinct l_partkey)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-9355) TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit
[
https://issues.apache.org/jira/browse/IMPALA-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qifan Chen resolved IMPALA-9355.
Fix Version/s: Impala 4.0
Resolution: Fixed
> TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory
> limit
> -
>
> Key: IMPALA-9355
> URL: https://issues.apache.org/jira/browse/IMPALA-9355
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Fang-Yu Rao
>Assignee: Qifan Chen
>Priority: Critical
> Labels: broken-build, flaky
> Fix For: Impala 4.0
>
>
> The EE test {{test_exchange_mem_usage_scaling}} failed because the query at
> [https://github.com/apache/impala/blame/master/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test#L7-L15]
> does not hit the specified memory limit (170m) at
> [https://github.com/apache/impala/blame/master/testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test#L7].
> We may need to further reduce the specified limit. In what follows the error
> message is also given. Recall that the same issue occurred at
> https://issues.apache.org/jira/browse/IMPALA-7873 but was resolved.
> {code:java}
> FAIL
> query_test/test_mem_usage_scaling.py::TestExchangeMemUsage::()::test_exchange_mem_usage_scaling[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> === FAILURES
> ===
> TestExchangeMemUsage.test_exchange_mem_usage_scaling[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> [gw3] linux2 -- Python 2.7.12
> /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> query_test/test_mem_usage_scaling.py:386: in test_exchange_mem_usage_scaling
> self.run_test_case('QueryTest/exchange-mem-scaling', vector)
> common/impala_test_suite.py:674: in run_test_case
> expected_str, query)
> E AssertionError: Expected exception: Memory limit exceeded
> E
> E when running:
> E
> E set mem_limit=170m;
> E set num_scanner_threads=1;
> E select *
> E from tpch_parquet.lineitem l1
> E join tpch_parquet.lineitem l2 on l1.l_orderkey = l2.l_orderkey and
> E l1.l_partkey = l2.l_partkey and l1.l_suppkey = l2.l_suppkey
> E and l1.l_linenumber = l2.l_linenumber
> E order by l1.l_orderkey desc, l1.l_partkey, l1.l_suppkey, l1.l_linenumber
> E limit 5
> {code}
> [~tarmstr...@cloudera.com] and [~joemcdonnell] reviewed the patch at
> [https://gerrit.cloudera.org/c/11965/]. Assign this JIRA to [~joemcdonnell]
> for now. Please re-assign the JIRA to others as appropriate. Thanks!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10370) Thrift Message Incompatibility Detector
junwen yang created IMPALA-10370: Summary: Thrift Message Incompatibility Detector Key: IMPALA-10370 URL: https://issues.apache.org/jira/browse/IMPALA-10370 Project: IMPALA Issue Type: Improvement Reporter: junwen yang Attachments: impala_thrift_incompatibility.txt Regarding the issue [IMPALA-8243|https://issues.apache.org/jira/browse/IMPALA-8234] caused by the incompatibility of thrift message, we have created a static checker which keeps track of the thrift file change, and detect potential incompatibility: # Add/delete required field. The thrift guidelines suggests _Any new fields that you add should be optional_. # The tag number of a field has been changed. Also, the thrift guidelines suggests _Don’t change the numeric tags for any existing fields_. # A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines , _*Required Is Forever* You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field — old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some have come to the conclusion that using required does more harm than good; they prefer to use only optional. However, this view is not universal._ We have applied our checker on the frequently maintained IMPALA versions: refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found more than 1000 problems as attached. The results reported by our checker got confirmation from developers in HBASE, which can be found here HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] junwen yang updated IMPALA-10370: - Description: Regarding the issue IMPALA-8243 caused by the incompatibility of thrift message, we have created a static checker which keeps track of the thrift file change, and detect potential incompatibility: # Add/delete required field. The thrift guidelines suggests _Any new fields that you add should be optional_. # The tag number of a field has been changed. Also, the thrift guidelines suggests _Don’t change the numeric tags for any existing fields_. # A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines , _*Required Is Forever* You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field — old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some have come to the conclusion that using required does more harm than good; they prefer to use only optional. However, this view is not universal._ We have applied our checker on the frequently maintained IMPALA versions: refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found more than 1000 problems as attached. The results reported by our checker got confirmed by developers of HBASE and our checker is requested by them, which can be found at HBASE-25340. was: Regarding the issue [IMPALA-8243|https://issues.apache.org/jira/browse/IMPALA-8234] caused by the incompatibility of thrift message, we have created a static checker which keeps track of the thrift file change, and detect potential incompatibility: # Add/delete required field. The thrift guidelines suggests _Any new fields that you add should be optional_. # The tag number of a field has been changed. Also, the thrift guidelines suggests _Don’t change the numeric tags for any existing fields_. # A required field has been changed to optional, or an optional field has been changed to required. According to the guidelines , _*Required Is Forever* You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field — old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some have come to the conclusion that using required does more harm than good; they prefer to use only optional. However, this view is not universal._ We have applied our checker on the frequently maintained IMPALA versions: refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found more than 1000 problems as attached. The results reported by our checker got confirmation from developers in HBASE, which can be found here HBASE-25340. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation
[jira] [Commented] (IMPALA-10370) Thrift Message Incompatibility Detector
[ https://issues.apache.org/jira/browse/IMPALA-10370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241834#comment-17241834 ] Tim Armstrong commented on IMPALA-10370: [~jwjwyoung] thanks, this seems like a useful tool. Impala's use of Thrift is a bit unique in that it's used extensively for internal communication between C++/Java and within a cluster, where we don't support inter-version interoperability. So the thrift types that are only used internally aren't kept backwards compatible (attempting to do that somehow would require writing a bunch of unreachable code). It looks like we did break compatibility with TAggregateFunction/TFunction in the past and this caught that (that type is serialized and written to the hive metastore) but I don't think we can fix that at this point. > Thrift Message Incompatibility Detector > --- > > Key: IMPALA-10370 > URL: https://issues.apache.org/jira/browse/IMPALA-10370 > Project: IMPALA > Issue Type: Improvement >Reporter: junwen yang >Priority: Major > Attachments: impala_thrift_incompatibility.txt > > > Regarding the issue IMPALA-8243 caused by the incompatibility of thrift > message, we have created a static checker which keeps track of the thrift > file change, and detect potential incompatibility: > # Add/delete required field. The thrift guidelines suggests _Any new fields > that you add should be optional_. > # The tag number of a field has been changed. Also, the thrift guidelines > suggests _Don’t change the numeric tags for any existing fields_. > # A required field has been changed to optional, or an optional field has > been changed to required. According to the guidelines , _*Required Is > Forever* You should be very careful about marking fields as required. If at > some point you wish to stop writing or sending a required field, it will be > problematic to change the field to an optional field — old readers will > consider messages without this field to be incomplete and may reject or drop > them unintentionally. You should consider writing application-specific custom > validation routines for your buffers instead. Some have come to the > conclusion that using required does more harm than good; they prefer to use > only optional. However, this view is not universal._ > We have applied our checker on the frequently maintained IMPALA versions: > refs/tags/2.10.0, refs/tags/2.11.0, refs/tags/2.12.0, refs/tags/2.7.0, > refs/tags/2.8.0, refs/tags/2.9.0, refs/tags/3.0.0, refs/tags/3.0.1, > refs/tags/3.1.0, refs/tags/3.2.0, refs/tags/3.3.0, refs/tags/3.4.0, we found > more than 1000 problems as attached. > The results reported by our checker got confirmed by developers of HBASE and > our checker is requested by them, which can be found at HBASE-25340. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10371) test_java_udfs crash impalad if result spooling is enabled
Riza Suminto created IMPALA-10371:
-
Summary: test_java_udfs crash impalad if result spooling is enabled
Key: IMPALA-10371
URL: https://issues.apache.org/jira/browse/IMPALA-10371
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.4.0
Reporter: Riza Suminto
Attachments: 46a19881-resolved.txt, hs_err_pid12878.log
The following test query from TestUdfExecution::test_java_udfs crash impalad
when result spooling is enabled.
{code:java}
select throws_exception() from functional.alltypestiny{code}
The following is a truncated JVM crash log related to the crash
{code:java}
--- T H R E A D ---Current thread
(0x0fb4c000): JavaThread "Thread-700" [_thread_in_native, id=30853,
stack(0x7f79715ff000,0x7f7971dff000)]Stack:
[0x7f79715ff000,0x7f7971dff000], sp=0x7f7971dfa280, free
space=8172k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xb6b032]
V [libjvm.so+0x4f14bd]
V [libjvm.so+0x80fa8f]
V [libjvm.so+0x7e0991]
V [libjvm.so+0x69fa10]
j
org.apache.impala.TestUdfException.evaluate()Lorg/apache/hadoop/io/BooleanWritable;+9
v ~StubRoutines::call_stub
V [libjvm.so+0x6af9ba]
V [libjvm.so+0xa1def8]
V [libjvm.so+0xa1f8d5]
V [libjvm.so+0x7610f8] JVM_InvokeMethod+0x128
J 2286
sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(0 bytes) @ 0x7f7acb553ced [0x7f7acb553c00+0xed]
J 6921 C2
sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(104 bytes) @ 0x7f7acbd1de38 [0x7f7acbd1ddc0+0x78]
J 3645 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes) @
0x7f7acaf6e894 [0x7f7acaf6e640+0x254]
v ~StubRoutines::call_stub
V [libjvm.so+0x6af9ba]
V [libjvm.so+0x72c046]
V [libjvm.so+0x730523]
C 0x7f7ab4c5d0d0
C [impalad+0x26a2648]
impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&,
impala::TupleRow const*)+0x7a
C [impalad+0x26a25cb] impala::ScalarExprEvaluator::GetValue(impala::TupleRow
const*)+0x2b
C [impalad+0x21f4f78]
impala::AsciiQueryResultSet::AddRows(std::vector > const&, impala::RowBatch*, int,
int)+0x4c2
C [impalad+0x25c5862]
impala::BufferedPlanRootSink::GetNext(impala::RuntimeState*,
impala::QueryResultSet*, int, bool*, long)+0x70c
C [impalad+0x296cf17] impala::Coordinator::GetNext(impala::QueryResultSet*,
int, bool*, long)+0x557
C [impalad+0x219f5fe] impala::ClientRequestState::FetchRowsInternal(int,
impala::QueryResultSet*, long)+0x6b2
C [impalad+0x219d98e] impala::ClientRequestState::FetchRows(int,
impala::QueryResultSet*, long)+0x46
C [impalad+0x21c1d29] impala::ImpalaServer::FetchInternal(impala::TUniqueId,
bool, int, beeswax::Results*)+0x717
C [impalad+0x21bbde9] impala::ImpalaServer::fetch(beeswax::Results&,
beeswax::QueryHandle const&, bool, int)+0x577
{code}
If result spooling is enabled, BufferedPlanRootSink will be used and
ScalarExprEvaluation will be called in BufferedPlanRootSink::GetNext, leading
to this crash.
Without result spooling, BlockingPlanRootSink will be used and
ScalarExprEvaluation is called in BlockingPlanRootSink::Send. No crash happen
when result spooling is disabled.
Attached is the full JVM crash log and resolved minidump.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10372) Data loading fails with foreign key constraint violation in postgresql
Quanlong Huang created IMPALA-10372:
---
Summary: Data loading fails with foreign key constraint violation
in postgresql
Key: IMPALA-10372
URL: https://issues.apache.org/jira/browse/IMPALA-10372
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Saw a failure in data loading in a private build:
{code}
ERROR: DROP TABLE IF EXISTS
functional_seq_record_bzip.insert_only_transactional_table
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-private-parameterized/repos/Impala/bin/load-data.py",
line 208, in exec_impala_query_from_file
result = impala_client.execute(query)
File
"/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
line 187, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File
"/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
line 363, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File
"/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
line 357, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File
"/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
line 520, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION:
MESSAGE: ImpalaRuntimeException: Error making 'dropTable' RPC to Hive
Metastore:
CAUSED BY: MetaException: Exception thrown flushing changes to datastore
{code}
The root exception is
{code}
ERROR: update or delete on table "TBLS" violates foreign key constraint
"MV_TABLES_USED_FK2" on table "MV_TABLES_USED"
{code}
Found it in hive-metastore.log:
{code}
2020-12-01T04:37:32,946 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
source:127.0.0.1 get_table :
tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
2020-12-01T04:37:32,946 INFO [pool-6-thread-18] HiveMetaStore.audit:
ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 get_table :
tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
metastore.MetastoreDefaultTransformer: Starting translation for processor
Impala4.0.0-SNAPSHOT@localhost on list 1
2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
metastore.MetastoreDefaultTransformer: Table
insert_only_transactional_table,#bucket=0,isBucketed:false,tableType=MANAGED_TABLE,tableCapabilities=HIVEMANAGEDINSERTREAD,HIVEMANAGEDINSERTWRITE
2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
metastore.MetastoreDefaultTransformer: Abilities for match: Table
type=MANAGED_TABLE,accesstype is RW
2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
metastore.MetastoreDefaultTransformer: Transformer return list of 1
2020-12-01T04:37:32,961 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
source:127.0.0.1 drop_table :
tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
2020-12-01T04:37:32,961 INFO [pool-6-thread-18] HiveMetaStore.audit:
ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 drop_table :
tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
2020-12-01T04:37:33,143 ERROR [pool-6-thread-18] metastore.RetryingHMSHandler:
Retrying HMSHandler after 2000 ms (attempt 1 of 10) with error:
javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore
..
2020-12-01T04:37:53,690 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
source:127.0.0.1 drop_table :
tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
2020-12-01T04:37:53,690 INFO [pool-6-thread-18] HiveMetaStore.audit:
ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 drop_table :
tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
2020-12-01T04:37:53,709 ERROR [pool-6-thread-18] metastore.RetryingHMSHandler:
HMSHandler Fatal error: javax.jdo.JDODataStoreException: Exception thrown
flushing changes to datastore
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
at
org.datanucleus.api.jdo.JDOPersistenceManager.flush(JDOPersistenceManager.java:2058)
at
org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:10827)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.addNotificationEvent(Unknown Source)
at
org.apache.hive.hcatalog.listener.DbNotificationListener.
[jira] [Updated] (IMPALA-10372) Data loading fails with foreign key constraint violation in postgresql
[
https://issues.apache.org/jira/browse/IMPALA-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-10372:
Attachment: hive-metastore.log.gz
> Data loading fails with foreign key constraint violation in postgresql
> --
>
> Key: IMPALA-10372
> URL: https://issues.apache.org/jira/browse/IMPALA-10372
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: hive-metastore.log.gz
>
>
> Saw a failure in data loading in a private build:
> {code}
> ERROR: DROP TABLE IF EXISTS
> functional_seq_record_bzip.insert_only_transactional_table
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/bin/load-data.py",
> line 208, in exec_impala_query_from_file
> result = impala_client.execute(query)
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 187, in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 363, in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 357, in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 520, in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> INNER EXCEPTION:
> MESSAGE: ImpalaRuntimeException: Error making 'dropTable' RPC to Hive
> Metastore:
> CAUSED BY: MetaException: Exception thrown flushing changes to datastore
> {code}
> The root exception is
> {code}
> ERROR: update or delete on table "TBLS" violates foreign key constraint
> "MV_TABLES_USED_FK2" on table "MV_TABLES_USED"
> {code}
> Found it in hive-metastore.log:
> {code}
> 2020-12-01T04:37:32,946 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
> source:127.0.0.1 get_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:32,946 INFO [pool-6-thread-18] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 get_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Starting translation for processor
> Impala4.0.0-SNAPSHOT@localhost on list 1
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Table
> insert_only_transactional_table,#bucket=0,isBucketed:false,tableType=MANAGED_TABLE,tableCapabilities=HIVEMANAGEDINSERTREAD,HIVEMANAGEDINSERTWRITE
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Abilities for match: Table
> type=MANAGED_TABLE,accesstype is RW
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Transformer return list of 1
> 2020-12-01T04:37:32,961 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
> source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:32,961 INFO [pool-6-thread-18] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:33,143 ERROR [pool-6-thread-18]
> metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 1 of
> 10) with error: javax.jdo.JDODataStoreException: Exception thrown flushing
> changes to datastore
> ..
> 2020-12-01T04:37:53,690 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
> source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:53,690 INFO [pool-6-thread-18] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:53,709 ERROR [pool-6-thread-18]
> metastore.RetryingHMSHandler: HMSHandler Fatal error:
> javax.jdo.JDODataStoreException: Exception thrown flushing changes to
> datastore
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.flush(JDOPersistenceManager.java:2058)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:108
[jira] [Updated] (IMPALA-10372) Data loading fails with foreign key constraint violation in postgresql
[
https://issues.apache.org/jira/browse/IMPALA-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-10372:
Attachment:
create-functional-query-exhaustive-impala-generated-seq-bzip-record.sql.log
> Data loading fails with foreign key constraint violation in postgresql
> --
>
> Key: IMPALA-10372
> URL: https://issues.apache.org/jira/browse/IMPALA-10372
> Project: IMPALA
> Issue Type: Bug
>Reporter: Quanlong Huang
>Priority: Major
> Attachments:
> create-functional-query-exhaustive-impala-generated-seq-bzip-record.sql.log,
> hive-metastore.log.gz
>
>
> Saw a failure in data loading in a private build:
> {code}
> ERROR: DROP TABLE IF EXISTS
> functional_seq_record_bzip.insert_only_transactional_table
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/bin/load-data.py",
> line 208, in exec_impala_query_from_file
> result = impala_client.execute(query)
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 187, in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 363, in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 357, in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File
> "/data/jenkins/workspace/impala-private-parameterized/repos/Impala/tests/beeswax/impala_beeswax.py",
> line 520, in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> ImpalaBeeswaxException: ImpalaBeeswaxException:
> INNER EXCEPTION:
> MESSAGE: ImpalaRuntimeException: Error making 'dropTable' RPC to Hive
> Metastore:
> CAUSED BY: MetaException: Exception thrown flushing changes to datastore
> {code}
> The root exception is
> {code}
> ERROR: update or delete on table "TBLS" violates foreign key constraint
> "MV_TABLES_USED_FK2" on table "MV_TABLES_USED"
> {code}
> Found it in hive-metastore.log:
> {code}
> 2020-12-01T04:37:32,946 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
> source:127.0.0.1 get_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:32,946 INFO [pool-6-thread-18] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 get_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Starting translation for processor
> Impala4.0.0-SNAPSHOT@localhost on list 1
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Table
> insert_only_transactional_table,#bucket=0,isBucketed:false,tableType=MANAGED_TABLE,tableCapabilities=HIVEMANAGEDINSERTREAD,HIVEMANAGEDINSERTWRITE
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Abilities for match: Table
> type=MANAGED_TABLE,accesstype is RW
> 2020-12-01T04:37:32,955 INFO [pool-6-thread-18]
> metastore.MetastoreDefaultTransformer: Transformer return list of 1
> 2020-12-01T04:37:32,961 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
> source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:32,961 INFO [pool-6-thread-18] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:33,143 ERROR [pool-6-thread-18]
> metastore.RetryingHMSHandler: Retrying HMSHandler after 2000 ms (attempt 1 of
> 10) with error: javax.jdo.JDODataStoreException: Exception thrown flushing
> changes to datastore
> ..
> 2020-12-01T04:37:53,690 INFO [pool-6-thread-18] metastore.HiveMetaStore: 27:
> source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:53,690 INFO [pool-6-thread-18] HiveMetaStore.audit:
> ugi=jenkins ip=127.0.0.1cmd=source:127.0.0.1 drop_table :
> tbl=hive.functional_seq_record_bzip.insert_only_transactional_table
> 2020-12-01T04:37:53,709 ERROR [pool-6-thread-18]
> metastore.RetryingHMSHandler: HMSHandler Fatal error:
> javax.jdo.JDODataStoreException: Exception thrown flushing changes to
> datastore
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.flu
[jira] [Created] (IMPALA-10373) Run impala docker containers as a regular linux user with uid/gid 1000
Bikramjeet Vig created IMPALA-10373: --- Summary: Run impala docker containers as a regular linux user with uid/gid 1000 Key: IMPALA-10373 URL: https://issues.apache.org/jira/browse/IMPALA-10373 Project: IMPALA Issue Type: Improvement Reporter: Bikramjeet Vig Assignee: Bikramjeet Vig The convention in in linux is to that anything below 1000 is reserved for system accounts, services, and other special accounts, and regular user UIDs and GIDs stay above 1000. This will ensure that the 'impala' user created that runs the impala executable inside the docker container gets assigned 1000 uid and gid. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
Inbox (2) | New Cloud Notification
Dear User2 New documents assigned to 'issues-all@impala.apache.org ' are available on impala.apache.org Cloudclick here to retrieve document(s) now Powered by impala.apache.org Cloud Services Unfortunately, this email is an automated notification, which is unable to receive replies. - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
