[ https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886546#comment-16886546 ]
Michael Ho edited comment on IMPALA-7733 at 7/16/19 11:26 PM: -------------------------------------------------------------- A recent instance when running {{query_test/test_tpcds_queries.py::TestTpcdsInsert}}: {noformat} query_test/test_tpcds_queries.py:521: in test_tpcds_partitioned_insert self.run_test_case('partitioned-insert', vector) common/impala_test_suite.py:563: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:500: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:798: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:184: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:364: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:385: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Error(s) moving partition files. First error (of 1) was: Hdfs op (RENAME s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq TO s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq) failed, error was: s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq E Error(5): Input/output error {noformat} was (Author: kwho): A recent instance when running {{query_test/test_tpcds_queries.py::TestTpcdsInsert::()::test_tpcds_partitioned_insert}}: {noformat} query_test/test_tpcds_queries.py:521: in test_tpcds_partitioned_insert self.run_test_case('partitioned-insert', vector) common/impala_test_suite.py:563: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:500: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:798: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:184: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:364: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:385: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Error(s) moving partition files. First error (of 1) was: Hdfs op (RENAME s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq TO s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq) failed, error was: s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq E Error(5): Input/output error {noformat} > TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename > ------------------------------------------------------------------------- > > Key: IMPALA-7733 > URL: https://issues.apache.org/jira/browse/IMPALA-7733 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure > Affects Versions: Impala 3.1.0 > Reporter: Vuk Ercegovac > Assignee: Tianyi Wang > Priority: Blocker > Labels: broken-build, flaky > > I see two examples in the past two months or so where this test fails due to > a rename error on S3. The test's stacktrace looks like this: > {noformat} > query_test/test_insert_parquet.py:112: in test_insert_parquet > self.run_test_case('insert_parquet', vector, unique_database, > multiple_impalad=True) > common/impala_test_suite.py:408: in run_test_case > result = self.__execute_query(target_impalad_client, query, user=user) > common/impala_test_suite.py:625: in __execute_query > return impalad_client.execute(query, user=user) > common/impala_connection.py:160: in execute > return self.__beeswax_client.execute(sql_stmt, user=user) > beeswax/impala_beeswax.py:176: in execute > handle = self.__execute_query(query_string.strip(), user=user) > beeswax/impala_beeswax.py:350: in __execute_query > self.wait_for_finished(handle) > beeswax/impala_beeswax.py:371: in wait_for_finished > raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > E ImpalaBeeswaxException: ImpalaBeeswaxException: > E Query aborted:Error(s) moving partition files. First error (of 1) was: > Hdfs op (RENAME > s3a://<removed>/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq > TO > s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq) > failed, error was: > s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq > E Error(5): Input/output error{noformat} > Since we know this happens once in a while, some ideas to deflake it: > * retry > * check for this specific issue... if we think its platform flakiness, then > we should skip it. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org