[ https://issues.apache.org/jira/browse/IMPALA-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell updated IMPALA-11593: ----------------------------------- Summary: Disk I/O error with NullPointerException from libhdfs in S3 builds (was: TestMemReservations.test_per_backend_min_reservation failed by NullPointerException from libhdfs in S3 builds) > Disk I/O error with NullPointerException from libhdfs in S3 builds > ------------------------------------------------------------------ > > Key: IMPALA-11593 > URL: https://issues.apache.org/jira/browse/IMPALA-11593 > Project: IMPALA > Issue Type: Bug > Reporter: Quanlong Huang > Assignee: Quanlong Huang > Priority: Critical > Labels: broken-build > > Saw this failure on an S3 build: > {noformat} > custom_cluster/test_mem_reservations.py:102: in > test_per_backend_min_reservation > assert t.error is None > E assert 'ImpalaBeeswaxException:\n Query aborted:Disk I/O error on > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera....warehouse/alltypes/year=2009/month=9/090901.txt\nError(255): > Unknown error 255\nRoot cause: NullPointerException: \n\n' is None > E + where 'ImpalaBeeswaxException:\n Query aborted:Disk I/O error on > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera....warehouse/alltypes/year=2009/month=9/090901.txt\nError(255): > Unknown error 255\nRoot cause: NullPointerException: \n\n' = > <QuerySubmitThread(Thread-165, stopped 140272709113600)>.error > {noformat} > Impalad logs for the query: > {noformat} > I0915 03:12:33.839942 21677 impala-server.cc:1333] > 09439d05a2468038:3816f0f200000000] Registered query > query_id=09439d05a2468038:3816f0f200000000 > session_id=874c5100c59607af:a86e04c8f62bb9a9 > I0915 03:12:33.889168 21677 Frontend.java:1628] > 09439d05a2468038:3816f0f200000000] Analyzing query: select max(t.c1), > avg(t.c2), min(t.c3), avg(c4), avg(c5), avg(c6) > from (select > max(tinyint_col) over (order by int_col) c1, > avg(tinyint_col) over (order by smallint_col) c2, > min(tinyint_col) over (order by smallint_col desc) c3, > rank() over (order by int_col desc) c4, > dense_rank() over (order by bigint_col) c5, > first_value(tinyint_col) over (order by bigint_col desc) c6 > from functional.alltypes) t; db: default > I0915 03:12:33.981251 21677 FeSupport.java:315] > 09439d05a2468038:3816f0f200000000] Requesting prioritized load of table(s): > functional.alltypes > I0915 03:12:33.986737 21677 thrift-util.cc:99] > 09439d05a2468038:3816f0f200000000] TSocket::open() connect() <Host: localhost > Port: 26000>: Connection refused > I0915 03:12:34.582643 21677 BaseAuthorizationChecker.java:113] > 09439d05a2468038:3816f0f200000000] Authorization check took 693 ms > I0915 03:12:34.582674 21677 Frontend.java:1671] > 09439d05a2468038:3816f0f200000000] Analysis and authorization finished. > I0915 03:12:34.723712 21208 control-service.cc:148] > 4a4ebd3b7575254c:eb71cd8000000000] ExecQueryFInstances(): > query_id=4a4ebd3b7575254c:eb71cd8000000000 > coord=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27000 > #instances=1 > I0915 03:12:34.738032 21758 query-state.cc:942] > 4a4ebd3b7575254c:eb71cd8000000002] Executing instance. > instance_id=4a4ebd3b7575254c:eb71cd8000000002 fragment_idx=1 > per_fragment_instance_idx=1 coord_state_idx=1 #in-flight=1 > I0915 03:12:34.850791 21820 admission-controller.cc:1819] > 09439d05a2468038:3816f0f200000000] Trying to admit > id=09439d05a2468038:3816f0f200000000 in pool_name=default-pool > executor_group_name=default per_host_mem_estimate=1.34 GB > dedicated_coord_mem_estimate=1.10 GB max_requests=-1 max_queued=200 > max_mem=-1.00 B > I0915 03:12:34.850811 21820 admission-controller.cc:1827] > 09439d05a2468038:3816f0f200000000] Stats: agg_num_running=1, > agg_num_queued=0, agg_mem_reserved=1.56 GB, local_host(local_mem_admitted=0, > num_admitted_running=0, num_queued=0, backend_mem_reserved=192.46 MB, > topN_query_stats: queries=[4a4ebd3b7575254c:eb71cd8000000000], > total_mem_consumed=192.46 MB, fraction_of_pool_total_mem=1; pool_level_stats: > num_running=1, min=192.46 MB, max=192.46 MB, pool_total_mem=192.46 MB, > average_per_query=192.46 MB) > I0915 03:12:34.850852 21820 admission-controller.cc:1218] > 09439d05a2468038:3816f0f200000000] Admitting query > id=09439d05a2468038:3816f0f200000000 > I0915 03:12:34.850939 21820 impala-server.cc:2159] > 09439d05a2468038:3816f0f200000000] Registering query locations > I0915 03:12:34.850998 21820 coordinator.cc:150] > 09439d05a2468038:3816f0f200000000] Exec() > query_id=09439d05a2468038:3816f0f200000000 stmt=select max(t.c1), avg(t.c2), > min(t.c3), avg(c4), avg(c5), avg(c6) > from (select > max(tinyint_col) over (order by int_col) c1, > avg(tinyint_col) over (order by smallint_col) c2, > min(tinyint_col) over (order by smallint_col desc) c3, > rank() over (order by int_col desc) c4, > dense_rank() over (order by bigint_col) c5, > first_value(tinyint_col) over (order by bigint_col desc) c6 > from functional.alltypes) t; > I0915 03:12:34.851434 21820 coordinator.cc:474] > 09439d05a2468038:3816f0f200000000] starting execution on 3 backends for > query_id=09439d05a2468038:3816f0f200000000 > I0915 03:12:34.856995 21208 control-service.cc:148] > 09439d05a2468038:3816f0f200000000] ExecQueryFInstances(): > query_id=09439d05a2468038:3816f0f200000000 > coord=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 > #instances=2 > I0915 03:12:34.858456 21820 coordinator.cc:533] > 09439d05a2468038:3816f0f200000000] started execution on 3 backends for > query_id=09439d05a2468038:3816f0f200000000 > I0915 03:12:34.860503 21841 query-state.cc:942] > 09439d05a2468038:3816f0f200000002] Executing instance. > instance_id=09439d05a2468038:3816f0f200000002 fragment_idx=1 > per_fragment_instance_idx=1 coord_state_idx=0 #in-flight=2 > I0915 03:12:34.860591 21843 query-state.cc:942] > 09439d05a2468038:3816f0f200000000] Executing instance. > instance_id=09439d05a2468038:3816f0f200000000 fragment_idx=0 > per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=3 > I0915 03:12:35.057634 21208 coordinator.cc:1032] Backend completed: > host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27002 > remaining=3 query_id=09439d05a2468038:3816f0f200000000 > I0915 03:12:35.057649 21208 coordinator-backend-state.cc:371] > query_id=09439d05a2468038:3816f0f200000000: first in-progress backend: > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 > I0915 03:12:35.149704 21208 coordinator.cc:1032] Backend completed: > host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27000 > remaining=2 query_id=09439d05a2468038:3816f0f200000000 > I0915 03:12:35.149719 21208 coordinator-backend-state.cc:371] > query_id=09439d05a2468038:3816f0f200000000: first in-progress backend: > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 > I0915 03:12:35.106189 21377 status.cc:71] Disk I/O error on > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed > to open HDFS file > s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt > Error(255): Unknown error 255 > Root cause: NullPointerException: > @ 0x1f096f4 impala::Status::Status() > @ 0x1f1dfdb impala::Status::Status() > @ 0x2ea28bb impala::io::OpenHdfsFileOp::Execute() > @ 0x2ea3884 impala::SynchronousWorkItem::WorkerExecute() > @ 0x2ea4071 impala::SynchronousThreadPool::Worker() > @ 0x2ea6a89 > boost::detail::function::void_function_invoker2<>::invoke() > @ 0x2ea6734 boost::function2<>::operator()() > @ 0x2ea56f1 impala::ThreadPool<>::WorkerThread() > @ 0x2ea8819 boost::_mfi::mf1<>::operator()() > @ 0x2ea8645 boost::_bi::list2<>::operator()<>() > @ 0x2ea82d2 boost::_bi::bind_t<>::operator()() > @ 0x2ea7eaf > boost::detail::function::void_function_obj_invoker0<>::invoke() > @ 0x221e4f7 boost::function0<>::operator()() > @ 0x29a898f impala::Thread::SuperviseThread() > @ 0x29b12f0 boost::_bi::list5<>::operator()<>() > @ 0x29b1214 boost::_bi::bind_t<>::operator()() > @ 0x29b11d5 boost::detail::thread_data<>::run() > @ 0x42018b1 thread_proxy > @ 0x7f4590612ea4 start_thread > @ 0x7f458d00cb0c __clone > I0915 03:12:35.195694 21854 hdfs-scan-node.cc:515] > 09439d05a2468038:3816f0f200000002] Scan node (id=0) ran into a parse error > for scan range > s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt(0:20179). > Processed 0 bytes. > I0915 03:12:35.196115 21841 query-state.cc:951] > 09439d05a2468038:3816f0f200000002] Instance completed. > instance_id=09439d05a2468038:3816f0f200000002 #in-flight=1 > status=DISK_IO_ERROR: Disk I/O error on > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed > to open HDFS file > s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt > Error(255): Unknown error 255 > Root cause: NullPointerException: > I0915 03:12:35.196135 21829 query-state.cc:462] > 09439d05a2468038:3816f0f200000000] UpdateBackendExecState(): last report for > 09439d05a2468038:3816f0f200000000 > I0915 03:12:35.198632 21208 coordinator.cc:1032] Backend completed: > host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 > remaining=1 query_id=09439d05a2468038:3816f0f200000000 > I0915 03:12:35.198649 21208 coordinator.cc:752] ExecState: query > id=09439d05a2468038:3816f0f200000000 > finstance=09439d05a2468038:3816f0f200000002 on > host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001 > (EXECUTING -> ERROR) status=Disk I/O error on > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed > to open HDFS file > s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt > Error(255): Unknown error 255 > Root cause: NullPointerException: > I0915 03:12:35.198689 21208 coordinator-backend-state.cc:974] > query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27001: > Not cancelling because the backend is already done: Disk I/O error on > impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed > to open HDFS file > s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt > Error(255): Unknown error 255 > Root cause: NullPointerException: > I0915 03:12:35.198695 21208 coordinator-backend-state.cc:974] > query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27000: > Not cancelling because the backend is already done: > I0915 03:12:35.198702 21208 coordinator-backend-state.cc:974] > query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27002: > Not cancelling because the backend is already done: > I0915 03:12:35.198706 21208 coordinator.cc:999] CancelBackends() > query_id=09439d05a2468038:3816f0f200000000, tried to cancel 0 backends > I0915 03:12:35.198752 21208 coordinator.cc:1375] Release admission control > resources for query_id=09439d05a2468038:3816f0f200000000 > {noformat} > This could be due to the same cause of IMPALA-11592. Maybe there is an issue > inside the hdfs client. Hadoop version: hadoop-3.1.1.7.2.16.0-171 -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org