[jira] [Reopened] (IMPALA-10072) Data load failures in ubuntu-16.04-from-scratch
[ https://issues.apache.org/jira/browse/IMPALA-10072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer reopened IMPALA-10072: -- I have seen the same callstack recently in several builds: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12265/console https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12267/console > Data load failures in ubuntu-16.04-from-scratch > --- > > Key: IMPALA-10072 > URL: https://issues.apache.org/jira/browse/IMPALA-10072 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sahil Takiar >Assignee: Laszlo Gaal >Priority: Major > Fix For: Impala 4.0 > > > Seems like there are consistent data load failures on several unrelated > patches: > [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11627/] > [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11629/|https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11629/#showFailuresLink] > [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11631/|https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11631/#showFailuresLink] > [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11633/|https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11633/#showFailuresLink] > [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11635/] > Almost all seem to be failing with an error like this: > {code:java} > 02:06:32 Loading nested parquet data (logging to > /home/ubuntu/Impala/logs/data_loading/load-nested.log)... > 02:08:06 FAILED (Took: 1 min 34 sec) > 02:08:06 '/home/ubuntu/Impala/testdata/bin/load_nested.py -t > tpch_nested_parquet -f parquet/none' failed. Tail of log: > 02:08:06 at javax.security.auth.Subject.doAs(Subject.java:422) > 02:08:06 at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) > 02:08:06 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882) > 02:08:06 > 02:08:06 at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:220) > 02:08:06 at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1361) > 02:08:06 at > org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732) > 02:08:06 at > org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756) > 02:08:06 at > org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756) > 02:08:06 at > org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756) > 02:08:06 at > org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:471) > 02:08:06 ... 17 more > 02:08:06 Caused by: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /test-warehouse/tpch_nested_parquet.db/.hive-staging_hive_2020-08-11_02-07-45_902_3668710725192096563-193/_task_tmp.-ext-10004/_tmp.00_3 > could only be written to 0 of the 1 minReplication nodes. There are 3 > datanode(s) running and 3 node(s) are excluded in this operation. > 02:08:06 at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2259) > 02:08:06 at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294) > 02:08:06 at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2773) > 02:08:06 at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:879) > 02:08:06 at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:583) > 02:08:06 at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > 02:08:06 at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528) > 02:08:06 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) > 02:08:06 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985) > 02:08:06 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913) > 02:08:06 at java.security.AccessController.doPrivileged(Native Method) > 02:08:06 at javax.security.auth.Subject.doAs(Subject.java:422) > 02:08:06 at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) > 02:08:06 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882) > 02:08:06 > 02:08:06 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1553) > 02:08:06 at org.apache.hadoop.ipc.Client.call(Client.java:1499) > 02:08:06 at org.apache.hadoop.ipc.Client.call(Client.java:1396) > 02:08:06 at > org.apache.hado
[jira] [Created] (IMPALA-10222) Implement CREATE TABLE AS SELECT for Iceberg tables
Zoltán Borók-Nagy created IMPALA-10222: -- Summary: Implement CREATE TABLE AS SELECT for Iceberg tables Key: IMPALA-10222 URL: https://issues.apache.org/jira/browse/IMPALA-10222 Project: IMPALA Issue Type: Sub-task Reporter: Zoltán Borók-Nagy Add support for CTAS statements for Iceberg tables. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10223) Implement INSERT OVERWRITE for non-partitioned Iceberg tables
Zoltán Borók-Nagy created IMPALA-10223: -- Summary: Implement INSERT OVERWRITE for non-partitioned Iceberg tables Key: IMPALA-10223 URL: https://issues.apache.org/jira/browse/IMPALA-10223 Project: IMPALA Issue Type: Sub-task Reporter: Zoltán Borók-Nagy Add support for INSERT OVERWRITE statements for Iceberg tables. Use Iceberg's OverwriteFiles API for this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10222) Implement CREATE TABLE AS SELECT for Iceberg tables
[ https://issues.apache.org/jira/browse/IMPALA-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy updated IMPALA-10222: --- Labels: impala-iceberg (was: ) > Implement CREATE TABLE AS SELECT for Iceberg tables > --- > > Key: IMPALA-10222 > URL: https://issues.apache.org/jira/browse/IMPALA-10222 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Priority: Major > Labels: impala-iceberg > > Add support for CTAS statements for Iceberg tables. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10213) Handle block location for Ozone
[ https://issues.apache.org/jira/browse/IMPALA-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208692#comment-17208692 ] Attila Doroszlai commented on IMPALA-10213: --- bq. Is it similar to classic HDFS with 3-way replication, or more like HDFS with erasure coding? Currently only 3-way replication is available. EC and other configurable storage classes are in the design phase. > Handle block location for Ozone > --- > > Key: IMPALA-10213 > URL: https://issues.apache.org/jira/browse/IMPALA-10213 > Project: IMPALA > Issue Type: Sub-task >Reporter: Attila Doroszlai >Priority: Major > > Currently Impala treats Ozone as a remote filesystem, similar to S3A, ADLS > etc. Ozone provides block location info in its Hadoop-compatible FS > implementations. Also, Ozone can be colocated with Impala daemons. It would > be nice if Impala could be improved to use Ozone's location info to support > locality of execution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-532) Impala should tolerate bad locale settings.
[ https://issues.apache.org/jira/browse/IMPALA-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208847#comment-17208847 ] Vincent Tran commented on IMPALA-532: - This still occurs on Impala 3.1.0 by the way. Has boost not been upgraded since? {noformat} 0 libc-2.22.so + 0x34fd7 1 libc-2.22.so + 0x363aa 2 ld-2.22.so + 0x14f60 3 libstdc++.so.6 + 0x78110 4 libstdc++.so.6!_fini + 0x2510 5 libstdc++.so.6!_fini + 0x2d5b 6 libc-2.22.so + 0x75cc7 7 libc-2.22.so + 0x75cc7 8 libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler() [vterminate.cc : 95 + 0x5] 9 libstdc++.so.6!__cxxabiv1::__terminate(void (*)()) [eh_terminate.cc : 47 + 0x2] 10 libstdc++.so.6!std::terminate() [eh_terminate.cc : 57 + 0xd] 11 libstdc++.so.6!__cxxabiv1::__cxa_throw [eh_throw.cc : 87 + 0x5] 12 libstdc++.so.6!std::__throw_runtime_error(char const*) [functexcept.cc : 107 + 0x16] 13 libstdc++.so.6!std::locale::facet::_S_create_c_locale(__locale_struct*&, char const*, __locale_struct*) [c++locale.cc : 137 + 0xc] 14 libstdc++.so.6!std::locale::_Impl::_Impl(char const*, unsigned long) [localename.cc : 182 + 0xa] 15 libstdc++.so.6!std::locale::locale(char const*) [localename.cc : 128 + 0x18] 16 impalad!boost::filesystem::path::codecvt() + 0x53 17 impalad!impala::HdfsScanNodeBase::Prepare(impala::RuntimeState*) [hdfs-scan-node-base.cc : 209 + 0x5] 18 impalad!impala::HdfsScanNode::Prepare(impala::RuntimeState*) [hdfs-scan-node.cc : 159 + 0x15] 19 impalad!impala::FragmentInstanceState::Prepare() [fragment-instance-state.cc : 207 + 0x18] 20 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc : 75 + 0x5] 21 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) [query-state.cc : 478 + 0xf] 22 impalad!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function, impala::ThreadDebugInfo const*, impala::Promise*) [function_template.hpp : 767 + 0x7] 23 impalad!boost::detail::thread_data, impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list5, boost::_bi::value, boost::_bi::value >, boost::_bi::value, boost::_bi::value*> > > >::run() [bind.hpp : 525 + 0x6] 24 impalad!thread_proxy + 0xda 25 libpthread-2.22.so + 0x874a 26 libc-2.22.so + 0xecf6d{noformat} > Impala should tolerate bad locale settings. > --- > > Key: IMPALA-532 > URL: https://issues.apache.org/jira/browse/IMPALA-532 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 1.1 >Reporter: Ishaan Joshi >Priority: Minor > > Currently, the Statestore does not tolerate a bad locale setting and crashes > while starting up. > {code} > USE_DEBUG_BUILD=false > + perl -pi -e > 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE#g' > > /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags > perl: warning: Setting locale failed. > perl: warning: Please check that your locale settings: > LANGUAGE = (unset), > LC_ALL = (unset), > LANG = "fr_FR.UTF-8" > are supported and installed on your system. > perl: warning: Falling back to the standard locale ("C"). > + '[' -f > /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd > ']' > + chmod 600 > /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd > + false > + export > IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail > + IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail > + '[' impalad = statestore ']' > + '[' statestore = statestore ']' > + exec > /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored > --flagfile=/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags > terminate called after throwing an instance of 'std::runtime_error' > what(): locale::facet::_S_create_c_locale name not valid > {code} > It should fall back to the standard locale ("C"), if the user's locale is > messed up. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-532) Impala should tolerate bad locale settings.
[ https://issues.apache.org/jira/browse/IMPALA-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208849#comment-17208849 ] Vincent Tran commented on IMPALA-532: - It does take a minidump to lead me to this Jira. It'd be useful to print a fatal-level message into the log if Impala can detect invalid locale. > Impala should tolerate bad locale settings. > --- > > Key: IMPALA-532 > URL: https://issues.apache.org/jira/browse/IMPALA-532 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 1.1 >Reporter: Ishaan Joshi >Priority: Minor > > Currently, the Statestore does not tolerate a bad locale setting and crashes > while starting up. > {code} > USE_DEBUG_BUILD=false > + perl -pi -e > 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE#g' > > /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags > perl: warning: Setting locale failed. > perl: warning: Please check that your locale settings: > LANGUAGE = (unset), > LC_ALL = (unset), > LANG = "fr_FR.UTF-8" > are supported and installed on your system. > perl: warning: Falling back to the standard locale ("C"). > + '[' -f > /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd > ']' > + chmod 600 > /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd > + false > + export > IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail > + IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail > + '[' impalad = statestore ']' > + '[' statestore = statestore ']' > + exec > /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored > --flagfile=/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags > terminate called after throwing an instance of 'std::runtime_error' > what(): locale::facet::_S_create_c_locale name not valid > {code} > It should fall back to the standard locale ("C"), if the user's locale is > messed up. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10224) Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC
Attila Jeges created IMPALA-10224: - Summary: Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC Key: IMPALA-10224 URL: https://issues.apache.org/jira/browse/IMPALA-10224 Project: IMPALA Issue Type: New Feature Components: Backend Reporter: Attila Jeges Assignee: Attila Jeges PingImpalaService/PingImpalaHS2Service RPC calls expose the coordinator's debug web url to clients like impala shell. Since the debug web UI is not something that end-users will necessarily have access to, we should have a server option to send an empty string instead of the real url to the impala client signalling that the debug web ui is not available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10224) Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC
[ https://issues.apache.org/jira/browse/IMPALA-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges updated IMPALA-10224: -- Affects Version/s: Impala 3.4.0 > Add startup flag not to expose debug web url via > PingImpalaService/PingImpalaHS2Service RPC > --- > > Key: IMPALA-10224 > URL: https://issues.apache.org/jira/browse/IMPALA-10224 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > > PingImpalaService/PingImpalaHS2Service RPC calls expose the coordinator's > debug web url to clients like impala shell. Since the debug web UI is not > something that end-users will necessarily have access to, we should have a > server option to send an empty string instead of the real url to the impala > client signalling that the debug web ui is not available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9954) RpcRecvrTime can be negative
[ https://issues.apache.org/jira/browse/IMPALA-9954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209020#comment-17209020 ] Riza Suminto commented on IMPALA-9954: -- Found the root cause problem for negative value in RpcNetworkTime counter. Function EndDataStreamCompleteCb() is using resp_ instead of eos_resp_ to compute total_time_ns. The locking in TransmitDataCompleteCb() does not seem to be an issue, but I still think that it should be acquired in the very beginning just like how it goes in EndDataStreamCompleteCb(). I filed a separate JIRA for this issue in IMPALA-10220. > RpcRecvrTime can be negative > > > Key: IMPALA-9954 > URL: https://issues.apache.org/jira/browse/IMPALA-9954 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Sahil Takiar >Priority: Major > Attachments: profile_034e7209bd98c96c_9a448dfc.txt > > > Saw this on a recent version of master. Attached the full runtime profile. > {code:java} > KrpcDataStreamSender (dst_id=2):(Total: 9.863ms, non-child: 3.185ms, > % non-child: 32.30%) > ExecOption: Unpartitioned Sender Codegen Disabled: not needed >- BytesSent (500.000ms): 0, 0 >- NetworkThroughput: (Avg: 4.34 MB/sec ; Min: 4.34 MB/sec ; Max: > 4.34 MB/sec ; Number of samples: 1) >- RpcNetworkTime: (Avg: 3.562ms ; Min: 679.676us ; Max: 6.445ms ; > Number of samples: 2) >- RpcRecvrTime: (Avg: -151281.000ns ; Min: -231485.000ns ; Max: > -71077.000ns ; Number of samples: 2) >- EosSent: 1 (1) >- PeakMemoryUsage: 416.00 B (416) >- RowsSent: 100 (100) >- RpcFailure: 0 (0) >- RpcRetry: 0 (0) >- SerializeBatchTime: 2.880ms >- TotalBytesSent: 28.67 KB (29355) >- UncompressedRowBatchSize: 69.29 KB (70950) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8291) 'DESCRIBE EXTENDED ..' does not display constraint information
[ https://issues.apache.org/jira/browse/IMPALA-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shant Hovsepian resolved IMPALA-8291. - Fix Version/s: Impala 4.0 Resolution: Fixed > 'DESCRIBE EXTENDED ..' does not display constraint information > -- > > Key: IMPALA-8291 > URL: https://issues.apache.org/jira/browse/IMPALA-8291 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Shant Hovsepian >Priority: Major > Fix For: Impala 4.0 > > > Currently, DESCRIBE EXTENDED table_name command does not display constraint > information like primary key / Foreign key information for tables created > through Hive. > This work must also be extended to tables created through Impala once we have > support for pk/fk in create table syntax. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org