[jira] [Reopened] (IMPALA-10072) Data load failures in ubuntu-16.04-from-scratch
[
https://issues.apache.org/jira/browse/IMPALA-10072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer reopened IMPALA-10072:
--
I have seen the same callstack recently in several builds:
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12265/console
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12267/console
> Data load failures in ubuntu-16.04-from-scratch
> ---
>
> Key: IMPALA-10072
> URL: https://issues.apache.org/jira/browse/IMPALA-10072
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Sahil Takiar
>Assignee: Laszlo Gaal
>Priority: Major
> Fix For: Impala 4.0
>
>
> Seems like there are consistent data load failures on several unrelated
> patches:
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11627/]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11629/|https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11629/#showFailuresLink]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11631/|https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11631/#showFailuresLink]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11633/|https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11633/#showFailuresLink]
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/11635/]
> Almost all seem to be failing with an error like this:
> {code:java}
> 02:06:32 Loading nested parquet data (logging to
> /home/ubuntu/Impala/logs/data_loading/load-nested.log)...
> 02:08:06 FAILED (Took: 1 min 34 sec)
> 02:08:06 '/home/ubuntu/Impala/testdata/bin/load_nested.py -t
> tpch_nested_parquet -f parquet/none' failed. Tail of log:
> 02:08:06 at javax.security.auth.Subject.doAs(Subject.java:422)
> 02:08:06 at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> 02:08:06 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> 02:08:06
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:220)
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1361)
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> 02:08:06 at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:471)
> 02:08:06 ... 17 more
> 02:08:06 Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test-warehouse/tpch_nested_parquet.db/.hive-staging_hive_2020-08-11_02-07-45_902_3668710725192096563-193/_task_tmp.-ext-10004/_tmp.00_3
> could only be written to 0 of the 1 minReplication nodes. There are 3
> datanode(s) running and 3 node(s) are excluded in this operation.
> 02:08:06 at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2259)
> 02:08:06 at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
> 02:08:06 at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2773)
> 02:08:06 at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:879)
> 02:08:06 at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:583)
> 02:08:06 at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 02:08:06 at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> 02:08:06 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> 02:08:06 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> 02:08:06 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> 02:08:06 at java.security.AccessController.doPrivileged(Native Method)
> 02:08:06 at javax.security.auth.Subject.doAs(Subject.java:422)
> 02:08:06 at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> 02:08:06 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> 02:08:06
> 02:08:06 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1553)
> 02:08:06 at org.apache.hadoop.ipc.Client.call(Client.java:1499)
> 02:08:06 at org.apache.hadoop.ipc.Client.call(Client.java:1396)
> 02:08:06 at
> org.apache.hado
[jira] [Created] (IMPALA-10222) Implement CREATE TABLE AS SELECT for Iceberg tables
Zoltán Borók-Nagy created IMPALA-10222: -- Summary: Implement CREATE TABLE AS SELECT for Iceberg tables Key: IMPALA-10222 URL: https://issues.apache.org/jira/browse/IMPALA-10222 Project: IMPALA Issue Type: Sub-task Reporter: Zoltán Borók-Nagy Add support for CTAS statements for Iceberg tables. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10223) Implement INSERT OVERWRITE for non-partitioned Iceberg tables
Zoltán Borók-Nagy created IMPALA-10223: -- Summary: Implement INSERT OVERWRITE for non-partitioned Iceberg tables Key: IMPALA-10223 URL: https://issues.apache.org/jira/browse/IMPALA-10223 Project: IMPALA Issue Type: Sub-task Reporter: Zoltán Borók-Nagy Add support for INSERT OVERWRITE statements for Iceberg tables. Use Iceberg's OverwriteFiles API for this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10222) Implement CREATE TABLE AS SELECT for Iceberg tables
[ https://issues.apache.org/jira/browse/IMPALA-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy updated IMPALA-10222: --- Labels: impala-iceberg (was: ) > Implement CREATE TABLE AS SELECT for Iceberg tables > --- > > Key: IMPALA-10222 > URL: https://issues.apache.org/jira/browse/IMPALA-10222 > Project: IMPALA > Issue Type: Sub-task >Reporter: Zoltán Borók-Nagy >Priority: Major > Labels: impala-iceberg > > Add support for CTAS statements for Iceberg tables. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10213) Handle block location for Ozone
[ https://issues.apache.org/jira/browse/IMPALA-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208692#comment-17208692 ] Attila Doroszlai commented on IMPALA-10213: --- bq. Is it similar to classic HDFS with 3-way replication, or more like HDFS with erasure coding? Currently only 3-way replication is available. EC and other configurable storage classes are in the design phase. > Handle block location for Ozone > --- > > Key: IMPALA-10213 > URL: https://issues.apache.org/jira/browse/IMPALA-10213 > Project: IMPALA > Issue Type: Sub-task >Reporter: Attila Doroszlai >Priority: Major > > Currently Impala treats Ozone as a remote filesystem, similar to S3A, ADLS > etc. Ozone provides block location info in its Hadoop-compatible FS > implementations. Also, Ozone can be colocated with Impala daemons. It would > be nice if Impala could be improved to use Ozone's location info to support > locality of execution. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-532) Impala should tolerate bad locale settings.
[
https://issues.apache.org/jira/browse/IMPALA-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208847#comment-17208847
]
Vincent Tran commented on IMPALA-532:
-
This still occurs on Impala 3.1.0 by the way. Has boost not been upgraded since?
{noformat}
0 libc-2.22.so + 0x34fd7
1 libc-2.22.so + 0x363aa
2 ld-2.22.so + 0x14f60
3 libstdc++.so.6 + 0x78110
4 libstdc++.so.6!_fini + 0x2510
5 libstdc++.so.6!_fini + 0x2d5b
6 libc-2.22.so + 0x75cc7
7 libc-2.22.so + 0x75cc7
8 libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler() [vterminate.cc : 95
+ 0x5]
9 libstdc++.so.6!__cxxabiv1::__terminate(void (*)()) [eh_terminate.cc : 47 +
0x2]
10 libstdc++.so.6!std::terminate() [eh_terminate.cc : 57 + 0xd]
11 libstdc++.so.6!__cxxabiv1::__cxa_throw [eh_throw.cc : 87 + 0x5]
12 libstdc++.so.6!std::__throw_runtime_error(char const*) [functexcept.cc : 107
+ 0x16]
13 libstdc++.so.6!std::locale::facet::_S_create_c_locale(__locale_struct*&,
char const*, __locale_struct*) [c++locale.cc : 137 + 0xc]
14 libstdc++.so.6!std::locale::_Impl::_Impl(char const*, unsigned long)
[localename.cc : 182 + 0xa]
15 libstdc++.so.6!std::locale::locale(char const*) [localename.cc : 128 + 0x18]
16 impalad!boost::filesystem::path::codecvt() + 0x53
17 impalad!impala::HdfsScanNodeBase::Prepare(impala::RuntimeState*)
[hdfs-scan-node-base.cc : 209 + 0x5]
18 impalad!impala::HdfsScanNode::Prepare(impala::RuntimeState*)
[hdfs-scan-node.cc : 159 + 0x15]
19 impalad!impala::FragmentInstanceState::Prepare() [fragment-instance-state.cc
: 207 + 0x18]
20 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc :
75 + 0x5]
21 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
[query-state.cc : 478 + 0xf]
22 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
const&, boost::function, impala::ThreadDebugInfo const*,
impala::Promise*) [function_template.hpp : 767 +
0x7]
23 impalad!boost::detail::thread_data,
impala::ThreadDebugInfo const*, impala::Promise*), boost::_bi::list5,
boost::_bi::value, boost::_bi::value >,
boost::_bi::value,
boost::_bi::value*> > > >::run()
[bind.hpp : 525 + 0x6]
24 impalad!thread_proxy + 0xda
25 libpthread-2.22.so + 0x874a
26 libc-2.22.so + 0xecf6d{noformat}
> Impala should tolerate bad locale settings.
> ---
>
> Key: IMPALA-532
> URL: https://issues.apache.org/jira/browse/IMPALA-532
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 1.1
>Reporter: Ishaan Joshi
>Priority: Minor
>
> Currently, the Statestore does not tolerate a bad locale setting and crashes
> while starting up.
> {code}
> USE_DEBUG_BUILD=false
> + perl -pi -e
> 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE#g'
>
> /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags
> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
> LANGUAGE = (unset),
> LC_ALL = (unset),
> LANG = "fr_FR.UTF-8"
> are supported and installed on your system.
> perl: warning: Falling back to the standard locale ("C").
> + '[' -f
> /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd
> ']'
> + chmod 600
> /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd
> + false
> + export
> IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail
> + IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail
> + '[' impalad = statestore ']'
> + '[' statestore = statestore ']'
> + exec
> /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored
> --flagfile=/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags
> terminate called after throwing an instance of 'std::runtime_error'
> what(): locale::facet::_S_create_c_locale name not valid
> {code}
> It should fall back to the standard locale ("C"), if the user's locale is
> messed up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-532) Impala should tolerate bad locale settings.
[
https://issues.apache.org/jira/browse/IMPALA-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17208849#comment-17208849
]
Vincent Tran commented on IMPALA-532:
-
It does take a minidump to lead me to this Jira. It'd be useful to print a
fatal-level message into the log if Impala can detect invalid locale.
> Impala should tolerate bad locale settings.
> ---
>
> Key: IMPALA-532
> URL: https://issues.apache.org/jira/browse/IMPALA-532
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 1.1
>Reporter: Ishaan Joshi
>Priority: Minor
>
> Currently, the Statestore does not tolerate a bad locale setting and crashes
> while starting up.
> {code}
> USE_DEBUG_BUILD=false
> + perl -pi -e
> 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE#g'
>
> /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags
> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
> LANGUAGE = (unset),
> LC_ALL = (unset),
> LANG = "fr_FR.UTF-8"
> are supported and installed on your system.
> perl: warning: Falling back to the standard locale ("C").
> + '[' -f
> /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd
> ']'
> + chmod 600
> /var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/.htpasswd
> + false
> + export
> IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail
> + IMPALA_BIN=/opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/sbin-retail
> + '[' impalad = statestore ']'
> + '[' statestore = statestore ']'
> + exec
> /opt/cloudera/parcels/IMPALA-1.1-1.p0.8/lib/impala/../../bin/statestored
> --flagfile=/var/run/cloudera-scm-agent/process/2469-impala-STATESTORE/impala-conf/state_store_flags
> terminate called after throwing an instance of 'std::runtime_error'
> what(): locale::facet::_S_create_c_locale name not valid
> {code}
> It should fall back to the standard locale ("C"), if the user's locale is
> messed up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10224) Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC
Attila Jeges created IMPALA-10224: - Summary: Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC Key: IMPALA-10224 URL: https://issues.apache.org/jira/browse/IMPALA-10224 Project: IMPALA Issue Type: New Feature Components: Backend Reporter: Attila Jeges Assignee: Attila Jeges PingImpalaService/PingImpalaHS2Service RPC calls expose the coordinator's debug web url to clients like impala shell. Since the debug web UI is not something that end-users will necessarily have access to, we should have a server option to send an empty string instead of the real url to the impala client signalling that the debug web ui is not available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10224) Add startup flag not to expose debug web url via PingImpalaService/PingImpalaHS2Service RPC
[ https://issues.apache.org/jira/browse/IMPALA-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Jeges updated IMPALA-10224: -- Affects Version/s: Impala 3.4.0 > Add startup flag not to expose debug web url via > PingImpalaService/PingImpalaHS2Service RPC > --- > > Key: IMPALA-10224 > URL: https://issues.apache.org/jira/browse/IMPALA-10224 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 3.4.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > > PingImpalaService/PingImpalaHS2Service RPC calls expose the coordinator's > debug web url to clients like impala shell. Since the debug web UI is not > something that end-users will necessarily have access to, we should have a > server option to send an empty string instead of the real url to the impala > client signalling that the debug web ui is not available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9954) RpcRecvrTime can be negative
[
https://issues.apache.org/jira/browse/IMPALA-9954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209020#comment-17209020
]
Riza Suminto commented on IMPALA-9954:
--
Found the root cause problem for negative value in RpcNetworkTime counter.
Function EndDataStreamCompleteCb() is using resp_ instead of eos_resp_ to
compute total_time_ns.
The locking in TransmitDataCompleteCb() does not seem to be an issue, but I
still think that it should be acquired in the very beginning just like how it
goes in EndDataStreamCompleteCb().
I filed a separate JIRA for this issue in IMPALA-10220.
> RpcRecvrTime can be negative
>
>
> Key: IMPALA-9954
> URL: https://issues.apache.org/jira/browse/IMPALA-9954
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Sahil Takiar
>Priority: Major
> Attachments: profile_034e7209bd98c96c_9a448dfc.txt
>
>
> Saw this on a recent version of master. Attached the full runtime profile.
> {code:java}
> KrpcDataStreamSender (dst_id=2):(Total: 9.863ms, non-child: 3.185ms,
> % non-child: 32.30%)
> ExecOption: Unpartitioned Sender Codegen Disabled: not needed
>- BytesSent (500.000ms): 0, 0
>- NetworkThroughput: (Avg: 4.34 MB/sec ; Min: 4.34 MB/sec ; Max:
> 4.34 MB/sec ; Number of samples: 1)
>- RpcNetworkTime: (Avg: 3.562ms ; Min: 679.676us ; Max: 6.445ms ;
> Number of samples: 2)
>- RpcRecvrTime: (Avg: -151281.000ns ; Min: -231485.000ns ; Max:
> -71077.000ns ; Number of samples: 2)
>- EosSent: 1 (1)
>- PeakMemoryUsage: 416.00 B (416)
>- RowsSent: 100 (100)
>- RpcFailure: 0 (0)
>- RpcRetry: 0 (0)
>- SerializeBatchTime: 2.880ms
>- TotalBytesSent: 28.67 KB (29355)
>- UncompressedRowBatchSize: 69.29 KB (70950) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8291) 'DESCRIBE EXTENDED ..' does not display constraint information
[ https://issues.apache.org/jira/browse/IMPALA-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shant Hovsepian resolved IMPALA-8291. - Fix Version/s: Impala 4.0 Resolution: Fixed > 'DESCRIBE EXTENDED ..' does not display constraint information > -- > > Key: IMPALA-8291 > URL: https://issues.apache.org/jira/browse/IMPALA-8291 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Anurag Mantripragada >Assignee: Shant Hovsepian >Priority: Major > Fix For: Impala 4.0 > > > Currently, DESCRIBE EXTENDED table_name command does not display constraint > information like primary key / Foreign key information for tables created > through Hive. > This work must also be extended to tables created through Impala once we have > support for pk/fk in create table syntax. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
