[jira] [Created] (HADOOP-19174) Tez and hive jobs fail due to google's protobuf 2.5.0 in classpath
Bilwa S T created HADOOP-19174: -- Summary: Tez and hive jobs fail due to google's protobuf 2.5.0 in classpath Key: HADOOP-19174 URL: https://issues.apache.org/jira/browse/HADOOP-19174 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T There are two issues here: 1. We are running tez 0.10.3 which uses hadoop 3.3.6 version. Tez has protobuf version 3.21.1 Below is the exception we get. This is due to protobuf-2.5.0 in our hadoop classpath java.lang.IllegalAccessError: class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto tried to access private field com.google.protobuf.AbstractMessage.memoizedSize (org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto and com.google.protobuf.AbstractMessage are in unnamed module of loader 'app') at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.getSerializedSize(DAGProtos.java:21636) at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:75) at org.apache.tez.common.TezUtils.writeConfInPB(TezUtils.java:170) at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:83) at org.apache.tez.common.TezUtils.createUserPayloadFromConf(TezUtils.java:101) at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:436) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.Subject.doAs(Subject.java:439) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597) at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384) 2024-04-18 16:27:54,741 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked 2024-04-18 16:27:54,743 [INFO] [shutdown-hook-0] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STOPPED java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null at org.apache.tez.dag.app.DAGAppMaster.initiateStop(DAGAppMaster.java:2111) at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:2126) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterShutdownHook.run(DAGAppMaster.java:2432) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2024-04-18 16:27:54,744 [WARN] [Thread-2] |util.ShutdownHookManager|: ShutdownHook 'DAGAppMasterShutdownHook' failed, java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95) Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null at org.apache.tez.dag.app.DAGAppMaster.initiateStop(DAGAppMaster.java:2111) at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:2126) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterShutdownHook.run(DAGAppMaster.java:2432) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.ja
[jira] [Created] (HADOOP-19163) Upgrade protobuf version to 3.24.4
Bilwa S T created HADOOP-19163: -- Summary: Upgrade protobuf version to 3.24.4 Key: HADOOP-19163 URL: https://issues.apache.org/jira/browse/HADOOP-19163 Project: Hadoop Common Issue Type: Bug Components: hadoop-thirdparty Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19058) [JDK-17] TestCryptoOutputStreamClosing#testUnderlyingOutputStreamClosedWhenExceptionClosing fails
Bilwa S T created HADOOP-19058: -- Summary: [JDK-17] TestCryptoOutputStreamClosing#testUnderlyingOutputStreamClosedWhenExceptionClosing fails Key: HADOOP-19058 URL: https://issues.apache.org/jira/browse/HADOOP-19058 Project: Hadoop Common Issue Type: Sub-task Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18986) Upgrade Zookeeper to 3.8.2
Bilwa S T created HADOOP-18986: -- Summary: Upgrade Zookeeper to 3.8.2 Key: HADOOP-18986 URL: https://issues.apache.org/jira/browse/HADOOP-18986 Project: Hadoop Common Issue Type: Improvement Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18775) Add instrumentation access check for /logs servlet
Bilwa S T created HADOOP-18775: -- Summary: Add instrumentation access check for /logs servlet Key: HADOOP-18775 URL: https://issues.apache.org/jira/browse/HADOOP-18775 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T Currently AdminAuthorizedServlet is configured for /logs which would always check if user has admin access. Just like other servelets, log should also have instrumentation check before the admin access check. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18603) NPE in LdapAuthenticationHandler as disableHostNameVerification is never initialized
Bilwa S T created HADOOP-18603: -- Summary: NPE in LdapAuthenticationHandler as disableHostNameVerification is never initialized Key: HADOOP-18603 URL: https://issues.apache.org/jira/browse/HADOOP-18603 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T Steps to reproduce this issue: Enable ldap auth with tls by configuring these in core-site.xml 1. hadoop.http.authentication.multi-scheme-auth-handler.schemes = basic 2. hadoop.http.authentication.multi-scheme-auth-handler.schemes.basic.handler = ldap 3. hadoop.http.authentication.ldap.enablestarttls = true -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18566) Set Cache-Control no-store max-age as 0 header on all dynamic content
Bilwa S T created HADOOP-18566: -- Summary: Set Cache-Control no-store max-age as 0 header on all dynamic content Key: HADOOP-18566 URL: https://issues.apache.org/jira/browse/HADOOP-18566 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17763) DistCp job fails when AM is killed
Bilwa S T created HADOOP-17763:
--
Summary: DistCp job fails when AM is killed
Key: HADOOP-17763
URL: https://issues.apache.org/jira/browse/HADOOP-17763
Project: Hadoop Common
Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T
Job fails as tasks fail with below exception
{code:java}
2021-06-11 18:48:47,047 | ERROR | IPC Server handler 0 on 27101 | Task:
attempt_1623387358383_0006_m_00_1000 - exited :
java.io.FileNotFoundException: File does not exist:
hdfs://hacluster/staging-dir/dsperf/.staging/_distcp-646531269/fileList.seq
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1637)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1630)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1645)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1863)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1886)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$1.run(YarnChild.java:183)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:177)
| TaskAttemptListenerImpl.java:304{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0
+1(non-binding) 1. Deployed 3 node cluster 2. Browsed through Web UI (RM, NM) 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort) 4. Verified basic yarn commands Thanks, Bilwa -Original Message- From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com] Sent: 12 July 2020 18:32 To: hemanth boyina Cc: Iñigo Goiri ; Vinayakumar B ; Brahma Reddy Battula ; mapreduce-dev ; Hdfs-dev ; Hadoop Common ; yarn-dev Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0 +1(binding) Deployed HDFS and Yarn Cluster > Verified basic shell commands > Ran some jobs > Verified UI -Surendra On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina wrote: > +1(non-binding) > Deployed Cluster with Namenodes and Router *)verified shell commands > *)Executed various jobs *)Browsed UI's > > > Thanks, > HemanthBoyina > > > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri, wrote: > > > +1 (Binding) > > > > Deployed a cluster on Azure VMs with: > > * 3 VMs with HDFS Namenodes and Routers > > * 2 VMs with YARN Resource Managers > > * 5 VMs with HDFS Datanodes and Node Managers > > > > Tests: > > * Executed Tergagen+Terasort+Teravalidate. > > * Executed wordcount. > > * Browsed through the Web UI. > > > > > > > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B > > > > wrote: > > > > > +1 (Binding) > > > > > > -Verified all checksums and Signatures. > > > -Verified site, Release notes and Change logs > > > + May be changelog and release notes could be grouped based on > > > the project at second level for better look (this needs to be > > > supported > from > > > yetus) > > > -Tested in x86 local 3-node docker cluster. > > > + Built from source with OpenJdk 8 and Ubuntu 18.04 > > > + Deployed 3 node docker cluster > > > + Ran various Jobs (wordcount, Terasort, Pi, etc) > > > > > > No Issues reported. > > > > > > -Vinay > > > > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages > > > > > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04 > > > > ARM/aarch64, openjdk-11-jdk > > > > > > > > - checked some web UIs (NN, DN, RM, NM) > > > > > > > > - Executed a wordcount, TeraGen, TeraSort and TeraValidate > > > > > > > > - Executed a TestDFSIO job > > > > > > > > - Executed a Pi job > > > > > > > > BR, > > > > Liusheng > > > > > > > > Zhenyu Zheng 于2020年7月10日周五 下午3:45写道： > > > > > > > > > +1 (non-binding) > > > > > > > > > > - Verified all hashes and checksums > > > > > - Tested on ARM platform for the following actions: > > > > > + Built from source on Ubuntu 18.04, OpenJDK 8 > > > > > + Deployed a pseudo cluster > > > > > + Ran some example jobs(grep, wordcount, pi) > > > > > + Ran teragen/terasort/teravalidate > > > > > + Ran TestDFSIO job > > > > > > > > > > BR, > > > > > > > > > > Zhenyu > > > > > > > > > > On Fri, Jul 10, 2020 at 2:40 PM Akira Ajisaka > > > > > > > > > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > - Verified checksums and signatures. > > > > > > - Built from the source with CentOS 7 and OpenJDK 8. > > > > > > - Successfully upgraded HDFS to 3.3.0-RC0 in our development > > cluster > > > > > (with > > > > > > RBF, security, and OpenJDK 11) for end-users. No issues reported. > > > > > > - The document looks good. > > > > > > - Deployed pseudo cluster and ran some MapReduce jobs. > > > > > > > > > > > > Thanks, > > > > > > Akira > > > > > > > > > > > > > > > > > > On Tue, Jul 7, 2020 at 7:27 AM Brahma Reddy Battula < > > > bra...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > This is the first release candidate for the first release > > > > > > > of > > Apache > > > > > > > Hadoop 3.3.0 > > > > > > > line. > > > > > > > > > > > > > > It contains *1644[1]* fixed jira issues since 3.2.1 which > > include a > > > > lot > > > > > > of > > > > > > > features and improvements(read the full set of release notes). > > > > > > > > > > > > > > Below feature additions are the highlights of the release. > > > > > > > > > > > > > > - ARM Support > > > > > > > - Enhancements and new features on S3a,S3Guard,ABFS > > > > > > > - Java 11 Runtime support and TLS 1.3. > > > > > > > - Support Tencent Cloud COS File System. > > > > > > > - Added security to HDFS Router. > > > > > > > - Support non-volatile storage class memory(SCM) in HDFS > > > > > > > cache > > > > > directives > > > > > > > - Support Interactive Docker Shell for running Containers. > > > > > > > - Scheduling of opportunistic containers > > > > > > > - A pluggable device plugin framework to ease vendor > > > > > > > plugin > > > > development > > > > > > > > > > > > > > *The RC0 artifacts are at*: > > > > > > > http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/ > > > > > > > > > > > > > > *First release to include ARM binary, Have a check.* *RC > > > > > > > tag is *release-3.3.0-RC0. > > > > > > > > > > > > > > > > > > > > > *The maven artifacts
[jira] [Created] (HADOOP-17119) Netty upgrade to 9.4.x causes MR app fail with IOException
Bilwa S T created HADOOP-17119:
--
Summary: Netty upgrade to 9.4.x causes MR app fail with IOException
Key: HADOOP-17119
URL: https://issues.apache.org/jira/browse/HADOOP-17119
Project: Hadoop Common
Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T
I think we should catch IOException here instead of BindException in
HttpServer2#bindForPortRange
{code:java}
for(Integer port : portRanges) {
if (port == startPort) {
continue;
}
Thread.sleep(100);
listener.setPort(port);
try {
bindListener(listener);
return;
} catch (BindException ex) {
// Ignore exception. Move to next port.
ioException = ex;
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16391) Duplicate values in rpcDetailedMetrics
Bilwa S T created HADOOP-16391: -- Summary: Duplicate values in rpcDetailedMetrics Key: HADOOP-16391 URL: https://issues.apache.org/jira/browse/HADOOP-16391 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Attachments: image-2019-06-25-20-30-15-395.png In RpcDetailedMetrics init is called two times . Once for deferredRpcrates and other one rates metrics which causes duplicate values in RM and NM metrics. !image-2019-06-25-20-30-15-395.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15523) Shell command timeout given is in seconds whereas it is taken as millisec while scheduling
Bilwa S T created HADOOP-15523:
--
Summary: Shell command timeout given is in seconds whereas it is
taken as millisec while scheduling
Key: HADOOP-15523
URL: https://issues.apache.org/jira/browse/HADOOP-15523
Project: Hadoop Common
Issue Type: Bug
Reporter: Bilwa S T
ShellBasedUnixGroupsMapping has a property
{{hadoop.security.groups.shell.command.timeout}} to control how long to wait
for the fetch groups command which can be configured in seconds. but while
scheduling the time taken is millisecs. so currently if u give value as 60s, it
is taken as 60ms.
{code:java}
timeout = conf.getTimeDuration(
CommonConfigurationKeys.
HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT_SECS,
CommonConfigurationKeys.
HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT_SECS_DEFAULT,
TimeUnit.SECONDS);{code}
Time unit given is in seconds but it should be millisecs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
