[jira] [Created] (YARN-8870) Add submarine installation scripts

2018-10-10 Thread Xun Liu (JIRA) 
Xun Liu created YARN-8870:
-

 Summary: Add submarine installation scripts
 Key: YARN-8870
 URL: https://issues.apache.org/jira/browse/YARN-8870
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xun Liu


In order to reduce the deployment difficulty of Hadoop {Submarine} DNS, Docker, 
GPU, Network, graphics card, operating system kernel modification and other 
components, I specially developed this installation script to deploy Hadoop 
{Submarine} runtime environment, providing one-click installation Scripts, 
which can also be used to install, uninstall, start, and stop individual 
components step by step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8869) YARN client might not work correctly with RM Rest API for Kerberos authentication

2018-10-10 Thread Eric Yang (JIRA) 
Eric Yang created YARN-8869:
---

 Summary: YARN client might not work correctly with RM Rest API for 
Kerberos authentication
 Key: YARN-8869
 URL: https://issues.apache.org/jira/browse/YARN-8869
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 3.1.1, 3.1.0, 3.2.0
Reporter: Eric Yang


ApiServiceClient uses WebResource instead of Builder to pass Kerberos 
authorization header.  This may not work sometimes, and it is because 
WebResource.header() could be bind mashed build new Builder instance in some 
condition.  This article explained the details: 
https://juristr.com/blog/2015/05/jersey-webresource-ignores-headers/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8868) Set HTTPOnly attribute to Cookie

2018-10-10 Thread Chandni Singh (JIRA) 
Chandni Singh created YARN-8868:
---

 Summary: Set HTTPOnly attribute to Cookie
 Key: YARN-8868
 URL: https://issues.apache.org/jira/browse/YARN-8868
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Chandni Singh
Assignee: Chandni Singh


1. The program creates a cookie in Dispatcher.java at line 182, 185 and 199, 
but fails to set the HttpOnly flag to true.

2. The program creates a cookie in WebAppProxyServlet.java at line 141 and 388, 
but fails to set the HttpOnly flag to true.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8867) Retrieve the progress of resource localization

2018-10-10 Thread Chandni Singh (JIRA) 
Chandni Singh created YARN-8867:
---

 Summary: Retrieve the progress of resource localization
 Key: YARN-8867
 URL: https://issues.apache.org/jira/browse/YARN-8867
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn
Reporter: Chandni Singh
Assignee: Chandni Singh


Refer YARN-3854.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-10-10 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/922/

[Oct 9, 2018 2:33:13 AM] (yqlin) HDFS-13962. Add null check for add-replica 
pool to avoid lock acquiring.
[Oct 9, 2018 3:31:57 AM] (xiao) HDFS-13926. ThreadLocal aggregations for 
FileSystem.Statistics are
[Oct 9, 2018 3:55:34 AM] (aajisaka) HADOOP-15818. Fix deprecated 
maven-surefire-plugin configuration in
[Oct 9, 2018 4:06:14 AM] (tasanuma) HADOOP-15775. [JDK9] Add missing 
javax.activation-api dependency.
[Oct 9, 2018 8:11:07 AM] (yqlin) HDFS-13967. HDFS Router Quota Class Review. 
Contributed by BELUGA BEHR.
[Oct 9, 2018 9:46:41 AM] (stevel) HADOOP-15827. NPE in 
DynamoDBMetadataStore.lambda$listChildren for root
[Oct 9, 2018 12:13:04 PM] (aajisaka) MAPREDUCE-7035. Skip javadoc build for 
auto-generated sources in
[Oct 9, 2018 12:37:01 PM] (nanda) HDDS-450. Generate BlockCommitSequenceId in 
ContainerStateMachine for
[Oct 9, 2018 12:53:05 PM] (nanda) HDDS-599. Fix TestOzoneConfiguration 
TestOzoneConfigurationFields.
[Oct 9, 2018 4:34:17 PM] (sunilg) YARN-8853. [UI2] Application Attempts tab is 
not shown correctly when
[Oct 9, 2018 5:04:25 PM] (elek) HDDS-602. Bump Ozone version to 0.4.0-SNAPSHOT 
after ozone-0.3 branch
[Oct 9, 2018 5:18:27 PM] (wangda) YARN-8852. Add documentation for submarine 
installation details. (Zac
[Oct 9, 2018 5:29:40 PM] (inigoiri) YARN-7652. Handle AM register requests 
asynchronously in
[Oct 9, 2018 6:04:43 PM] (arp) HDDS-590. Add unit test for HDDS-583. 
Contributed by Namit Maheshwari.
[Oct 9, 2018 6:27:03 PM] (jlowe) MAPREDUCE-7130. Rumen crashes trying to handle 
MRAppMaster recovery
[Oct 9, 2018 7:00:21 PM] (inigoiri) HDFS-13979. Review StateStoreFileSystemImpl 
Class. Contributed by BELUGA
[Oct 9, 2018 7:02:12 PM] (stevel) HADOOP-15825. ABFS: Enable some tests for 
namespace not enabled account
[Oct 9, 2018 7:43:04 PM] (bharat) HDDS-585. Handle common request identifiers 
in a transparent way.
[Oct 9, 2018 7:54:45 PM] (eyang) YARN-8845.  Removed unused hadoop.registry.rm 
reference.
[Oct 9, 2018 8:37:42 PM] (aengineer) HDDS-577. Support S3 buckets as first 
class objects in Ozone Manager -
[Oct 9, 2018 10:21:19 PM] (bharat) HDDS-583. SCM returns zero as the return 
code, even when invalid options
[Oct 9, 2018 11:44:32 PM] (aengineer) HDDS-478. Log files related to each 
daemon doesn't have proper startup
[Oct 9, 2018 11:57:39 PM] (aengineer) HDDS-559. fs.default.name is deprecated. 
Contributed by  Dinesh
[Oct 10, 2018 12:03:21 AM] (inigoiri) HDFS-11396. 
TestNameNodeMetadataConsistency#testGenerationStampInFuture
[Oct 10, 2018 12:32:04 AM] (aengineer) HDDS-568. ozone sh volume info, update, 
delete operations fail when
[Oct 10, 2018 12:52:06 AM] (aengineer) HDDS-604. Correct Ozone getOzoneConf 
description. Contributed by Dinesh
[Oct 10, 2018 1:16:57 AM] (aajisaka) HADOOP-15832. Upgrade BouncyCastle to 
1.60. Contributed by Robert
[Oct 10, 2018 1:19:00 AM] (aengineer) HDDS-443. Create reusable ProgressBar 
utility for freon tests.
[Oct 10, 2018 1:32:27 AM] (wwei) YARN-8858. CapacityScheduler should respect 
maximum node resource when




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized 
   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage 
   hadoop.yarn.client.api.impl.TestAMRMProxy 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   hadoop.yarn.sls.TestSLSRunner 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/922/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/922/artifact/out/diff-compile-javac-root.txt
  [300K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/922/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/922/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/922/artifact/out/pathlen.txt
  [12K]

   pylint:

Re: Resourcemanager Failing on OpenJDK11

2018-10-10 Thread Jeremiah Adams 
Akira,

Java 8 is going to cost us all a lot of money come January.  Java 9 and Java 10 
are a waste of time to support. 9 and 10 will no longer receive security 
patches. The LTS is 11 and where we should all move unless one has deep pockets.

What is the process for getting community support for Java11? 

Is it enough to simply start completing the tasks on the Hadoop and Java 9, 10 
and 11 Wiki?


Jeremiah Adams
Software Engineer
www.helixeducation.com
Blog | Twitter | Facebook | LinkedIn


From: Akira Ajisaka 
Sent: Tuesday, October 2, 2018 9:37 PM
To: Jeremiah Adams
Cc: yarn-dev@hadoop.apache.org
Subject: Re: Resourcemanager Failing on OpenJDK11

Hi Jeremiah,

I wrote a patch to fix this issue in
https://url.emailprotection.link/?aXbNFujPDP4Cfs-6OQ7jlf8T9J_NulRd7SGvkTYwMFSCcpT8rfYP_77HQteuHN0GqNu5ThkVsxWK-Ca0mFfv2cw~~

There is also a wiki to document the progress of Java 9, 10, and 11 support.
https://url.emailprotection.link/?aqtDT9nGZlDfQ6CDagckDXG_qA-bYuc0oOhrPxAJMZ8UvE7Xxij3gkTp-tag4iCp0lulSdnHJIAUtI4wnRr7z_jKarDj5icmYom3DOfadSyYa3mOomaEzNFas3PeOfvND

Now there is no community consensus regarding the plan of Java 11 support,
however, I'd like to support Java 11 in Apache Hadoop 3.3 release.

Thanks,
Akira
2018年10月3日(水) 4:14 Jeremiah Adams :
>
> I am doing some testing of OpenJDK 11 and YARN, Kafka and Samza. I found that 
> I YARN Resourcemanager will not run on OpenJDK11. I didn’t see any tasking in 
> JIRA regarding Java 11.
>
> What are YARN’s plans regarding OpenJDK11 and the changes to Oracle support 
> and release cadences? Is there an Epic or Stories regarding Java 11 that I 
> can add this issue to?
>
>
> The issue with Resourcemanager is the WebAppContext failing:
>
>
> Caused by: java.lang.NoClassDefFoundError: javax/activation/DataSource
>
>
> The Activation package was lumped in with J2EE and CORBA for removal. 
> Deprecated in v. 9, marked for removal and removed in v.11. Now gone.
>
>
>
> Output from the logs:
>
>
>
>
> 2018-10-02 07:36:37,410 WARN org.eclipse.jetty.webapp.WebAppContext: Failed 
> startup of context 
> o.e.j.w.WebAppContext@43d3aba5{/,file:///private/var/folders/9y/92nwpmbd6pjf4m68mkcw29z4gn/T/jetty-0.0.0.0-8042-node-_-any-10842369110863142525.dir/webapp/,UNAVAILABLE}{/node}
>
> com.google.inject.ProvisionException: Unable to provision, see the following 
> errors:
>
>
> 1) Error injecting constructor, java.lang.NoClassDefFoundError: 
> javax/activation/DataSource
>
>  at 
> org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver.(JAXBContextResolver.java:52)
>
>  at 
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer$NMWebApp.setup(WebServer.java:153)
>
>  while locating 
> org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver
>
>
> 1 error
>
>at 
> com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025)
>
>at 
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
>
>at 
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory$GuiceInstantiatedComponentProvider.getInstance(GuiceComponentProviderFactory.java:345)
>
>at 
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory$ManagedSingleton.(IoCProviderFactory.java:202)
>
>at 
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory.wrap(IoCProviderFactory.java:123)
>
>at 
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory._getComponentProvider(IoCProviderFactory.java:116)
>
>at 
> com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153)
>
>at 
> com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:278)
>
>at 
> com.sun.jersey.core.spi.component.ProviderServices.getProviders(ProviderServices.java:151)
>
>at 
> com.sun.jersey.core.spi.factory.ContextResolverFactory.init(ContextResolverFactory.java:83)
>
>at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1332)
>
>at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180)
>
>at 
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799)
>
>at 
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795)
>
>at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
>
>at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:795)
>
>at 
> com.sun.jersey.guice.spi.container.servlet.GuiceContainer.initiate(GuiceContainer.java:121)
>
>at 
> com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:339)
>
>at 
> com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:605)
>
>at 
>

[jira] [Created] (YARN-8866) Fix a parsing error for crossdomain.xml

2018-10-10 Thread Takanobu Asanuma (JIRA) 
Takanobu Asanuma created YARN-8866:
--

 Summary: Fix a parsing error for crossdomain.xml
 Key: YARN-8866
 URL: https://issues.apache.org/jira/browse/YARN-8866
 Project: Hadoop YARN
  Issue Type: Bug
  Components: build, yarn-ui-v2
Reporter: Takanobu Asanuma
Assignee: Takanobu Asanuma


[QBT|https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/] reports a 
parsing error for crossdomain.xml in hadoop-yarn-ui.
{noformat}
Parsing Error(s): 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8865) RMStateStore contains large number of expired RMDelegationToken

2018-10-10 Thread Wilfred Spiegelenburg (JIRA) 
Wilfred Spiegelenburg created YARN-8865:
---

 Summary: RMStateStore contains large number of expired 
RMDelegationToken
 Key: YARN-8865
 URL: https://issues.apache.org/jira/browse/YARN-8865
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 3.1.0
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg


When the RM state store is restored expired delegation tokens are restored and 
added to the system. These expired tokens do not get cleaned up or removed. The 
exact reason why the tokens are still in the store is not clear. We have seen 
as many as 250,000 tokens in the store some of which were 2 years old.

This has two side effects:
* for the zookeeper store this leads to a jute buffer exhaustion issue and 
prevents the RM from becoming active.
* restore takes longer than needed and heap usage is higher than it should be

We should not restore already expired tokens since they cannot be renewed or 
used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org