[jira] [Created] (YARN-6397) Support kerberos deployment for yarn-native-service REST API

2017-03-26 Thread Jian He (JIRA) 
Jian He created YARN-6397:
-

 Summary: Support kerberos deployment for yarn-native-service REST 
API
 Key: YARN-6397
 URL: https://issues.apache.org/jira/browse/YARN-6397
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He


Current service REST API doesn't support kerberos deployment. User should be 
able to run their service with kerberos on by specifying keytabs, principal 
etc. with the REST API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node

2017-03-26 Thread zhihai xu (JIRA) 
zhihai xu created YARN-6396:
---

 Summary: Call verifyAndCreateRemoteLogDir at service 
initialization instead of application initialization to decrease load for name 
node
 Key: YARN-6396
 URL: https://issues.apache.org/jira/browse/YARN-6396
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: log-aggregation
Affects Versions: 3.0.0-alpha2
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor


Call verifyAndCreateRemoteLogDir at service initialization instead of 
application initialization to decrease load for name node.
Currently for every application at each Node, verifyAndCreateRemoteLogDir will 
be called before doing log aggregation, This will be a non trivial overhead for 
name node in a large cluster since verifyAndCreateRemoteLogDir calls 
getFileStatus. Once the remote log directory is created successfully, it is not 
necessary to call it again. It will be better to call 
verifyAndCreateRemoteLogDir at LogAggregationService service initialization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6395) Integrate service app master to write data into ATSv2.

2017-03-26 Thread Rohith Sharma K S (JIRA) 
Rohith Sharma K S created YARN-6395:
---

 Summary: Integrate service app master to write data into ATSv2.
 Key: YARN-6395
 URL: https://issues.apache.org/jira/browse/YARN-6395
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S


Integration of ATSv2 with native service app master. And ATSv2 data's are used 
for UI rendering. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6394) Support specifying YARN related params in the service REST API

2017-03-26 Thread Jian He (JIRA) 
Jian He created YARN-6394:
-

 Summary: Support specifying YARN related params in the service 
REST API
 Key: YARN-6394
 URL: https://issues.apache.org/jira/browse/YARN-6394
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn-native-services
Reporter: Jian He


Today user can specify a lot of YARN parameters (such as LogAggregationContext. 
i.e. all the fields in ApplicationSubmissionContext). The service REST API 
hasn't accounted for so far. We need a way to allow users to specify those 
configs when submit their service app. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6393) Create a API class for yarn-native-service user-facing constants

2017-03-26 Thread Jian He (JIRA) 
Jian He created YARN-6393:
-

 Summary: Create a API class for yarn-native-service user-facing 
constants
 Key: YARN-6393
 URL: https://issues.apache.org/jira/browse/YARN-6393
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He


User can use some constants  in the json input spec file for later substitution.
e.g. if user specifies $HOSTNAME in the env section of the input file, it'll be 
substituted by AM with the actual host name. We'll need to create an API class 
and clearly documents it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6392) add submit time to Application Summary log

2017-03-26 Thread zhihai xu (JIRA) 
zhihai xu created YARN-6392:
---

 Summary: add submit time to Application Summary log
 Key: YARN-6392
 URL: https://issues.apache.org/jira/browse/YARN-6392
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: resourcemanager
Affects Versions: 3.0.0-alpha2
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor


add submit time to Application Summary log, application submit time will be 
passed to Application Master in env variable "APP_SUBMIT_TIME_ENV". It is a 
very important parameter, So it will be useful to log it in Application Summary.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6391) Support specifying extra options from yarn-native-service CLI

2017-03-26 Thread Jian He (JIRA) 
Jian He created YARN-6391:
-

 Summary: Support specifying extra options from yarn-native-service 
CLI
 Key: YARN-6391
 URL: https://issues.apache.org/jira/browse/YARN-6391
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He


The CLI has been changed to take the same json input spec as YARN-4692.
We should also have a way to allow for substituting individual field of the 
json spec file from CLI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6390) Support service assembly

2017-03-26 Thread Jian He (JIRA) 
Jian He created YARN-6390:
-

 Summary: Support service assembly
 Key: YARN-6390
 URL: https://issues.apache.org/jira/browse/YARN-6390
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-native-services
Reporter: Jian He


An assembly is a hierarchical app-of-apps. Say, an assembly could be a 
combination of zookeeper + hbase + kafka. 
This functionality was there in slider, need to re-implement this in the new 
yarn-native-service framework.
Also, the new yarn-native-service UI needs to account for the assembly concept.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

2017-03-26 Thread Junping Du 
Quick update: the missing bits in svn release directories (reported in 
https://issues.apache.org/jira/browse/INFRA-13749) already get resolved with 
workaround of bypassing 200M file size limitation. We should document this in 
our release wiki.
However, a new issue is found that releasing bits are missing from mirror sites 
which get reported in https://issues.apache.org/jira/browse/INFRA-13755. The 
Apache infrastructure team is investigating right now. Will send release news 
when issue get fixed. Stay tuned!

Thanks,

Junping

From: Junping Du 
Sent: Friday, March 24, 2017 6:21 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Thanks Allen, I already add it back in JIRA's 2.8.0 release date.
However, I met another issue in uploading our 2.8.0 release bits to SVN: 
hadoop-2.8.0.tar.gz get uploaded failed with following exception:

Adding  (bin)  hadoop-2.8.0/hadoop-2.8.0.tar.gz
Transmitting file data .svn: E175002: Commit failed (details follow):
svn: E175002: PUT request on 
'/repos/dist/!svn/txr/18902-g1s/release/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz'
 failed

Other (smaller) files get uploaded successfully. Once I suspected it could due 
to no space, so I removed some old release bits (like: 2.5.2) but still no 
luck. So may be the size of hadoop-2.8.0.tar.gz become a problem here?
I already reported the issue in 
https://issues.apache.org/jira/browse/INFRA-13749, but will appreciate if 
someone got idea on how to get through this.

Thanks,

Junping

From: Allen Wittenauer 
Sent: Thursday, March 23, 2017 5:27 PM
To: Junping Du
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [RESULT] [VOTE] Release Apache Hadoop 2.8.0 (RC3)

Just a heads up.  Looks like some removed the Finish Date off of 2.8.0 in JIRA. 
 It needs to be put back to match what is in the artifacts that we voted on.

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2017-03-26 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/

[Mar 25, 2017 4:56:19 PM] (sunilg) YARN-5956. Refactor ClientRMService for 
unify error handling across
[Mar 25, 2017 9:02:25 PM] (junping_du) Add 2.8.0 RELEASENOTEs.md and CHANGES.md.
[Mar 25, 2017 9:05:20 PM] (junping_du) Add HDFS jdiff for 2.8.0 release.
[Mar 25, 2017 9:28:48 PM] (junping_du) Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 
release.
[Mar 25, 2017 9:30:38 PM] (junping_du) Set jdiff.stable.api to 2.8.0




-1 overall


The following subsystems voted -1:
compile unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration 
   hadoop.hdfs.TestRollingUpgrade 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
  

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-compile-root.txt
  [140K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-compile-root.txt
  [140K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-compile-root.txt
  [140K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [52K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/269/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-ui.txt
  [8.0K]

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-03-26 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/

[Mar 25, 2017 4:56:19 PM] (sunilg) YARN-5956. Refactor ClientRMService for 
unify error handling across
[Mar 25, 2017 9:02:25 PM] (junping_du) Add 2.8.0 RELEASENOTEs.md and CHANGES.md.
[Mar 25, 2017 9:05:20 PM] (junping_du) Add HDFS jdiff for 2.8.0 release.
[Mar 25, 2017 9:28:48 PM] (junping_du) Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 
release.
[Mar 25, 2017 9:30:38 PM] (junping_du) Set jdiff.stable.api to 2.8.0




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.security.TestRaceWhenRelogin 
   hadoop.net.TestDNS 
   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestDiskFailures 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps 
   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun 
   
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity 
   
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-compile-javac-root.txt
  [184K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-patch-shellcheck.txt
  [24K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [136K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [336K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [324K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
  [24K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/357/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org