Re: 2.8 Release activities

[Junping Du]

Haohui,
I am waiting for a special security release on branch-2.8 get out to resume 
the release work for production release of 2.8. You should be on security alias 
and ask for update there.

Thanks,

Junping


From: Haohui Mai 
Sent: Wednesday, July 12, 2017 11:48 PM
To: Hadoop Common; yarn-dev@hadoop.apache.org; Hdfs-dev
Subject: 2.8 Release activities

Hi,

Just curious -- what is the current status of the 2.8 release? It looks
like the release process for some time.

There are 5 or 6 blocker / critical bugs of the upcoming 2.8 release:

https://issues.apache.org/jira/browse/YARN-6654?jql=project%20in%20(HDFS%2C%20HADOOP%2C%20MAPREDUCE%2C%20YARN)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20%22Target%20Version%2Fs%22%20in%20(2.8.2%2C%202.8.3)

I think we can address them in reasonable amount of effort.

We are interested in putting 2.8.x in production and it would be great to
have a maintenance Apache release for the 2.8 line.

I wonder, are there any concerns of not getting the release out? We might
be able to get some helps internally to fix the issues in the 2.8 lines. I
can also volunteer to be the release manager for 2.8.2 if it requires more
effort to coordinate to push the release out.

Regards,
Haohui



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Konstantin Shvachko]

Hi everybody.

We have been doing some internal testing of Hadoop 2.7.4. The testing is
going well.
Did not find any major issues on our workloads.
Used an internal tool called Dynamometer to check NameNode performance on
real cluster traces. Good.
Overall test cluster performance looks good.
Some more testing is still going on.

I plan to build an RC next week. If there are no objection.

Thanks,
--Konst

On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko 
wrote:

> Hey guys.
>
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
>
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
>
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
>
> There is a plan to enable BigTop for 2.7.4 testing.
>
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
>
> Regards,
> --Konstantin
>
>
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka 
> wrote:
>
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence account.
>>
>> -Akira
>>
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira/browse/YARN-6304
>>>
>>> I will raise a JIRAs to back port it.
>>>
>>> @Akira , could  you help to add these JIRAs into wiki?
>>>
>>> Thanks & Regards
>>> Rohith Sharma K S
>>>
>>> On 29 May 2017 at 12:19, Akira Ajisaka  wrote:
>>>
>>> Created a page for 2.7.4 release.
 https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4

 If you want to edit this wiki, please ping me.

 Regards,
 Akira


 On 2017/05/23 4:42, Brahma Reddy Battula wrote:

 Hi Konstantin Shvachko
>
>
> how about creating a wiki page for 2.7.4 release status like 2.8 and
> trunk in following link.??
>
>
> https://cwiki.apache.org/confluence/display/HADOOP
>
>
> 
> From: Konstantin Shvachko 
> Sent: Saturday, May 13, 2017 3:58 AM
> To: Akira Ajisaka
> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: About 2.7.4 Release
>
> Latest update on the links and filters. Here is the correct link for
> the
> filter:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
> requestId=12340814
>
> Also updated: https://s.apache.org/Dzg4
>
> Had to do some Jira debugging. Sorry for confusion.
>
> Thanks,
> --Konstantin
>
> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
> shv.had...@gmail.com>
> wrote:
>
> Hey Akira,
>
>>
>> I didn't have private filters. Most probably Jira caches something.
>> Your filter is in the right direction, but for some reason it lists
>> only
>> 22 issues, while mine has 29.
>> It misses e.g. YARN-5543 > a/browse/YARN-5543>
>> .
>>
>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>> shared it with "everybody", and updated my link to point to that
>> filter.
>> So
>> you can use any of the three methods below to get the correct list:
>> 1. Go to https://s.apache.org/Dzg4
>> 2. Go to the filter via
>> https://issues.apache.org/jira/issues?filter=12340814
>>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>> 3. On Advanced issues search page paste this:
>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels =
>> release-blocker
>> AND "Target Version/s" = 2.7.4
>>
>> Hope this solves the confusion for which issues are included.
>> Please LMK if it doesn't, as it is important.
>>
>> Thanks,
>> --Konstantin
>>
>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka 
>> wrote:
>>
>> Hi Konstantin,
>>
>>>
>>> Thank you for volunteering as release manager!
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>

 I couldn't see the link. Maybe is it private filter?
>>>
>>> Here is a link I generated: https://s.apache.org/ehKy

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/

[Jul 12, 2017 8:52:56 PM] (jlowe) YARN-6797. TimelineWriter does not fully 
consume the POST response.
[Jul 12, 2017 9:15:04 PM] (szetszwo) HDFS-6874. Add GETFILEBLOCKLOCATIONS 
operation to HttpFS.  Contributed
[Jul 12, 2017 10:40:45 PM] (xgong) YARN-6689. PlacementRule should be 
configurable. (Jonathan Hung via
[Jul 12, 2017 11:26:19 PM] (xyao) HDFS-11502. Datanode UI should display 
hostname based on JMX bean
[Jul 13, 2017 11:18:29 AM] (sunilg) YARN-5731. Preemption calculation is not 
accurate when reserved
[Jul 13, 2017 12:41:43 PM] (iwasakims) HADOOP-14646.




-1 overall


The following subsystems voted -1:
compile mvninstall unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc javac


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.namenode.TestDecommissioningStatus 
   hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer 
   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation 
   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy 
   hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService 
   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown 
   hadoop.yarn.server.timeline.TestRollingLevelDB 
   hadoop.yarn.server.timeline.TestTimelineDataManager 
   hadoop.yarn.server.timeline.TestLeveldbTimelineStore 
   hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore 
   hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore 
   
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer 
   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector 
   hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore 
   hadoop.yarn.server.resourcemanager.TestRMRestart 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore 
   hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient 
   hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore 
   hadoop.yarn.applications.distributedshell.TestDistributedShell 
   hadoop.mapred.TestShuffleHandler 
   hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands 
   
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
   org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels 
  

   mvninstall:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-mvninstall-root.txt
  [620K]

   compile:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-compile-root.txt
  [20K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-compile-root.txt
  [20K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-compile-root.txt
  [20K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-unit-hadoop-assemblies.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [680K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [80K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/374/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  

[jira] [Created] (YARN-6821) Move FederationStateStore SQL DDL files from test to main

[Subru Krishnan (JIRA)]

Subru Krishnan created YARN-6821:


 Summary: Move FederationStateStore SQL DDL files from test to main
 Key: YARN-6821
 URL: https://issues.apache.org/jira/browse/YARN-6821
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Subru Krishnan


The FederationStateStore SQL DDL files are currently in _src/test_ as there's 
no compile time dependency. This jira proposes to move them to _src/main_ to 
ensure they are part of the distro.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6820) Restrict read access to timelineservice v2 data

[Vrushali C (JIRA)]

Vrushali C created YARN-6820:


 Summary: Restrict read access to timelineservice v2 data 
 Key: YARN-6820
 URL: https://issues.apache.org/jira/browse/YARN-6820
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vrushali C


Need to provide a way to restrict read access in ATSv2. Not all users should be 
able to read all entities. On the flip side, some folks may not need any read 
restrictions, so we need to provide a way to disable this access restriction as 
well. 

Initially this access restriction could be done in a simple way via a whitelist 
of users allowed to read data. That set of users can read all data, no other 
user can read any data. Can be turned off for all users to read all data.

Could be stored in a "domain" table in hbase perhaps. Or a configuration 
setting for the cluster. Or something else that's simple enough. ATSv1 has a 
concept of domain for isolating users for reading. Would be good to keep that 
in consideration. 

In ATSv1, domain offers a namespace for Timeline server allowing users to host 
multiple entities, isolating them from other users and applications. A “Domain” 
in ATSV1 primarily stores owner info, read and& write ACL information, created 
and modified time stamp information. Each Domain is identified by an ID which 
must be unique across all users in the YARN cluster.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/463/

[Jul 12, 2017 9:37:39 AM] (stevel) HADOOP-14581. Restrict setOwner to list of 
user when security is enabled
[Jul 12, 2017 10:38:32 AM] (aajisaka) YARN-6809. Fix typo in 
ResourceManagerHA.md. Contributed by Yeliang
[Jul 12, 2017 8:52:56 PM] (jlowe) YARN-6797. TimelineWriter does not fully 
consume the POST response.
[Jul 12, 2017 9:15:04 PM] (szetszwo) HDFS-6874. Add GETFILEBLOCKLOCATIONS 
operation to HttpFS.  Contributed
[Jul 12, 2017 10:40:45 PM] (xgong) YARN-6689. PlacementRule should be 
configurable. (Jonathan Hung via
[Jul 12, 2017 11:26:19 PM] (xyao) HDFS-11502. Datanode UI should display 
hostname based on JMX bean




-1 overall


The following subsystems voted -1:
compile findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs-client 
   Possible exposure of partially initialized object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:[line 2888] 
   org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) 
makes inefficient use of keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:[line 105] 

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to 
return value of called method Dereferenced at 
JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus()
 due to return value of called method Dereferenced at JournalNode.java:[line 
302] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String)
 unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId 
At HdfsServerConstants.java:[line 193] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int)
 unconditionally sets the field force At HdfsServerConstants.java:force At 
HdfsServerConstants.java:[line 217] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean)
 unconditionally sets the field isForceFormat At 
HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean)
 unconditionally sets the field isInteractiveFormat At 
HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 
237] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, 
int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at 
DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File,
 File, int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at DataStorage.java:[line 1339] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:[line 258] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, 
BasicFileAttributes) due to return value of called method Dereferenced at 
NNUpgradeUtil.java:org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path,
 BasicFileAttributes) due to return value of called method Dereferenced at 
NNUpgradeUtil.java:[line 133] 
   Useless condition:argv.length >= 1 at this point At DFSAdmin.java:[line 
2085] 
   Useless condition:numBlocks == -1 at this point At 
ImageLoaderCurrent.java:[line 727] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Useless object stored in variable removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:[line 642] 
   

[jira] [Created] (YARN-6819) Application report fails if app rejected due to nodesize

[Bibin A Chundatt (JIRA)]

Bibin A Chundatt created YARN-6819:
--

 Summary: Application report fails if app rejected due to nodesize
 Key: YARN-6819
 URL: https://issues.apache.org/jira/browse/YARN-6819
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt


YARN-5006 application rejected and previous state is not set after skipping 
save to store.
We should set  previous state as FAILED in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

2.8 Release activities

[Haohui Mai]

Hi,

Just curious -- what is the current status of the 2.8 release? It looks
like the release process for some time.

There are 5 or 6 blocker / critical bugs of the upcoming 2.8 release:

https://issues.apache.org/jira/browse/YARN-6654?jql=project%20in%20(HDFS%2C%20HADOOP%2C%20MAPREDUCE%2C%20YARN)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20%22Target%20Version%2Fs%22%20in%20(2.8.2%2C%202.8.3)

I think we can address them in reasonable amount of effort.

We are interested in putting 2.8.x in production and it would be great to
have a maintenance Apache release for the 2.8 line.

I wonder, are there any concerns of not getting the release out? We might
be able to get some helps internally to fix the issues in the 2.8 lines. I
can also volunteer to be the release manager for 2.8.2 if it requires more
effort to coordinate to push the release out.

Regards,
Haohui