Re: About 2.7.4 Release

2017-06-15 Thread Konstantin Shvachko 
Hey guys.

An update on 2.7.4 progress.
We are down to 4 blockers. There is some work remaining on those.
https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
Would be good if people could follow up on review comments.

I looked through nightly Jenkins build results for 2.7.4 both on Apache
Jenkins and internal.
Some test fail intermittently, but there no consistent failures. I filed
HDFS-11985 to track some of them.
https://issues.apache.org/jira/browse/HDFS-11985
I do not currently consider these failures as blockers. LMK if some of them
are.

We started internal testing of branch-2.7 on one of our smallish (100+
nodes) test clusters.
Will update on the results.

There is a plan to enable BigTop for 2.7.4 testing.

Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
Thank you everybody for contributing to this effort.

Regards,
--Konstantin


On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka  wrote:

> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>> solve RM HA unstability issues.
>> https://issues.apache.org/jira/browse/YARN-5333
>> https://issues.apache.org/jira/browse/YARN-5988
>> https://issues.apache.org/jira/browse/YARN-6304
>>
>> I will raise a JIRAs to back port it.
>>
>> @Akira , could  you help to add these JIRAs into wiki?
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>> On 29 May 2017 at 12:19, Akira Ajisaka  wrote:
>>
>> Created a page for 2.7.4 release.
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>
>>> If you want to edit this wiki, please ping me.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>
>>> Hi Konstantin Shvachko


 how about creating a wiki page for 2.7.4 release status like 2.8 and
 trunk in following link.??


 https://cwiki.apache.org/confluence/display/HADOOP


 
 From: Konstantin Shvachko 
 Sent: Saturday, May 13, 2017 3:58 AM
 To: Akira Ajisaka
 Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
 yarn-...@hadoop.apache.org
 Subject: Re: About 2.7.4 Release

 Latest update on the links and filters. Here is the correct link for the
 filter:
 https://issues.apache.org/jira/secure/IssueNavigator.jspa?
 requestId=12340814

 Also updated: https://s.apache.org/Dzg4

 Had to do some Jira debugging. Sorry for confusion.

 Thanks,
 --Konstantin

 On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
 shv.had...@gmail.com>
 wrote:

 Hey Akira,

>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists
> only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543  a/browse/YARN-5543>
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that
> filter.
> So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka 
> wrote:
>
> Hi Konstantin,
>
>>
>> Thank you for volunteering as release manager!
>>
>> Actually the original link works fine: https://s.apache.org/Dzg4
>>
>>>
>>> I couldn't see the link. Maybe is it private filter?
>>
>> Here is a link I generated: https://s.apache.org/ehKy
>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>
>> Thanks and Regards,
>> Akira
>>
>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>
>> Hi Brahma Reddy Battula,
>>
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>> Your link excludes closed and resolved issues, which needs
>>> backporting,
>>> and
>>> which we cannot reopen, as discussed in this thread earlier.
>>>
>>> Looked through the issues you proposed:
>>>
>>> HDFS-9311 
>>> Seems like a new feature. It helps failover to

[jira] [Reopened] (HDFS-11736) OIV tests should not write outside 'target' directory.

2017-06-15 Thread Akira Ajisaka (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reopened HDFS-11736:
--

> OIV tests should not write outside 'target' directory.
> --
>
> Key: HDFS-11736
> URL: https://issues.apache.org/jira/browse/HDFS-11736
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Yiqun Lin
>  Labels: newbie++, test
> Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
>
> Attachments: HDFS-11736.001.patch, HDFS-11736.002.patch, 
> HDFS-11736.003.patch, HDFS-11736-branch-2.7.001.patch, 
> HDFS-11736-branch-2.7.002.patch
>
>
> A few tests use {{Files.createTempDir()}} from Guava package, but do not set 
> {{java.io.tmpdir}} system property. Thus the temp directory is created in 
> unpredictable places and is not being cleaned up by {{mvn clean}}.
> This was probably introduced in {{TestOfflineImageViewer}} and then 
> replicated in {{TestCheckpoint}}, {{TestStandbyCheckpoints}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11984) Ozone: Ensures listKey lists all required key fields

2017-06-15 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HDFS-11984:
--

 Summary: Ozone: Ensures listKey lists all required key fields
 Key: HDFS-11984
 URL: https://issues.apache.org/jira/browse/HDFS-11984
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Weiwei Yang


HDFS-11782 implements the listKey operation which only lists the basic key 
fields, we need to make sure it return all required fields

# version
# md5hash
# createdOn
# size
# keyName
# dataFileName

this task is depending on the work of HDFS-11886. See more discussion [here | 
https://issues.apache.org/jira/browse/HDFS-11782?focusedCommentId=16045562=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045562].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11983) Add documentation for metrics in KSMMetrics to OzoneMetrics.md

2017-06-15 Thread Yiqun Lin (JIRA) 
Yiqun Lin created HDFS-11983:


 Summary: Add documentation for metrics in KSMMetrics to 
OzoneMetrics.md
 Key: HDFS-11983
 URL: https://issues.apache.org/jira/browse/HDFS-11983
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: documentation, ozone
Affects Versions: HDFS-7240
Reporter: Yiqun Lin
Assignee: Yiqun Lin


Metrics defined in KSMMetrics are not documented in OzoneMetrics.md. This JIRA 
will track on this. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11982) Add markdown documentation about Ozone

2017-06-15 Thread Mukul Kumar Singh (JIRA) 
Mukul Kumar Singh created HDFS-11982:


 Summary: Add markdown documentation about Ozone
 Key: HDFS-11982
 URL: https://issues.apache.org/jira/browse/HDFS-11982
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
 Fix For: HDFS-7240






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11981) Add markdown documentation about Cblock

2017-06-15 Thread Mukul Kumar Singh (JIRA) 
Mukul Kumar Singh created HDFS-11981:


 Summary: Add markdown documentation about Cblock
 Key: HDFS-11981
 URL: https://issues.apache.org/jira/browse/HDFS-11981
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
 Fix For: HDFS-7240


Cblock should be documented in detail using markdown. This documentation should 
include information about setting up various Cblock components, usage and 
configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11980) Add documentation for metrics in CBlockTargetMetrics to OzoneMetrics.md

2017-06-15 Thread Mukul Kumar Singh (JIRA) 
Mukul Kumar Singh created HDFS-11980:


 Summary: Add documentation for metrics in CBlockTargetMetrics to 
OzoneMetrics.md
 Key: HDFS-11980
 URL: https://issues.apache.org/jira/browse/HDFS-11980
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
 Fix For: HDFS-7240


Metrics defined in CBlockTargetMetrics are not documented in OzoneMetrics.md.

This bug will track documentation of metrics to the bug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11979) Ozone: TestContainerPersistence never uses MiniOzoneCluster

2017-06-15 Thread Tsz Wo Nicholas Sze (JIRA) 
Tsz Wo Nicholas Sze created HDFS-11979:
--

 Summary: Ozone: TestContainerPersistence never uses 
MiniOzoneCluster
 Key: HDFS-11979
 URL: https://issues.apache.org/jira/browse/HDFS-11979
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone, test
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'

2017-06-15 Thread wenxin he (JIRA) 
wenxin he created HDFS-11978:


 Summary: Remove invalid '-usage' command of 'ec' and add missing 
commands 'addPolicies' 'listCodecs'
 Key: HDFS-11978
 URL: https://issues.apache.org/jira/browse/HDFS-11978
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: documentation
Affects Versions: 3.0.0-alpha4
Reporter: wenxin he
Priority: Minor


Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md.
Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11977) Ozone: cannot enable test debug/trace log

2017-06-15 Thread Tsz Wo Nicholas Sze (JIRA) 
Tsz Wo Nicholas Sze created HDFS-11977:
--

 Summary: Ozone: cannot enable test debug/trace log
 Key: HDFS-11977
 URL: https://issues.apache.org/jira/browse/HDFS-11977
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone, test
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Priority: Minor


Interestingly, the test debug/trace logs are not printed for Ozone classes even 
if we invoke GenericTestUtils.setLogLevel(log, Level.ALL).  Other classes such 
as Object do not have such problem.  Here is a test:
{code}
  @Test
  public void testLogLevel() throws Exception {
runTestLogLevel(StorageContainerManager.class);
runTestLogLevel(Object.class);
  }
  static void runTestLogLevel(Class clazz) throws Exception {
final Logger log = LoggerFactory.getLogger(clazz);
GenericTestUtils.setLogLevel(log, Level.ALL);
log.trace(clazz.getSimpleName() + " trace log");
log.debug(clazz.getSimpleName() + " debug log");
log.info(clazz.getSimpleName() + " info log");
log.warn(clazz.getSimpleName() + " warn log");
log.error(clazz.getSimpleName() + " error log");
  }
{code}
Output:
{code}
2017-06-15 00:19:07,133 [Thread-0] INFO   - StorageContainerManager info log
2017-06-15 00:19:07,135 [Thread-0] WARN   - StorageContainerManager warn log
2017-06-15 00:19:07,135 [Thread-0] ERROR  - StorageContainerManager error 
log
2017-06-15 00:19:07,135 [Thread-0] TRACE 
lang.Object(TestOzoneContainer.java:runTestLogLevel(64)) - Object trace log
2017-06-15 00:19:07,135 [Thread-0] DEBUG 
lang.Object(TestOzoneContainer.java:runTestLogLevel(65)) - Object debug log
2017-06-15 00:19:07,135 [Thread-0] INFO  
lang.Object(TestOzoneContainer.java:runTestLogLevel(66)) - Object info log
2017-06-15 00:19:07,135 [Thread-0] WARN  
lang.Object(TestOzoneContainer.java:runTestLogLevel(67)) - Object warn log
2017-06-15 00:19:07,135 [Thread-0] ERROR 
lang.Object(TestOzoneContainer.java:runTestLogLevel(68)) - Object error log
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org