[jira] [Created] (HDFS-11673) Handle failures of Datanodes with Provided storage

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-11673:
-

 Summary: Handle failures of Datanodes with Provided storage
 Key: HDFS-11673
 URL: https://issues.apache.org/jira/browse/HDFS-11673
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti


Blocks on {{Provided}} storage should become unavailable only if all Datanodes 
that are configured with {{Provided}} storage become unavailable. Even if one 
Datanode with {{Provided}} storage is available, all blocks on the {{Provided}} 
storage should be accessible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11672) Fix some misleading log messages related to EC block groups

[Andrew Wang (JIRA)]

Andrew Wang created HDFS-11672:
--

 Summary: Fix some misleading log messages related to EC block 
groups
 Key: HDFS-11672
 URL: https://issues.apache.org/jira/browse/HDFS-11672
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: erasure-coding
Affects Versions: 3.0.0-alpha1
Reporter: Andrew Wang
Priority: Minor


I turned the NameNode's BlockStateChange and hdfs.StateChange logs up to debug, 
noticed some out of date log messages like the following:

{noformat}
2017-04-18 17:19:50,639 [IPC Server handler 4 on 45133] DEBUG hdfs.StateChange 
(FSDirWriteFileOp.java:addBlock(507)) - DIR* FSDirectory.addBlock: 
/test2RecoveryTasksForSameBlockGroup with blk_-9223372036854775792_1001 block 
is added to the in-memory file system

2017-04-18 17:19:50,836 [Block report processor] DEBUG BlockStateChange 
(LowRedundancyBlocks.java:update(353)) - BLOCK* 
NameSystem.LowRedundancyBlock.update: blk_-9223372036854775792_1001 has only 8 
replicas and needs 9 replicas so is added to neededReconstructions at priority 
level 2

2017-04-18 17:19:51,235 [IPC Server handler 4 on 45133] DEBUG hdfs.StateChange 
(FSNamesystem.java:closeFile(3740)) - closeFile: 
/test2RecoveryTasksForSameBlockGroup with 1 blocks is persisted to the file 
system
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

[Andrew Wang (JIRA)]

Andrew Wang created HDFS-11671:
--

 Summary: 
TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails
 Key: HDFS-11671
 URL: https://issues.apache.org/jira/browse/HDFS-11671
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: erasure-coding, test
Affects Versions: 3.0.0-alpha2
Reporter: Andrew Wang


This failed on a unit test run with 3.0.0-alpha2.

{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hdfs.server.namenode.TestReconstructStripedBlocks.test2RecoveryTasksForSameBlockGroup(TestReconstructStripedBlocks.java:223)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Hadoop s3 integration for Spark

[Steve Loughran]

On 14 Apr 2017, at 17:46, Afshin, Bardia 
> wrote:

Hello community.

I’m considering consuming s3 objects via Hadoop via s3a protocol. The main 
purpose of this is to utilize Spark to access s3, and it seems like the only 
formal protocol / integration for doing so is Hadoop. The process that I am 
implementing is rather formal and straight forward. It will download the 
contents of a s3 objects, remove some columns from the csv file, and PUT the 
object into another bucket on s3. Is there any reason doing a simple GET on the 
object is not as performant if not better than utilizing Hadoop s3a protocol? 
This is the page that I’m getting my reference from 
https://wiki.apache.org/hadoop/AmazonS3

1. use S3A.  
https://www.slideshare.net/steve_l/apache-spark-and-object-stores-for-london-spark-user-group
2. Use Hadoop 2.8 for maximum performance.
3. consider using alternative formats than CSV for your data, as follow-on 
processing suffers. Better use a column format like ORC and make effective use 
of predicate pushdowns.




The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

[jira] [Created] (HDFS-11670) [SPS]: Add storagepolicy command to check the status of storage policy Satisfier

[Surendra Singh Lilhore (JIRA)]

Surendra Singh Lilhore created HDFS-11670:
-

 Summary: [SPS]: Add storagepolicy command to check the status of 
storage policy Satisfier
 Key: HDFS-11670
 URL: https://issues.apache.org/jira/browse/HDFS-11670
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-10285
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore


Its good to have one command to check SPS is enabled or not. Based on this user 
can take the decision to run the Mover.

{noformat}
hdfs storagepolicies -policysatisfierstatus
{noformat} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/292/

[Mar 31, 2017 5:30:35 PM] (epayne) YARN-6354. LeveldbRMStateStore can parse 
invalid keys when recovering
[Mar 31, 2017 7:10:27 PM] (arp) HDFS-11603. Improve slow mirror/disk warnings 
in BlockReceiver.
[Mar 31, 2017 8:50:29 PM] (arp) HDFS-11560. Expose slow disks via NameNode JMX. 
Contributed by Hanisha
[Apr 1, 2017 3:04:26 AM] (liuml07) HADOOP-14267. Make DistCpOptions immutable. 
Contributed by Mingliang Liu
[Apr 1, 2017 11:07:34 PM] (varunsaxena) YARN-6414. ATSv2 HBase related tests 
fail due to guava version upgrade
[Apr 1, 2017 11:24:12 PM] (varunsaxena) YARN-6377. 
NMTimelinePublisher#serviceStop does not stop timeline
[Apr 3, 2017 4:06:54 AM] (aajisaka) MAPREDUCE-6824. 
TaskAttemptImpl#createCommonContainerLaunchContext is
[Apr 3, 2017 2:35:09 PM] (weichiu) HDFS-11515. -du throws 
ConcurrentModificationException. Contributed by
[Apr 3, 2017 2:57:28 PM] (weichiu) HDFS-11598. Improve -setrep for Erasure 
Coded files. Contributed by
[Apr 3, 2017 6:07:14 PM] (liuml07) HADOOP-14268. Fix markdown itemization in 
hadoop-aws documents.
[Apr 4, 2017 3:13:14 AM] (cdouglas) HADOOP-14271. Correct spelling of 
'occurred' and variants. Contributed
[Apr 4, 2017 6:03:59 PM] (liuml07) HADOOP-14272. Azure: WasbRemoteCallHelper 
should use String equals for
[Apr 4, 2017 6:10:28 PM] (liuml07) HADOOP-14273. Azure: NativeAzureFileSystem 
should respect config for
[Apr 4, 2017 8:38:25 PM] (wang) HADOOP-14104. Client should always ask namenode 
for kms provider path.
[Apr 4, 2017 9:39:01 PM] (wangda) YARN-6109. Add an ability to convert 
ChildQueue to ParentQueue. (Xuan
[Apr 5, 2017 12:08:33 AM] (kasha) YARN-6436. 
TestSchedulingPolicy#testParseSchedulingPolicy timeout is too
[Apr 5, 2017 12:56:21 AM] (mingma) YARN-6004. Refactor
[Apr 5, 2017 3:45:28 AM] (aajisaka) HDFS-9651. All web UIs should include a 
robots.txt file. Contributed by
[Apr 5, 2017 3:47:45 AM] (aajisaka) HDFS-9651. Add missing robots.txt files.
[Apr 5, 2017 4:08:35 AM] (varunsaxena) YARN-6420. RM startup failure due to 
wrong order in nodelabel editlog
[Apr 5, 2017 6:05:24 AM] (wang) HDFS-11538. Move ClientProtocol HA proxies into 
hadoop-hdfs-client.
[Apr 5, 2017 8:33:07 AM] (aajisaka) YARN-5007. Remove deprecated constructors 
of MiniYARNCluster and
[Apr 5, 2017 5:30:45 PM] (jlowe) YARN-6403. Invalid local resource request can 
raise NPE and make NM
[Apr 5, 2017 6:28:30 PM] (wangda) YARN-6406. Remove SchedulerRequestKeys when 
no more pending
[Apr 5, 2017 7:50:41 PM] (varunsaxena) YARN-6437. 
TestSignalContainer#testSignalRequestDeliveryToNM fails
[Apr 5, 2017 10:12:03 PM] (wang) HADOOP-14280. Fix compilation of 
TestKafkaMetrics.
[Apr 5, 2017 10:33:08 PM] (arp) HDFS-11628. Clarify the behavior of HDFS Mover 
in documentation.
[Apr 5, 2017 10:43:14 PM] (kasha) YARN-6448. Continuous scheduling thread 
crashes while sorting nodes.
[Apr 5, 2017 11:04:09 PM] (wang) HFDS-11596. hadoop-hdfs-client jar is in the 
wrong directory in release
[Apr 5, 2017 11:07:24 PM] (templedf) YARN-6381. FSAppAttempt has several 
variables that should be final
[Apr 6, 2017 12:25:40 AM] (templedf) MAPREDUCE-6846. Fragments specified for 
libjar paths are not handled
[Apr 6, 2017 12:28:00 AM] (xyao) HDFS-11302. Improve Logging for 
SSLHostnameVerifier. Contributed by Chen
[Apr 6, 2017 12:54:30 AM] (arp) HDFS-11131. 
TestThrottledAsyncChecker#testCancellation is flaky.
[Apr 6, 2017 4:45:22 AM] (rohithsharmaks) YARN-6424. TimelineCollector is not 
stopped when an app finishes in RM.
[Apr 6, 2017 9:24:36 PM] (jlowe) YARN-6288. Exceptions during aggregated log 
writes are mishandled.
[Apr 6, 2017 9:33:16 PM] (xyao) HDFS-11362. StorageDirectory should initialize 
a non-null default
[Apr 6, 2017 11:11:55 PM] (xyao) HDFS-11608. HDFS write crashed with block size 
greater than 2 GB.
[Apr 6, 2017 11:54:43 PM] (mingma) YARN-5797. Add metrics to the node manager 
for cleaning the PUBLIC and
[Apr 6, 2017 11:59:21 PM] (zhz) HADOOP-14276. Add a nanosecond API to 
Time/Timer/FakeTimer. Contributed
[Apr 7, 2017 12:44:47 AM] (rkanter) MAPREDUCE-6201. TestNetworkedJob fails on 
trunk (pbacsko via rkanter)
[Apr 7, 2017 6:12:50 AM] (sunilg) YARN-6258. localBaseAddress for CORS proxy 
configuration is not working
[Apr 7, 2017 3:59:21 PM] (aw) HADOOP-14202. fix jsvc/secure user var 
inconsistencies
[Apr 7, 2017 7:04:44 PM] (jzhuge) HADOOP-14174. Set default ADLS access token 
provider type to
[Apr 7, 2017 8:43:06 PM] (wang) HADOOP-14287. Compiling trunk with -DskipShade 
fails. Contributed by
[Apr 7, 2017 9:28:07 PM] (rkanter) YARN-6368. Decommissioning an NM results in 
a -1 exit code
[Apr 7, 2017 10:46:11 PM] (wang) HDFS-11633. FSImage failover disables all 
erasure coding policies.
[Apr 7, 2017 11:46:28 PM] (wang) HDFS-11623. Move system erasure coding 
policies into hadoop-hdfs-client.
[Apr 8, 2017 12:31:59 AM] (kasha) YARN-6298. Metric preemptCall is not used in 
new preemption. (Yufei Gu
[Apr 8, 2017 

[jira] [Created] (HDFS-11669) [SPS]: Add option in "setStoragePolicy" command to satisfy the policy.

[Surendra Singh Lilhore (JIRA)]

Surendra Singh Lilhore created HDFS-11669:
-

 Summary: [SPS]: Add option in "setStoragePolicy" command to 
satisfy the policy.
 Key: HDFS-11669
 URL: https://issues.apache.org/jira/browse/HDFS-11669
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode, shell
Affects Versions: HDFS-10285
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore


Add one new option {{-satisfypolicy}} in {{setStoragePolicy}} command to 
satisfy the storage policy.

{noformat}
hdfs storagepolicies -setStoragePolicy -path  -policy  
-satisfypolicy
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Jason Lowe]

Thanks for the pointers, Sean!  According to the infrastructure team, 
apparently it was a typo in the protection scheme that allowed the trunk force 
push to go through.  
 
https://issues.apache.org/jira/browse/INFRA-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971643#comment-15971643
   
Jason
 On Monday, April 17, 2017 3:05 PM, Sean Busbey  wrote:
 

 disallowing force pushes to trunk was done back in:

* August 2014: INFRA-8195
* February 2016: INFRA-11136

On Mon, Apr 17, 2017 at 11:18 AM, Jason Lowe
 wrote:
> I found at least one commit that was dropped, MAPREDUCE-6673.  I was able to 
> cherry-pick the original commit hash since it was recorded in the commit 
> email.
> This begs the question of why we're allowing force pushes to trunk.  I 
> thought we asked to have that disabled the last time trunk was accidentally 
> clobbered?
> Jason
>
>
>    On Monday, April 17, 2017 10:18 AM, Arun Suresh  wrote:
>
>
>  Hi
>
> I had the Apr-14 eve version of trunk on my local machine. I've pushed that.
> Don't know if anything was committed over the weekend though.
>
> Cheers
> -Arun
>
> On Mon, Apr 17, 2017 at 7:17 AM, Anu Engineer 
> wrote:
>
>> Hi Allen,
>>
>> https://issues.apache.org/jira/browse/INFRA-13902
>>
>> That happened with ozone branch too. It was an inadvertent force push.
>> Infra has advised us to force push the latest branch if you have it.
>>
>> Thanks
>> Anu
>>
>>
>> On 4/17/17, 7:10 AM, "Allen Wittenauer"  wrote:
>>
>> >Looks like someone reset HEAD back to Mar 31.
>> >
>> >Sent from my iPad
>> >
>> >> On Apr 16, 2017, at 12:08 AM, Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>> >>
>> >> For more details, see https://builds.apache.org/job/
>> hadoop-qbt-trunk-java8-linux-x86/378/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -1 overall
>> >>
>> >>
>> >> The following subsystems voted -1:
>> >>    docker
>> >>
>> >>
>> >> Powered by Apache Yetus 0.5.0-SNAPSHOT  http://yetus.apache.org
>> >>
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>> >-
>> >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>>
>
>
>



-- 
busbey


   

[jira] [Created] (HDFS-11668) Ozone: misc improvements for SCM CLI

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-11668:
--

 Summary: Ozone: misc improvements for SCM CLI
 Key: HDFS-11668
 URL: https://issues.apache.org/jira/browse/HDFS-11668
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Once HDFS-11649 is done, there are some improvements need to be done in order 
to make SCM CLI better to use in a pseudo cluster, this includes

# HDFS-11649 adds java classes for CLIs, we will need to add shell code to 
expose these commands
# Better error messages when missing some key ozone configurations, e.g 
{{ozone.scm.names}}, {{ozone.scm.datanode.id}} ... etc
# Property {{ozone.enabled}} is not honored, don't know why yet
# Better logging. Currently {{DatanodeStateMachine}} prints very limited logs, 
adds some more logs to indicate state transition is necessary.

The ultimate goal of this ticket is to ensure SCM CLI can work nicely on a 
pseudo cluster.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/380/

[Mar 31, 2017 5:30:35 PM] (epayne) YARN-6354. LeveldbRMStateStore can parse 
invalid keys when recovering
[Mar 31, 2017 7:10:27 PM] (arp) HDFS-11603. Improve slow mirror/disk warnings 
in BlockReceiver.
[Mar 31, 2017 8:50:29 PM] (arp) HDFS-11560. Expose slow disks via NameNode JMX. 
Contributed by Hanisha
[Apr 1, 2017 3:04:26 AM] (liuml07) HADOOP-14267. Make DistCpOptions immutable. 
Contributed by Mingliang Liu
[Apr 1, 2017 11:07:34 PM] (varunsaxena) YARN-6414. ATSv2 HBase related tests 
fail due to guava version upgrade
[Apr 1, 2017 11:24:12 PM] (varunsaxena) YARN-6377. 
NMTimelinePublisher#serviceStop does not stop timeline
[Apr 3, 2017 4:06:54 AM] (aajisaka) MAPREDUCE-6824. 
TaskAttemptImpl#createCommonContainerLaunchContext is
[Apr 3, 2017 2:35:09 PM] (weichiu) HDFS-11515. -du throws 
ConcurrentModificationException. Contributed by
[Apr 3, 2017 2:57:28 PM] (weichiu) HDFS-11598. Improve -setrep for Erasure 
Coded files. Contributed by
[Apr 3, 2017 6:07:14 PM] (liuml07) HADOOP-14268. Fix markdown itemization in 
hadoop-aws documents.
[Apr 4, 2017 3:13:14 AM] (cdouglas) HADOOP-14271. Correct spelling of 
'occurred' and variants. Contributed
[Apr 4, 2017 6:03:59 PM] (liuml07) HADOOP-14272. Azure: WasbRemoteCallHelper 
should use String equals for
[Apr 4, 2017 6:10:28 PM] (liuml07) HADOOP-14273. Azure: NativeAzureFileSystem 
should respect config for
[Apr 4, 2017 8:38:25 PM] (wang) HADOOP-14104. Client should always ask namenode 
for kms provider path.
[Apr 4, 2017 9:39:01 PM] (wangda) YARN-6109. Add an ability to convert 
ChildQueue to ParentQueue. (Xuan
[Apr 5, 2017 12:08:33 AM] (kasha) YARN-6436. 
TestSchedulingPolicy#testParseSchedulingPolicy timeout is too
[Apr 5, 2017 12:56:21 AM] (mingma) YARN-6004. Refactor
[Apr 5, 2017 3:45:28 AM] (aajisaka) HDFS-9651. All web UIs should include a 
robots.txt file. Contributed by
[Apr 5, 2017 3:47:45 AM] (aajisaka) HDFS-9651. Add missing robots.txt files.
[Apr 5, 2017 4:08:35 AM] (varunsaxena) YARN-6420. RM startup failure due to 
wrong order in nodelabel editlog
[Apr 5, 2017 6:05:24 AM] (wang) HDFS-11538. Move ClientProtocol HA proxies into 
hadoop-hdfs-client.
[Apr 5, 2017 8:33:07 AM] (aajisaka) YARN-5007. Remove deprecated constructors 
of MiniYARNCluster and
[Apr 5, 2017 5:30:45 PM] (jlowe) YARN-6403. Invalid local resource request can 
raise NPE and make NM
[Apr 5, 2017 6:28:30 PM] (wangda) YARN-6406. Remove SchedulerRequestKeys when 
no more pending
[Apr 5, 2017 7:50:41 PM] (varunsaxena) YARN-6437. 
TestSignalContainer#testSignalRequestDeliveryToNM fails
[Apr 5, 2017 10:12:03 PM] (wang) HADOOP-14280. Fix compilation of 
TestKafkaMetrics.
[Apr 5, 2017 10:33:08 PM] (arp) HDFS-11628. Clarify the behavior of HDFS Mover 
in documentation.
[Apr 5, 2017 10:43:14 PM] (kasha) YARN-6448. Continuous scheduling thread 
crashes while sorting nodes.
[Apr 5, 2017 11:04:09 PM] (wang) HFDS-11596. hadoop-hdfs-client jar is in the 
wrong directory in release
[Apr 5, 2017 11:07:24 PM] (templedf) YARN-6381. FSAppAttempt has several 
variables that should be final
[Apr 6, 2017 12:25:40 AM] (templedf) MAPREDUCE-6846. Fragments specified for 
libjar paths are not handled
[Apr 6, 2017 12:28:00 AM] (xyao) HDFS-11302. Improve Logging for 
SSLHostnameVerifier. Contributed by Chen
[Apr 6, 2017 12:54:30 AM] (arp) HDFS-11131. 
TestThrottledAsyncChecker#testCancellation is flaky.
[Apr 6, 2017 4:45:22 AM] (rohithsharmaks) YARN-6424. TimelineCollector is not 
stopped when an app finishes in RM.
[Apr 6, 2017 9:24:36 PM] (jlowe) YARN-6288. Exceptions during aggregated log 
writes are mishandled.
[Apr 6, 2017 9:33:16 PM] (xyao) HDFS-11362. StorageDirectory should initialize 
a non-null default
[Apr 6, 2017 11:11:55 PM] (xyao) HDFS-11608. HDFS write crashed with block size 
greater than 2 GB.
[Apr 6, 2017 11:54:43 PM] (mingma) YARN-5797. Add metrics to the node manager 
for cleaning the PUBLIC and
[Apr 6, 2017 11:59:21 PM] (zhz) HADOOP-14276. Add a nanosecond API to 
Time/Timer/FakeTimer. Contributed
[Apr 7, 2017 12:44:47 AM] (rkanter) MAPREDUCE-6201. TestNetworkedJob fails on 
trunk (pbacsko via rkanter)
[Apr 7, 2017 6:12:50 AM] (sunilg) YARN-6258. localBaseAddress for CORS proxy 
configuration is not working
[Apr 7, 2017 3:59:21 PM] (aw) HADOOP-14202. fix jsvc/secure user var 
inconsistencies
[Apr 7, 2017 7:04:44 PM] (jzhuge) HADOOP-14174. Set default ADLS access token 
provider type to
[Apr 7, 2017 8:43:06 PM] (wang) HADOOP-14287. Compiling trunk with -DskipShade 
fails. Contributed by
[Apr 7, 2017 9:28:07 PM] (rkanter) YARN-6368. Decommissioning an NM results in 
a -1 exit code
[Apr 7, 2017 10:46:11 PM] (wang) HDFS-11633. FSImage failover disables all 
erasure coding policies.
[Apr 7, 2017 11:46:28 PM] (wang) HDFS-11623. Move system erasure coding 
policies into hadoop-hdfs-client.
[Apr 8, 2017 12:31:59 AM] (kasha) YARN-6298. Metric preemptCall is not used in 
new preemption. (Yufei Gu
[Apr 8, 2017 

[jira] [Created] (HDFS-11667) Block Storage:Handling flushing of incomplete block id buffers during shutdown

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDFS-11667:


 Summary: Block Storage:Handling flushing of incomplete block id 
buffers during shutdown
 Key: HDFS-11667
 URL: https://issues.apache.org/jira/browse/HDFS-11667
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh


Currently, whenever the cache shutdown not, zero blocks are written to 
DirtyLog. This change will ensure that only required number of blocks are 
written to the DirtyLog.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11666) Ozone: Fix compile error due to differential package name

[Yiqun Lin (JIRA)]

Yiqun Lin created HDFS-11666:


 Summary: Ozone: Fix compile error due to differential package name
 Key: HDFS-11666
 URL: https://issues.apache.org/jira/browse/HDFS-11666
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Yiqun Lin
Assignee: Yiqun Lin


There are some classes's package name which doesn't match its actual path. This 
problem will not influence the execution of maven commands but will detected by 
IDE.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

RE: About 2.7.4 Release

[Brahma Reddy Battula]

Hi All

Any update on 2.7.4 ..?  Gentle Remainder!! Let me know anything I can help on 
this..



Regards
Brahma Reddy Battula

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: 08 March 2017 04:22
To: Sangjin Lee
Cc: Marton Elek; Hadoop Common; yarn-...@hadoop.apache.org; Hdfs-dev; 
mapreduce-...@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Our release steps are documented on the wiki:

2.6/2.7:

https://wiki.apache.org/hadoop/HowToReleasePreDSBCR

2.8+:
https://wiki.apache.org/hadoop/HowToRelease

I think given the push toward 2.8 and 3.0, there's less interest in 
streamlining the 2.6 and 2.7 release processes. CHANGES.txt is the biggest 
pain, and that's fixed in 2.8+.

Current pain points for 2.8+ include:

# fixing up JIRA versions and the release notes, though I somewhat addressed 
this with the versions script for 3.x # making and staging an RC and sending 
the vote email still requires a lot of manual steps # publishing the release is 
also quite manual

I think the RC issues can be attacked with enough scripting. Steve had an ant 
file that automated a lot of this for slider. I think it'd be nice to have a 
nightly Jenkins job that builds an RC, since I've spent a day or two for each 
3.x alpha fixing build issues.

Publishing can be attacked via a mix of scripting and revamping the darned 
website. Forrest is pretty bad compared to the newer static site generators out 
there (e.g. need to write XML instead of markdown, it's hard to review a 
staging site because of all the absolute links, hard to customize, did I 
mention XML?), and the look and feel of the site is from the 00s. We don't 
actually have that much site content, so it should be possible to migrate to a 
new system.

On Tue, Mar 7, 2017 at 9:13 AM, Sangjin Lee  wrote:

> I don't think there should be any linkage between releasing 2.8.0 and 
> 2.7.4. If we have a volunteer for releasing 2.7.4, we should go full 
> speed ahead. We still need a volunteer from a PMC member or a 
> committer as some tasks may require certain privileges, but I don't 
> think it precludes working with others to close down the release.
>
> I for one would like to see more frequent releases, and being able to 
> automate release steps more would go a long way.
>
> On Tue, Mar 7, 2017 at 2:16 AM, Marton Elek  wrote:
>
> > Is there any reason to wait for 2.8 with 2.7.4?
> >
> > Unfortunately the previous  thread about release cadence has been 
> > ended without final decision. But if I understood well, there was 
> > more or less
> an
> > agreement about that it would be great to achieve more frequent 
> > releases, if possible (with or without written rules and EOL policy).
> >
> > I personally prefer to be more closer to the scheduling part of the
> > proposal:
> >
> > "A minor release on the latest major line should be every 6 months, 
> > and a maintenance release on a minor release (as there may be 
> > concurrently maintained minor releases) every 2 months".
> >
> > I don't know what is the hardest part of creating new 
> > minor/maintenance releases. But if the problems are technical 
> > (smoketesting, unit tests,
> old
> > release script, anything else) I would be happy to do any task for 
> > new maintenance releases (or more frequent releases).
> >
> > Regards,
> > Marton
> >
> >
> > 
> > From: Akira Ajisaka 
> > Sent: Tuesday, March 07, 2017 7:34 AM
> > To: Brahma Reddy Battula; Hadoop Common; yarn-...@hadoop.apache.org; 
> > Hdfs-dev; mapreduce-...@hadoop.apache.org
> > Subject: Re: About 2.7.4 Release
> >
> > Probably 2.8.0 will be released soon.
> > https://issues.apache.org/jira/browse/HADOOP-13866?
> > focusedCommentId=15898379=com.atlassian.jira.
> > plugin.system.issuetabpanels:comment-tabpanel#comment-15898379
> >
> > I'm thinking 2.7.4 release process starts after 2.8.0 release, so 
> > 2.7.4 will be released in April or May. (hopefully)
> >
> > Thoughts?
> >
> > Regards,
> > Akira
> >
> > On 2017/03/01 21:01, Brahma Reddy Battula wrote:
> > > Hi All
> > >
> > > It has been six months for branch-2.7 release.. is there any near 
> > > plan
> > for 2.7.4..?
> > >
> > >
> > > Thanks
> > > Brahma Reddy Battula
> > >
> > >
> >
> > 
> > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
> >
> > 
> > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>