[jira] [Created] (MAPREDUCE-6934) downlink.data is written to CWD

[Allen Wittenauer (JIRA)]

Allen Wittenauer created MAPREDUCE-6934:
---

 Summary: downlink.data is written to CWD
 Key: MAPREDUCE-6934
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6934
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Affects Versions: 3.0.0-beta1
Reporter: Allen Wittenauer
Priority: Minor


When using Pipes, the downlink.data stream is written to the current working 
directory.  This is a big of a problem when running MR jobclient tests in 
parallel as the file is written outside of target.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/483/

[Aug 2, 2017 7:03:25 AM] (jianhe) YARN-6872. [Addendum patch] Ensure apps could 
run given NodeLabels are
[Aug 2, 2017 11:48:06 AM] (stevel) HADOOP-14709. Fix checkstyle warnings in 
ContractTestUtils. Contributed
[Aug 2, 2017 3:59:33 PM] (epayne) YARN-6846. Nodemanager can fail to fully 
delete application local
[Aug 2, 2017 4:25:19 PM] (yufei) YARN-6895. [FairScheduler] Preemption 
reservation may cause regular
[Aug 2, 2017 5:53:22 PM] (epayne) YARN-5349.
[Aug 2, 2017 6:25:05 PM] (mackrorysd) HADOOP-13595. Rework hadoop_usage to be 
broken up by
[Aug 2, 2017 7:12:48 PM] (cdouglas) HDFS-6984. Serialize FileStatus via 
protobuf.
[Aug 2, 2017 9:22:46 PM] (manojpec) HDFS-9388. Decommission related code to 
support Maintenance State for
[Aug 3, 2017 1:57:10 PM] (sunilg) YARN-6678. Handle IllegalStateException in 
Async Scheduling mode of
[Aug 3, 2017 4:52:35 PM] (haibochen) YARN-6674 Add memory cgroup settings for 
opportunistic containers. 
[Aug 3, 2017 4:56:51 PM] (haibochen) YARN-6673 Add cpu cgroup configurations 
for opportunistic containers.
[Aug 3, 2017 6:33:37 PM] (yufei) YARN-6832. Tests use 
assertTrue(equals(...)) instead of
[Aug 3, 2017 6:44:34 PM] (yufei) MAPREDUCE-6914. Tests use 
assertTrue(equals(...)) instead of
[Aug 3, 2017 9:18:03 PM] (subru) YARN-6932. Fix 
TestFederationRMFailoverProxyProvider test case failure.
[Aug 3, 2017 10:44:51 PM] (wang) HDFS-12131. Add some of the FSNamesystem JMX 
values as metrics.
[Aug 4, 2017 4:15:40 AM] (Arun Suresh) YARN-5977. ContainerManagementProtocol 
changes to support change of
[Aug 4, 2017 5:35:57 AM] (aajisaka) HADOOP-14706. Adding a helper method to 
determine whether a log is Log4j
[Aug 4, 2017 10:09:08 AM] (stevel) HADOOP-14126. Remove jackson, joda and other 
transient aws SDK




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs-client 
   Possible exposure of partially initialized object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:[line 2906] 
   org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) 
makes inefficient use of keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:[line 105] 

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to 
return value of called method Dereferenced at 
JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus()
 due to return value of called method Dereferenced at JournalNode.java:[line 
302] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String)
 unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId 
At HdfsServerConstants.java:[line 193] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int)
 unconditionally sets the field force At HdfsServerConstants.java:force At 
HdfsServerConstants.java:[line 217] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean)
 unconditionally sets the field isForceFormat At 
HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean)
 unconditionally sets the field isInteractiveFormat At 
HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 
237] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, 
int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at 
DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File,
 File, int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at DataStorage.java:[line 1339] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:[line 258] 
   Useless 

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

+1 (binding)

Looked through the src tarball. Checksum and signature match, skimmed
NOTICE/LICENSE, ran some unit tests. -C

On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko
 wrote:
> Hi everybody,
>
> Here is the next release of Apache Hadoop 2.7 line. The previous stable
> release 2.7.3 was available since 25 August, 2016.
> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> critical bug fixes and major optimizations. See more details in Release
> Note:
> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 08/04/2017.
>
> Please note that my up to date public key are available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Please don't forget to refresh the page if you've been there recently.
> There are other place on Apache sites, which may contain my outdated key.
>
> Thanks,
> --Konstantin

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Konstantin Shvachko]

Hi Wei-Chi,

There is bunch of performance jiras in Hadoop.
This one is not marked as blocker and seems to be hard to reproduce.
And I see that you are still discussing the problem and the solution with
Daryn in the jira.
I don't think it should block the release.
If this or anything else will prove to be a serious problem we can follow
up with a maintenance release containing hotfixes only.

Thanks,
--Konst

On Fri, Aug 4, 2017 at 9:24 AM, Wei-Chiu Chuang 
wrote:

> Hi,
> I'm sorry coming to this vote late.
> Daryn mentioned in HDFS-12136
>  that HDFS-11160
>  has a performance
> regression at DataNode due to the way it locks dataset lock.
>
> HDFS-11160 is in Hadoop 2.7.4. Would it be critical enough to warrant a
> stop for release?
>
> I myself can't reproduce the performance regression (assuming it only
> occurs under extreme workload). Would Daryn or other Yahoo folks comment?
>
> On Thu, Aug 3, 2017 at 10:31 PM, Akira Ajisaka 
> wrote:
>
>> +1 (binding)
>>
>> - Verified the checksum and the signature of the source tarball
>> - Built from source with CentOS 7.2 and OpenJDK 1.8.0_141
>> - Built Hive 2.1.0/2.3.0 and Tez 0.8.5/0.9.0 with Hadoop 2.7.4 artifacts
>> - Built single node cluster and ran some Hive on Tez queries successfully
>>
>> Regards,
>> Akira
>>
>>
>> On 2017/08/04 0:25, Kuhu Shukla wrote:
>>
>>> +1 (non-binding)
>>>
>>> 1. Verified signatures and digests.
>>> 2. Built source.
>>> 3. Installed on a pseudo-distributed cluster.
>>> 4. Ran sample MR jobs and Tez example jobs like orderedwordcount
>>> successfully.
>>>
>>> Thank you Konstantin and others for this release.
>>>
>>> Regards,
>>> Kuhu
>>>
>>>
>>>
>>> On Thursday, August 3, 2017, 7:19:07 AM CDT, Sunil G 
>>> wrote:
>>>
>>>
>>> Thanks Konstantin
>>>
>>> +1 (binding)
>>>
>>> 1. Build tar ball from source package
>>> 2. Ran basic MR jobs and verified UI.
>>> 3. Enabled node labels and ran sleep job. Works fine.
>>> 4. Verified CLI commands related to node labels and its working fine.
>>> 5. RM WorkPreserving restart cases are also verified, and looks fine
>>>
>>> Thanks
>>> Sunil
>>>
>>>
>>>
>>> On Sun, Jul 30, 2017 at 4:59 AM Konstantin Shvachko <
>>> shv.had...@gmail.com>
>>> wrote:
>>>
>>> Hi everybody,

 Here is the next release of Apache Hadoop 2.7 line. The previous stable
 release 2.7.3 was available since 25 August, 2016.
 Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
 critical bug fixes and major optimizations. See more details in Release
 Note:
 http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html

 The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

 Please give it a try and vote on this thread. The vote will run for 5
 days
 ending 08/04/2017.

 Please note that my up to date public key are available from:
 https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
 Please don't forget to refresh the page if you've been there recently.
 There are other place on Apache sites, which may contain my outdated
 key.

 Thanks,
 --Konstantin


>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>
>
>
> --
> A very happy Clouderan
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Wei-Chiu Chuang]

Hi,
I'm sorry coming to this vote late.
Daryn mentioned in HDFS-12136
 that HDFS-11160
 has a performance
regression at DataNode due to the way it locks dataset lock.

HDFS-11160 is in Hadoop 2.7.4. Would it be critical enough to warrant a
stop for release?

I myself can't reproduce the performance regression (assuming it only
occurs under extreme workload). Would Daryn or other Yahoo folks comment?

On Thu, Aug 3, 2017 at 10:31 PM, Akira Ajisaka  wrote:

> +1 (binding)
>
> - Verified the checksum and the signature of the source tarball
> - Built from source with CentOS 7.2 and OpenJDK 1.8.0_141
> - Built Hive 2.1.0/2.3.0 and Tez 0.8.5/0.9.0 with Hadoop 2.7.4 artifacts
> - Built single node cluster and ran some Hive on Tez queries successfully
>
> Regards,
> Akira
>
>
> On 2017/08/04 0:25, Kuhu Shukla wrote:
>
>> +1 (non-binding)
>>
>> 1. Verified signatures and digests.
>> 2. Built source.
>> 3. Installed on a pseudo-distributed cluster.
>> 4. Ran sample MR jobs and Tez example jobs like orderedwordcount
>> successfully.
>>
>> Thank you Konstantin and others for this release.
>>
>> Regards,
>> Kuhu
>>
>>
>>
>> On Thursday, August 3, 2017, 7:19:07 AM CDT, Sunil G 
>> wrote:
>>
>>
>> Thanks Konstantin
>>
>> +1 (binding)
>>
>> 1. Build tar ball from source package
>> 2. Ran basic MR jobs and verified UI.
>> 3. Enabled node labels and ran sleep job. Works fine.
>> 4. Verified CLI commands related to node labels and its working fine.
>> 5. RM WorkPreserving restart cases are also verified, and looks fine
>>
>> Thanks
>> Sunil
>>
>>
>>
>> On Sun, Jul 30, 2017 at 4:59 AM Konstantin Shvachko > >
>> wrote:
>>
>> Hi everybody,
>>>
>>> Here is the next release of Apache Hadoop 2.7 line. The previous stable
>>> release 2.7.3 was available since 25 August, 2016.
>>> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
>>> critical bug fixes and major optimizations. See more details in Release
>>> Note:
>>> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>>>
>>> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>>>
>>> Please give it a try and vote on this thread. The vote will run for 5
>>> days
>>> ending 08/04/2017.
>>>
>>> Please note that my up to date public key are available from:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> Please don't forget to refresh the page if you've been there recently.
>>> There are other place on Apache sites, which may contain my outdated key.
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>


-- 
A very happy Clouderan

[jira] [Resolved] (MAPREDUCE-6933) Invalid event: TA_CONTAINER_LAUNCH_FAILED at KILLED

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved MAPREDUCE-6933.
---
Resolution: Duplicate

> Invalid event: TA_CONTAINER_LAUNCH_FAILED at KILLED
> ---
>
> Key: MAPREDUCE-6933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6933
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.1, 3.0.0-alpha4
>Reporter: lujie
>
> When I run a job on 0.23.1, I found a InvalidStateTransitonException:
> {code:java}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
> TA_CONTAINER_LAUNCH_FAILED at KILLED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:926)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:870)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:862)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> After I manually analyse the code of 3.0.0,I think this error may still 
> exists.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6933) Invalid event: TA_CONTAINER_LAUNCH_FAILED at KILLED

[lujie (JIRA)]

lujie created MAPREDUCE-6933:


 Summary: Invalid event: TA_CONTAINER_LAUNCH_FAILED at KILLED
 Key: MAPREDUCE-6933
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6933
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 3.0.0-alpha4, 0.23.1
Reporter: lujie


When I run a job on 0.23.1, I found a InvalidStateTransitonException:

{code:java}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
TA_CONTAINER_LAUNCH_FAILED at KILLED
at 
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:926)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:870)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:862)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
at java.lang.Thread.run(Thread.java:745)
{code}
After I manually analyse the code of 3.0.0,I think this error may still exists.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org