date:20170522

[jira] [Created] (YARN-6634) [API] Define an API for ResourceManager WebServices

2017-05-22 Thread Subru Krishnan (JIRA)

Subru Krishnan created YARN-6634:


 Summary: [API] Define an API for ResourceManager WebServices
 Key: YARN-6634
 URL: https://issues.apache.org/jira/browse/YARN-6634
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: resourcemanager
Affects Versions: 2.8.0
Reporter: Subru Krishnan
Priority: Critical


The RM exposes few REST queries but there's no clear API interface defined. 
This makes it painful to build either clients or extension components like 
Router (YARN-5412) that expose REST interfaces themselves. This jira proposes 
adding a RM WebServices protocol similar to the one we have for RPC, i.e. 
{{ApplicationClientProtocol}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6633) Backport YARN-4167 to branch 2.7

2017-05-22 Thread Inigo Goiri (JIRA)

Inigo Goiri created YARN-6633:
-

 Summary: Backport YARN-4167 to branch 2.7
 Key: YARN-6633
 URL: https://issues.apache.org/jira/browse/YARN-6633
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Inigo Goiri
Assignee: Inigo Goiri
 Attachments: YARN-4167-branch-2.7.patch





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6632) Backport YARN-3425 to branch 2.7

2017-05-22 Thread Inigo Goiri (JIRA)

Inigo Goiri created YARN-6632:
-

 Summary: Backport YARN-3425 to branch 2.7
 Key: YARN-6632
 URL: https://issues.apache.org/jira/browse/YARN-6632
 Project: Hadoop YARN
  Issue Type: Task
Reporter: Inigo Goiri
Assignee: Inigo Goiri
 Attachments: YARN-3425-branch-2.7.patch





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

2017-05-22 Thread Brahma Reddy Battula

Hi Konstantin Shvachko


how about creating a wiki page for 2.7.4 release status like 2.8 and trunk in 
following link.??


https://cwiki.apache.org/confluence/display/HADOOP



From: Konstantin Shvachko 
Sent: Saturday, May 13, 2017 3:58 AM
To: Akira Ajisaka
Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org
Subject: Re: About 2.7.4 Release

Latest update on the links and filters. Here is the correct link for the
filter:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12340814

Also updated: https://s.apache.org/Dzg4

Had to do some Jira debugging. Sorry for confusion.

Thanks,
--Konstantin

On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko 
wrote:

> Hey Akira,
>
> I didn't have private filters. Most probably Jira caches something.
> Your filter is in the right direction, but for some reason it lists only
> 22 issues, while mine has 29.
> It misses e.g. YARN-5543 
> .
>
> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
> shared it with "everybody", and updated my link to point to that filter. So
> you can use any of the three methods below to get the correct list:
> 1. Go to https://s.apache.org/Dzg4
> 2. Go to the filter via
> https://issues.apache.org/jira/issues?filter=12340814
>or by finding "Hadoop 2.7.4 release blockers" filter in the jira
> 3. On Advanced issues search page paste this:
> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
> AND "Target Version/s" = 2.7.4
>
> Hope this solves the confusion for which issues are included.
> Please LMK if it doesn't, as it is important.
>
> Thanks,
> --Konstantin
>
> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka  wrote:
>
>> Hi Konstantin,
>>
>> Thank you for volunteering as release manager!
>>
>> > Actually the original link works fine: https://s.apache.org/Dzg4
>> I couldn't see the link. Maybe is it private filter?
>>
>> Here is a link I generated: https://s.apache.org/ehKy
>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>
>> Thanks and Regards,
>> Akira
>>
>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>
>>> Hi Brahma Reddy Battula,
>>>
>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>> Your link excludes closed and resolved issues, which needs backporting,
>>> and
>>> which we cannot reopen, as discussed in this thread earlier.
>>>
>>> Looked through the issues you proposed:
>>>
>>> HDFS-9311 
>>> Seems like a new feature. It helps failover to standby node when primary
>>> is
>>> under heavy load, but it introduces new APIs, addresses, config
>>> parameters.
>>> And needs at least one follow up jira.
>>> Looks like a backward compatible change, though.
>>> Did you have a chance to run it in production?
>>>
>>> +1 on
>>> HDFS-10987 
[HDFS-10987] Make Decommission less expensive when lot of 
...
issues.apache.org
When user want to decommission a node which having 50M blocks ,it could hold 
the namesystem lock for long time.We've seen it is taking 36 sec. As we knew 
during this ...



>>> HDFS-9902 
[HDFS-9902] Support different values of dfs.datanode.du 
...
issues.apache.org
Now Hadoop support different storage type for DISK, SSD, ARCHIVE and RAM_DISK, 
but they share one configuration dfs.datanode.du.reserved. The DISK size may be 
several ...



>>> HDFS-8312 
Trash does not descent into child directories to check for 
...
issues.apache.org
HDFS trash does not descent into child directory to check if user has 
permission to delete files. For example: Run the following command to 
initialize directory ...



>>> HADOOP-14100 
Upgrade Jsch jar to latest version to fix vulnerability in 
...
issues.apache.org
Recently there was on vulnerability reported on jsch library. Its fixed in 
latest 0.1.54 version before CVE was made public. 
https://cve.mitre.org/cgi-bin/cvename.cgi ...



>>>
>>> Added them to 2.7.4 release. You should see them via the above link now.
>>> Would be good if you could attach backport patches for some of them?
>>>
>>> Appreciate your help,
>>> --Konstantin
>>>
>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>> brahmareddy.batt...@huawei.com> wrote:
>>>
>>>
 Looks following link is not correct..

 https://s.apache.org/Dzg4

 It should be like following..?

 https://s.apache.org/wi3U


 Apart from Konstantin mentioned,Following

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

2017-05-22 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/322/

[May 21, 2017 12:39:13 PM] (sunilg) YARN-5705. Show timeline data from ATS v2 
in new web UI. Contributed by
[May 22, 2017 4:04:06 AM] (yufei.gu) YARN-6111. Rumen input does't work in SLS. 
Contributed by Yufei Gu.
[May 22, 2017 8:40:06 AM] (sunilg) YARN-6584. Correct license headers in 
hadoop-common, hdfs, yarn and


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-05-22 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/411/

[May 21, 2017 8:23:35 AM] (liuml07) HDFS-11862. Option -v missing for du 
command in FileSystemShell.md.
[May 21, 2017 12:39:13 PM] (sunilg) YARN-5705. Show timeline data from ATS v2 
in new web UI. Contributed by
[May 22, 2017 4:04:06 AM] (yufei.gu) YARN-6111. Rumen input does't work in SLS. 
Contributed by Yufei Gu.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-common-project/hadoop-minikdc 
   Possible null pointer dereference in 
org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value of called 
method Dereferenced at 
MiniKdc.java:org.apache.hadoop.minikdc.MiniKdc.delete(File) due to return value 
of called method Dereferenced at MiniKdc.java:[line 368] 

FindBugs :

   module:hadoop-common-project/hadoop-auth 
   
org.apache.hadoop.security.authentication.server.MultiSchemeAuthenticationHandler.authenticate(HttpServletRequest,
 HttpServletResponse) makes inefficient use of keySet iterator instead of 
entrySet iterator At MultiSchemeAuthenticationHandler.java:of keySet iterator 
instead of entrySet iterator At MultiSchemeAuthenticationHandler.java:[line 
192] 

FindBugs :

   module:hadoop-common-project/hadoop-common 
   org.apache.hadoop.crypto.CipherSuite.setUnknownValue(int) 
unconditionally sets the field unknownValue At CipherSuite.java:unknownValue At 
CipherSuite.java:[line 44] 
   org.apache.hadoop.crypto.CryptoProtocolVersion.setUnknownValue(int) 
unconditionally sets the field unknownValue At 
CryptoProtocolVersion.java:unknownValue At CryptoProtocolVersion.java:[line 67] 
   Possible null pointer dereference in 
org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to return value of 
called method Dereferenced at 
FileUtil.java:org.apache.hadoop.fs.FileUtil.fullyDeleteOnExit(File) due to 
return value of called method Dereferenced at FileUtil.java:[line 118] 
   Possible null pointer dereference in 
org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path, 
File, Path, File) due to return value of called method Dereferenced at 
RawLocalFileSystem.java:org.apache.hadoop.fs.RawLocalFileSystem.handleEmptyDstDirectoryOnWindows(Path,
 File, Path, File) due to return value of called method Dereferenced at 
RawLocalFileSystem.java:[line 387] 
   Return value of org.apache.hadoop.fs.permission.FsAction.or(FsAction) 
ignored, but method has no side effect At FTPFileSystem.java:but method has no 
side effect At FTPFileSystem.java:[line 421] 
   Useless condition:lazyPersist == true at this point At 
CommandWithDestination.java:[line 502] 
   org.apache.hadoop.io.DoubleWritable.compareTo(DoubleWritable) 
incorrectly handles double value At DoubleWritable.java: At 
DoubleWritable.java:[line 78] 
   org.apache.hadoop.io.DoubleWritable$Comparator.compare(byte[], int, int, 
byte[], int, int) incorrectly handles double value At DoubleWritable.java:int) 
incorrectly handles double value At DoubleWritable.java:[line 97] 
   org.apache.hadoop.io.FloatWritable.compareTo(FloatWritable) incorrectly 
handles float value At FloatWritable.java: At FloatWritable.java:[line 71] 
   org.apache.hadoop.io.FloatWritable$Comparator.compare(byte[], int, int, 
byte[], int, int) incorrectly handles float value At FloatWritable.java:int) 
incorrectly handles float value At FloatWritable.java:[line 89] 
   Possible null pointer dereference in 
org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) due to return 
value of called method Dereferenced at 
IOUtils.java:org.apache.hadoop.io.IOUtils.listDirectory(File, FilenameFilter) 
due to return value of called method Dereferenced at IOUtils.java:[line 350] 
   org.apache.hadoop.io.erasurecode.ECSchema.toString() makes inefficient 
use of keySet iterator instead of entrySet iterator At ECSchema.java:keySet 
iterator instead of entrySet iterator At ECSchema.java:[line 193] 
   Possible bad parsing of shift operation in 
org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At 
Utils.java:operation in 
org.apache.hadoop.io.file.tfile.Utils$Version.hashCode() At Utils.java:[line 
398] 
   
org.apache.hadoop.metrics2.lib.DefaultMetricsFactory.setInstance(MutableMetricsFactory)
 unconditionally sets the field mmfImpl At DefaultMetricsFactory.java:mmfImpl 
At DefaultMetricsFactory.java:[line 49] 
   
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.setMiniClusterMode(boolean) 
unconditionally sets the field miniClusterMode At 
DefaultMetricsSystem.java:miniClusterMode At DefaultMetricsSystem.java:[line 
100] 
   Useless object

[jira] [Created] (YARN-6631) Refactor loader.js in new YARN-UI

2017-05-22 Thread Akhil PB (JIRA)

Akhil PB created YARN-6631:
--

 Summary: Refactor loader.js in new YARN-UI
 Key: YARN-6631
 URL: https://issues.apache.org/jira/browse/YARN-6631
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Akhil PB
Assignee: Akhil PB


Current loader.js file overwrites all other ENV properties configured in 
config.env file other than "rmWebAdderss" and "timelineWebAddress". This is 
ticket is meant to fix the above issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6630) Container worker dir could not recover when NM restart

2017-05-22 Thread Yang Wang (JIRA)

Yang Wang created YARN-6630:
---

 Summary: Container worker dir could not recover when NM restart
 Key: YARN-6630
 URL: https://issues.apache.org/jira/browse/YARN-6630
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yang Wang


When ContainerRetryPolicy is NEVER_RETRY, container worker dir will not be 
saved in NM state store. Then NM restarts, container.workDir is null, and may 
cause other exceptions.

{code:title=ContainerLaunch.java}
...
  private void recordContainerWorkDir(ContainerId containerId,
  String workDir) throws IOException{
container.setWorkDir(workDir);
if (container.isRetryContextSet()) {
  context.getNMStateStore().storeContainerWorkDir(containerId, workDir);
}
  }
{code}

{code:title=ContainerImpl.java}
  static class ResourceLocalizedWhileRunningTransition
  extends ContainerTransition {
...
  String linkFile = new Path(container.workDir, link).toString();
...
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6629) NPE occurred when container allocation proposal is applied but its resource requests are removed before

2017-05-22 Thread Tao Yang (JIRA)

Tao Yang created YARN-6629:
--

 Summary: NPE occurred when container allocation proposal is 
applied but its resource requests are removed before
 Key: YARN-6629
 URL: https://issues.apache.org/jira/browse/YARN-6629
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2, 2.9.0
Reporter: Tao Yang
Assignee: Tao Yang


Error log:
{code}
FATAL event.EventDispatcher (EventDispatcher.java:run(75)) - Error in handling 
event type NODE_UPDATE to the Event Dispatcher
java.lang.NullPointerException
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.allocate(AppSchedulingInfo.java:446)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:516)
at 
org.apache.hadoop.yarn.client.TestNegativePendingResource$1.answer(TestNegativePendingResource.java:225)
at 
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:31)
at org.mockito.internal.MockHandler.handle(MockHandler.java:97)
at 
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp$$EnhancerByMockitoWithCGLIB$$29eb8afc.apply()
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2396)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.submitResourceCommitRequest(CapacityScheduler.java:2281)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1247)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1236)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1325)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1112)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:987)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1367)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:143)
at 
org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
at java.lang.Thread.run(Thread.java:745)
{code}

Reproduce this error in chronological order:
1. AM started and requested 1 container with schedulerRequestKey#1 : 
ApplicationMasterService#allocate -->  CapacityScheduler#allocate --> 
SchedulerApplicationAttempt#updateResourceRequests --> 
AppSchedulingInfo#updateResourceRequests 
Added schedulerRequestKey#1 into schedulerKeyToPlacementSets
2. Scheduler allocatd 1 container for this request and accepted the proposal
3. AM removed this request
ApplicationMasterService#allocate -->  CapacityScheduler#allocate --> 
SchedulerApplicationAttempt#updateResourceRequests --> 
AppSchedulingInfo#updateResourceRequests --> 
AppSchedulingInfo#addToPlacementSets --> 
AppSchedulingInfo#updatePendingResources
Removed schedulerRequestKey#1 from schedulerKeyToPlacementSets)
4. Scheduler applied this proposal and wanted to deduct the pending resource
CapacityScheduler#tryCommit --> FiCaSchedulerApp#apply --> 
AppSchedulingInfo#allocate 
Throw NPE when called 
schedulerKeyToPlacementSets.get(schedulerRequestKey).allocate(schedulerKey, 
type, node);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6634) [API] Define an API for ResourceManager WebServices

[jira] [Created] (YARN-6633) Backport YARN-4167 to branch 2.7

[jira] [Created] (YARN-6632) Backport YARN-3425 to branch 2.7

Re: About 2.7.4 Release

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Created] (YARN-6631) Refactor loader.js in new YARN-UI

[jira] [Created] (YARN-6630) Container worker dir could not recover when NM restart

[jira] [Created] (YARN-6629) NPE occurred when container allocation proposal is applied but its resource requests are removed before

9 matches

Site Navigation

Mail list logo

Footer information