Re: Is it able to specify the percentage of memory instead of # of MB
That is not possible. The number provided (# of MB) is used to ask for container from YARN and YARN only accepts numbers. % of total on each node also becomes an issue when nodes can have varying value of total memory available. This is common for heterogeneous clusters where hosts may have different value for available memory and core, etc. On Mon, Dec 1, 2014 at 1:39 PM, hsy...@gmail.com wrote: > Hi guys, > > For each component instance, Is there a way to specify the % of total > memory of each node instead of # of MB? > > Best, > Siyuan > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (SLIDER-671) Publish slider artifacts to maven
[ https://issues.apache.org/jira/browse/SLIDER-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230557#comment-14230557 ] Srimanth Gunturi commented on SLIDER-671: - [~ste...@apache.org], if you could please provide how the dependency looks like (group, artifact, version) for other projects. Thanks. > Publish slider artifacts to maven > - > > Key: SLIDER-671 > URL: https://issues.apache.org/jira/browse/SLIDER-671 > Project: Slider > Issue Type: Improvement > Components: build >Affects Versions: Slider 0.60 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: Slider 0.60 > > > publish the slider 0.60.0 artifacts to maven. This may include changing the > pom and building; no other changes to the 0.60.0 line will take place so that > the source release will be that voted on -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-633) Slider should support invocation via Oozie
[
https://issues.apache.org/jira/browse/SLIDER-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230525#comment-14230525
]
ASF subversion and git services commented on SLIDER-633:
Commit b0031870a625ce78528b84b340b253f690ecef40 in incubator-slider's branch
refs/heads/releases/slider-0.50 from [~jmaron]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=b003187 ]
SLIDER-633 add processing for token file for oozie based slider invocations
> Slider should support invocation via Oozie
> --
>
> Key: SLIDER-633
> URL: https://issues.apache.org/jira/browse/SLIDER-633
> Project: Slider
> Issue Type: Improvement
>Affects Versions: Slider 0.50
>Reporter: Lee Yang
> Attachments: fix_oozie_launch.patch, slider_oozie.patch
>
>
> In a secure Hadoop installation, when attempting to launch a slider
> application via an Oozie shell-action, I see the following exception:
> {noformat}
> Stdoutput org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> Delegation Token can be issued only with kerberos or web authentication
> Stdoutput at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6757)
> Stdoutput at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:499)
> Stdoutput at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:921)
> Stdoutput at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> Stdoutput at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> Stdoutput at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> Stdoutput at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> Stdoutput at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> Stdoutput at java.security.AccessController.doPrivileged(Native Method)
> Stdoutput at javax.security.auth.Subject.doAs(Subject.java:415)
> Stdoutput at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
> Stdoutput at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> Stdoutput
> Stdoutput at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> Stdoutput at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> Stdoutput at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> Stdoutput at com.sun.proxy.$Proxy17.getDelegationToken(Unknown Source)
> Stdoutput at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:864)
> Stdoutput at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Stdoutput at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> Stdoutput at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Stdoutput at java.lang.reflect.Method.invoke(Method.java:601)
> Stdoutput at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> Stdoutput at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> Stdoutput at com.sun.proxy.$Proxy18.getDelegationToken(Unknown Source)
> Stdoutput at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:947)
> Stdoutput at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1305)
> Stdoutput at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:527)
> Stdoutput at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:505)
> Stdoutput at
> org.apache.slider.core.launch.AppMasterLauncher.addSecurityTokens(AppMasterLauncher.java:209)
> Stdoutput at
> org.apache.slider.core.launch.AppMasterLauncher.completeAppMasterLaunch(AppMasterLauncher.java:183)
> Stdoutput at
> org.apache.slider.core.launch.AppMasterLauncher.submitApplication(AppMasterLauncher.java:214)
> Stdoutput at
> org.apache.slider.client.SliderClient.launchApplication(SliderClient.java:1127)
> Stdoutput at
> org.apache.slider.client.SliderClient.startCluster(SliderClient.java:771)
> Stdoutput at
> org.apache.slider.client.SliderClient.actionCreate(SliderClient.java:515)
> Stdoutput at
> org.apache.slider.client.SliderClient.runService(SliderClient.java:295)
> Stdoutput at
> org.apache.slider.core.main.ServiceLauncher.launchService(ServiceLauncher.java:186)
> Stdoutput at
> org.apache.slider.core.
Is it able to specify the percentage of memory instead of # of MB
Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan
[jira] [Resolved] (SLIDER-677) Keytab less Kerberos auth support required for Hadoop 2.6
[ https://issues.apache.org/jira/browse/SLIDER-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Maron resolved SLIDER-677. --- Resolution: Fixed > Keytab less Kerberos auth support required for Hadoop 2.6 > - > > Key: SLIDER-677 > URL: https://issues.apache.org/jira/browse/SLIDER-677 > Project: Slider > Issue Type: Improvement > Components: security >Affects Versions: Slider 0.60 >Reporter: Andy Feng > Fix For: Slider 0.70 > > > We have use cases that Slider app will run for a few hours or less. For such > use cases, users should be able to launch Slider apps with "kinit" without > keytab files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-677) Keytab less Kerberos auth support required for Hadoop 2.6
[ https://issues.apache.org/jira/browse/SLIDER-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230400#comment-14230400 ] ASF subversion and git services commented on SLIDER-677: Commit 1bdf80cdf0dcc085e9195bfe80b2ebed35f1b2f9 in incubator-slider's branch refs/heads/develop from [~jmaron] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=1bdf80c ] SLIDER-677 enable operation without keytabs for short lived applications > Keytab less Kerberos auth support required for Hadoop 2.6 > - > > Key: SLIDER-677 > URL: https://issues.apache.org/jira/browse/SLIDER-677 > Project: Slider > Issue Type: Improvement > Components: security >Affects Versions: Slider 0.60 >Reporter: Andy Feng > Fix For: Slider 0.70 > > > We have use cases that Slider app will run for a few hours or less. For such > use cases, users should be able to launch Slider apps with "kinit" without > keytab files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-677) Keytab less Kerberos auth support required for Hadoop 2.6
[ https://issues.apache.org/jira/browse/SLIDER-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230401#comment-14230401 ] ASF subversion and git services commented on SLIDER-677: Commit fd0e7d36a7b23bbe484c206cc64e5756d09ab2d1 in incubator-slider's branch refs/heads/develop from [~jmaron] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=fd0e7d3 ] Merge branch 'feature/SLIDER-677_enable_keytabless_operation' into develop > Keytab less Kerberos auth support required for Hadoop 2.6 > - > > Key: SLIDER-677 > URL: https://issues.apache.org/jira/browse/SLIDER-677 > Project: Slider > Issue Type: Improvement > Components: security >Affects Versions: Slider 0.60 >Reporter: Andy Feng > Fix For: Slider 0.70 > > > We have use cases that Slider app will run for a few hours or less. For such > use cases, users should be able to launch Slider apps with "kinit" without > keytab files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-690) Add appConfig template for HBase Phoenix deployment
[ https://issues.apache.org/jira/browse/SLIDER-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved SLIDER-690. --- Resolution: Fixed > Add appConfig template for HBase Phoenix deployment > --- > > Key: SLIDER-690 > URL: https://issues.apache.org/jira/browse/SLIDER-690 > Project: Slider > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > > Phoenix is gaining popularity. > This JIRA provides template appConfig.xml which makes HBase + Phoenix > deployment functional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-690) Add appConfig template for HBase Phoenix deployment
[ https://issues.apache.org/jira/browse/SLIDER-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230087#comment-14230087 ] ASF subversion and git services commented on SLIDER-690: Commit 94970e9337bf8ae18c346ac8843736dc49c82ecd in incubator-slider's branch refs/heads/develop from [~tedyu] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=94970e9 ] SLIDER-690 Add appConfig template for HBase Phoenix deployment > Add appConfig template for HBase Phoenix deployment > --- > > Key: SLIDER-690 > URL: https://issues.apache.org/jira/browse/SLIDER-690 > Project: Slider > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > > Phoenix is gaining popularity. > This JIRA provides template appConfig.xml which makes HBase + Phoenix > deployment functional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLIDER-690) Add appConfig template for HBase Phoenix deployment
Ted Yu created SLIDER-690: - Summary: Add appConfig template for HBase Phoenix deployment Key: SLIDER-690 URL: https://issues.apache.org/jira/browse/SLIDER-690 Project: Slider Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Phoenix is gaining popularity. This JIRA provides template appConfig.xml which makes HBase + Phoenix deployment functional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLIDER-689) AgentWebPagesIT fails to run against a kerberos cluster
[
https://issues.apache.org/jira/browse/SLIDER-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14229881#comment-14229881
]
Steve Loughran commented on SLIDER-689:
---
...note that there is no YARN REST API client in the hadoop source tree other
than the tests, they all appear to use Jersey to talk to the insecure mini yarn
cluster
> AgentWebPagesIT fails to run against a kerberos cluster
> ---
>
> Key: SLIDER-689
> URL: https://issues.apache.org/jira/browse/SLIDER-689
> Project: Slider
> Issue Type: Bug
> Components: appmaster, Web & REST
>Affects Versions: Slider 0.70
>Reporter: Jonathan Maron
>Assignee: Steve Loughran
>
> The following failure is encountered when running the test against a windows
> secure cluster:
> 2014-11-26 17:13:27,234 [JUnit] DEBUG framework.SliderShell (?:call(?)) -
> export
> HADOOP_CONF_DIR=/Users/jmaron/dev/workspaces/slider-asf/incubator-slider/src/test/clusters/windows;
> export
> SLIDER_CONF_DIR=/Users/jmaron/dev/workspaces/slider-asf/incubator-slider/src/test/clusters/windows/slider;
> /Users/jmaron/dev/workspaces/slider-asf/incubator-slider/slider-assembly/target/slider-0.61-incubating-all/slider-0.61-incubating/bin/slider
> lookup --id application_1417013513278_0023 --out
> /Users/jmaron/dev/workspaces/slider-asf/incubator-slider/slider-funtest/target/launch6631360125544500978.json
> 2014-11-26 17:13:47,389 [JUnit] INFO test.SliderTestUtils
> (SliderTestUtils.groovy:fetchWebPageWithoutError(477)) - Fetching HTTP
> content at
> http://hdp01-head.hdp01.local:8088/proxy/application_1417013513278_0023/
> 2014-11-26 17:13:47,458 [JUnit] WARN httpclient.HttpMethodBase
> (HttpMethodBase.java:processCookieHeaders(1634)) - Invalid cookie header:
> "hadoop.auth=; Path=/; Domain=; Expires=Thu, 01-Jan-1970 00:00:00 GMT;
> HttpOnly". Blank value for domain attribute
> 2014-11-26 17:13:47,465 [JUnit] WARN httpclient.HttpMethodDirector
> (HttpMethodDirector.java:processWWWAuthChallenge(697)) - Unable to respond to
> any of these challenges: {negotiate=Negotiate}
> 2014-11-26 17:13:47,467 [JUnit] ERROR test.SliderTestUtils
> (SliderTestUtils.groovy:fetchWebPageWithoutError(489)) - Request to
> http://hdp01-head.hdp01.local:8088/proxy/application_1417013513278_0023/
> failed with exit code 401, body length 1430:
>
>
>
> Error 401 Authentication required
>
> HTTP ERROR 401
> Problem accessing /proxy/application_1417013513278_0023/. Reason:
> Authentication requiredPowered by
> Jetty://
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> It appears that, in a secure cluster, the
> SliderTestUtils.fetchWebPageWithoutError method may need to add support for
> SPNEGO/Kerberos
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
