[jira] [Commented] (SLIDER-950) Implement GetStatus API call for multiple applications
[ https://issues.apache.org/jira/browse/SLIDER-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993306#comment-14993306 ] thomas liu commented on SLIDER-950: --- Ah, Sounds good. I didn't look into this class SliderYarnClientImpl as it is not in the current code path of 'getStatus'. I'm trying to make use of it now > Implement GetStatus API call for multiple applications > -- > > Key: SLIDER-950 > URL: https://issues.apache.org/jira/browse/SLIDER-950 > Project: Slider > Issue Type: New Feature > Components: client >Affects Versions: Slider 0.80 >Reporter: thomas liu >Assignee: thomas liu > Fix For: Slider 0.90 > > > We found it useful to implement an API to allow getting status for multiple > applications in a cluster. We restrict the applications to a list of names we > pass into the API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Integrating KOYA into the Slider code base
Sure, I will file the JIRAs. Thomas On Thu, Nov 5, 2015 at 2:08 PM, Josh Elser wrote: > Yep, that's on us (so I've just read). > > Want to include that with the other JIRA issue Steve asked? We can figure > out which one of the PPMC will be responsible for it then. > > > Thomas Weise wrote: > >> Thanks, who will be responsible to put up the clearance page, Slider PPMC? >> >> >> On Thu, Nov 5, 2015 at 8:31 AM, Billie Rinaldi >> wrote: >> >> See http://incubator.apache.org/ip-clearance/. >>> >>> On Thu, Nov 5, 2015 at 8:08 AM, Josh Elser wrote: >>> >>> +1 sounds like a great idea! We will likely have some "paperwork" for the code grant (per ASF), but I think that's a very minor headache compared to what integrating KOYA >>> would >>> benefit Slider. Thomas Weise wrote: Dear Slider community, > > As you may know, KOYA (Kafka on YARN) was initiated by DataTorrent > about a >>> year ago with the aim of offering the option to run a Kafka cluster on > the >>> same infrastructure that our YARN native stream processing platform (now > Apache Apex (incubating) - http://apex.incubator.apache.org/) was > built > for. > > The KOYA repository is here: https://github.com/DataTorrent/KOYA > > This is to propose the merge of KOYA into the slider repository. We > believe > the ASF umbrella is the best option to take the project forward and > become >>> a viable option for Kafka deployment. > > Please share your thoughts on how we can take this forward. > > Thanks, > Thomas > > > >>
Re: Integrating KOYA into the Slider code base
Yep, that's on us (so I've just read). Want to include that with the other JIRA issue Steve asked? We can figure out which one of the PPMC will be responsible for it then. Thomas Weise wrote: Thanks, who will be responsible to put up the clearance page, Slider PPMC? On Thu, Nov 5, 2015 at 8:31 AM, Billie Rinaldi wrote: See http://incubator.apache.org/ip-clearance/. On Thu, Nov 5, 2015 at 8:08 AM, Josh Elser wrote: +1 sounds like a great idea! We will likely have some "paperwork" for the code grant (per ASF), but I think that's a very minor headache compared to what integrating KOYA would benefit Slider. Thomas Weise wrote: Dear Slider community, As you may know, KOYA (Kafka on YARN) was initiated by DataTorrent about a year ago with the aim of offering the option to run a Kafka cluster on the same infrastructure that our YARN native stream processing platform (now Apache Apex (incubating) - http://apex.incubator.apache.org/) was built for. The KOYA repository is here: https://github.com/DataTorrent/KOYA This is to propose the merge of KOYA into the slider repository. We believe the ASF umbrella is the best option to take the project forward and become a viable option for Kafka deployment. Please share your thoughts on how we can take this forward. Thanks, Thomas
[jira] [Created] (SLIDER-963) Write mock/unit tests for AA placement
Steve Loughran created SLIDER-963: - Summary: Write mock/unit tests for AA placement Key: SLIDER-963 URL: https://issues.apache.org/jira/browse/SLIDER-963 Project: Slider Issue Type: Sub-task Components: appmaster Affects Versions: Slider 0.90 Reporter: Steve Loughran Assignee: Steve Loughran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (SLIDER-947) build node map from yarn update reports; serve via REST/IPC
[ https://issues.apache.org/jira/browse/SLIDER-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SLIDER-947. --- Resolution: Fixed Fix Version/s: (was: Slider 2.0.0) Slider 0.90 > build node map from yarn update reports; serve via REST/IPC > --- > > Key: SLIDER-947 > URL: https://issues.apache.org/jira/browse/SLIDER-947 > Project: Slider > Issue Type: Sub-task > Components: appmaster, Web & REST >Affects Versions: Slider 0.90 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: Slider 0.90 > > Original Estimate: 2h > Remaining Estimate: 2h > > Before doing any anti-affinity allocation, update the state of the cluster > from the RM events, and make it visible via the IPC and REST APIs. > That way, we and the tests can see what is going on -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Integrating KOYA into the Slider code base
Thanks, who will be responsible to put up the clearance page, Slider PPMC? On Thu, Nov 5, 2015 at 8:31 AM, Billie Rinaldi wrote: > See http://incubator.apache.org/ip-clearance/. > > On Thu, Nov 5, 2015 at 8:08 AM, Josh Elser wrote: > > > +1 sounds like a great idea! > > > > We will likely have some "paperwork" for the code grant (per ASF), but I > > think that's a very minor headache compared to what integrating KOYA > would > > benefit Slider. > > > > > > Thomas Weise wrote: > > > >> Dear Slider community, > >> > >> As you may know, KOYA (Kafka on YARN) was initiated by DataTorrent > about a > >> year ago with the aim of offering the option to run a Kafka cluster on > the > >> same infrastructure that our YARN native stream processing platform (now > >> Apache Apex (incubating) - http://apex.incubator.apache.org/) was built > >> for. > >> > >> The KOYA repository is here: https://github.com/DataTorrent/KOYA > >> > >> This is to propose the merge of KOYA into the slider repository. We > >> believe > >> the ASF umbrella is the best option to take the project forward and > become > >> a viable option for Kafka deployment. > >> > >> Please share your thoughts on how we can take this forward. > >> > >> Thanks, > >> Thomas > >> > >> >
Slider-develop - Build # 729 - Fixed
The Apache Jenkins build system has built Slider-develop (build #729) Status: Fixed Check console output at https://builds.apache.org/job/Slider-develop/729/ to view the results.
Re: Integrating KOYA into the Slider code base
See http://incubator.apache.org/ip-clearance/. On Thu, Nov 5, 2015 at 8:08 AM, Josh Elser wrote: > +1 sounds like a great idea! > > We will likely have some "paperwork" for the code grant (per ASF), but I > think that's a very minor headache compared to what integrating KOYA would > benefit Slider. > > > Thomas Weise wrote: > >> Dear Slider community, >> >> As you may know, KOYA (Kafka on YARN) was initiated by DataTorrent about a >> year ago with the aim of offering the option to run a Kafka cluster on the >> same infrastructure that our YARN native stream processing platform (now >> Apache Apex (incubating) - http://apex.incubator.apache.org/) was built >> for. >> >> The KOYA repository is here: https://github.com/DataTorrent/KOYA >> >> This is to propose the merge of KOYA into the slider repository. We >> believe >> the ASF umbrella is the best option to take the project forward and become >> a viable option for Kafka deployment. >> >> Please share your thoughts on how we can take this forward. >> >> Thanks, >> Thomas >> >>
Re: Integrating KOYA into the Slider code base
+1 sounds like a great idea! We will likely have some "paperwork" for the code grant (per ASF), but I think that's a very minor headache compared to what integrating KOYA would benefit Slider. Thomas Weise wrote: Dear Slider community, As you may know, KOYA (Kafka on YARN) was initiated by DataTorrent about a year ago with the aim of offering the option to run a Kafka cluster on the same infrastructure that our YARN native stream processing platform (now Apache Apex (incubating) - http://apex.incubator.apache.org/) was built for. The KOYA repository is here: https://github.com/DataTorrent/KOYA This is to propose the merge of KOYA into the slider repository. We believe the ASF umbrella is the best option to take the project forward and become a viable option for Kafka deployment. Please share your thoughts on how we can take this forward. Thanks, Thomas
[jira] [Commented] (SLIDER-961) clean up SliderClient code
[ https://issues.apache.org/jira/browse/SLIDER-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991803#comment-14991803 ] ASF subversion and git services commented on SLIDER-961: Commit fed5d03469f87642fce740e56847a0e38e396242 in incubator-slider's branch refs/heads/develop from [~ste...@apache.org] [ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=fed5d03 ] SLIDER-961 fix inverted condition on install operation, plus improve log output in TestAgentClientProvider2 > clean up SliderClient code > -- > > Key: SLIDER-961 > URL: https://issues.apache.org/jira/browse/SLIDER-961 > Project: Slider > Issue Type: Improvement > Components: client >Affects Versions: Slider 0.81 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: Slider 0.90 > > Original Estimate: 1h > Remaining Estimate: 1h > > The slider client code is getting ugly > # too much repetition > # lots of checks & throw clauses making things complex > # stuff in the class which could be pushed elsewhere -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Immediate change to git
I've already hit this BTW: you can't rebase a feature locally and then force push the change out; while you can create a new branch, the old one can't be deleted (branches and tags are pretty similar, after all); so we'll end up filling up the branch list with lots of versions. > On 4 Nov 2015, at 10:30, Steve Loughran wrote: > > FYI. Also stops tags being deleted too. > > > Begin forwarded message: > > From: David Nalley mailto:da...@gnsa.us>> > Date: 3 November 2015 at 20:40:45 GMT > To: David Nalley mailto:da...@gnsa.us>> > Subject: Immediate change to git > > Hi folks, > > After the many emails you may have seen around Git, I am writing yet another. > > To date, on our git repos, we've only 'protected' master, trunk, and > release branches and tags. This has left other branches open to > rewriting, force pushes, and branch deletion. > > Recently, we've discovered that many projects (just under 50) have one > or more repos that are using something other than master or trunk as > their main development branch. In some cases this is a 'develop' > branch in others it's more like $project_version which leaves those > branches open to deletion, rewriting, etc. > > So today, we're taking an interim step of disabling non-fast-forward > pushes and branch deletion across all of our git repos. I emphasize > interim, as it's a stop-gap measure to get us back to the level of > protection we've set expectations for. I know that this will be > disruptive to many folks' way of operating in their git environment, > so we are hoping to make this interim solution short lived. If your > project has immediate needs that you find are blocked by this, please > do reach out to the Infrastructure team, and we will work to make sure > we can help with a timely workaround for those specific cases. > > The longer term solution to this issue may be a policy decision or it > might be a technical solution. I sadly don't know what that solution > will be. We are going to be discussing this on the public > infrastructure-dev mailing list, and I invite you to join us in that > discussion. > > --David > >
Slider-develop - Build # 728 - Still Failing
The Apache Jenkins build system has built Slider-develop (build #728) Status: Still Failing Check console output at https://builds.apache.org/job/Slider-develop/728/ to view the results.
[jira] [Resolved] (SLIDER-962) TestRoleHistoryContainerEvents.testNodeUpdated failing
[
https://issues.apache.org/jira/browse/SLIDER-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved SLIDER-962.
---
Resolution: Fixed
Fix Version/s: Slider 0.90
> TestRoleHistoryContainerEvents.testNodeUpdated failing
> --
>
> Key: SLIDER-962
> URL: https://issues.apache.org/jira/browse/SLIDER-962
> Project: Slider
> Issue Type: Bug
> Components: appmaster
>Affects Versions: Slider 0.90
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.90
>
>
> {code}
> org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated
> Failing for the past 1 build (Since Failed#724 )
> Took 0.14 sec.
> Error Message
> assert startSize - endSize == 1
>| | | |
>5 0 5 false
> Stacktrace
> org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert startSize
> - endSize == 1
>| | | |
>5 0 5 false
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
> at
> org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated(TestRoleHistoryContainerEvents.groovy:421)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-962) TestRoleHistoryContainerEvents.testNodeUpdated failing
[
https://issues.apache.org/jira/browse/SLIDER-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991645#comment-14991645
]
ASF subversion and git services commented on SLIDER-962:
Commit a66f7db839e606a2ae96d51f9f0a5d4d4760bd6e in incubator-slider's branch
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=a66f7db ]
SLIDER-962 TestRoleHistoryContainerEvents.testNodeUpdated failing
> TestRoleHistoryContainerEvents.testNodeUpdated failing
> --
>
> Key: SLIDER-962
> URL: https://issues.apache.org/jira/browse/SLIDER-962
> Project: Slider
> Issue Type: Bug
> Components: appmaster
>Affects Versions: Slider 0.90
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> {code}
> org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated
> Failing for the past 1 build (Since Failed#724 )
> Took 0.14 sec.
> Error Message
> assert startSize - endSize == 1
>| | | |
>5 0 5 false
> Stacktrace
> org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert startSize
> - endSize == 1
>| | | |
>5 0 5 false
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
> at
> org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated(TestRoleHistoryContainerEvents.groovy:421)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SLIDER-962) TestRoleHistoryContainerEvents.testNodeUpdated failing
[
https://issues.apache.org/jira/browse/SLIDER-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991603#comment-14991603
]
Steve Loughran commented on SLIDER-962:
---
SLIDER-947 breaks this, as node updates don't trigger a GC of unused nodes any
more
> TestRoleHistoryContainerEvents.testNodeUpdated failing
> --
>
> Key: SLIDER-962
> URL: https://issues.apache.org/jira/browse/SLIDER-962
> Project: Slider
> Issue Type: Bug
> Components: appmaster
>Affects Versions: Slider 0.90
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> {code}
> org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated
> Failing for the past 1 build (Since Failed#724 )
> Took 0.14 sec.
> Error Message
> assert startSize - endSize == 1
>| | | |
>5 0 5 false
> Stacktrace
> org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert startSize
> - endSize == 1
>| | | |
>5 0 5 false
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
> at
> org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated(TestRoleHistoryContainerEvents.groovy:421)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (SLIDER-962) TestRoleHistoryContainerEvents.testNodeUpdated failing
Steve Loughran created SLIDER-962:
-
Summary: TestRoleHistoryContainerEvents.testNodeUpdated failing
Key: SLIDER-962
URL: https://issues.apache.org/jira/browse/SLIDER-962
Project: Slider
Issue Type: Bug
Components: appmaster
Affects Versions: Slider 0.90
Reporter: Steve Loughran
Assignee: Steve Loughran
{code}
org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated
Failing for the past 1 build (Since Failed#724 )
Took 0.14 sec.
Error Message
assert startSize - endSize == 1
| | | |
5 0 5 false
Stacktrace
org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert startSize -
endSize == 1
| | | |
5 0 5 false
at
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
at
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:648)
at
org.apache.slider.server.appmaster.model.history.TestRoleHistoryContainerEvents.testNodeUpdated(TestRoleHistoryContainerEvents.groovy:421)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (SLIDER-961) clean up SliderClient code
[ https://issues.apache.org/jira/browse/SLIDER-961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SLIDER-961. --- Resolution: Fixed > clean up SliderClient code > -- > > Key: SLIDER-961 > URL: https://issues.apache.org/jira/browse/SLIDER-961 > Project: Slider > Issue Type: Improvement > Components: client >Affects Versions: Slider 0.81 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: Slider 0.90 > > Original Estimate: 1h > Remaining Estimate: 1h > > The slider client code is getting ugly > # too much repetition > # lots of checks & throw clauses making things complex > # stuff in the class which could be pushed elsewhere -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Integrating KOYA into the Slider code base
On 5 Nov 2015, at 04:35, Thomas Weise mailto:t...@apache.org>> wrote: Dear Slider community, As you may know, KOYA (Kafka on YARN) was initiated by DataTorrent about a year ago with the aim of offering the option to run a Kafka cluster on the same infrastructure that our YARN native stream processing platform (now Apache Apex (incubating) - http://apex.incubator.apache.org/) was built for. The KOYA repository is here: https://github.com/DataTorrent/KOYA This is to propose the merge of KOYA into the slider repository. We believe the ASF umbrella is the best option to take the project forward and become a viable option for Kafka deployment. Please share your thoughts on how we can take this forward. Thanks, Thomas I'm pretty excited about this. I would propose 1. create a slider JIRA, assign it to yourself. 2. add a subtask of "add koya to build", which would mean moving koya under slider-packages, and manipulating the toplevel build to include it. 3. add one "add tests of koya". It doesn't have to be too complex, but I'd like to see one to stand up a 1+ node koya instance, with a client finding and talking to Kafka. We could then turn on the chaos-monkey to see what happens -Steve
Re: Slider User Question - APIs to manage application rather than files
> On 4 Nov 2015, at 23:00, Manoj Samel wrote: > > Thanks Steve for the detailed update. > > To make sure I understood you correctly - > > 1. The REST API ** can be used today ** by setting > slider.feature.ws.insecure=true. yes. And once you do that, your code must talk directly to the URL of the Appmaster, not via the proxy. Once the flag is set, anything that can access the HTTP endpoint can manipulate the application, hence "insecure" > This will enable update verbs like PUT and DELETE, not just GET yes > 2. Your recommendation is to use the slider jar as library instead of #1 > yes, because that works in a secure cluster too. The API you can use is designed to support both a two-way IPC connection as well as the REST API (when enabled). > Is my understanding correct? > yes it is. > Thanks, > > On Wed, Nov 4, 2015 at 10:56 AM, Steve Loughran > wrote: > >> >> On 4 Nov 2015, at 01:53, Manoj Samel > manojsamelt...@gmail.com>> wrote: >> >> Thanks Gour, >> >> Is there a plan to provide REST APIs in near future ? >> >> In use cases where configurations need to be built on fly and components be >> added / removed on demand, the file /command line based approach does not >> scales well. >> >> Thanks, >> >> >> That'll be SLIDER-151, "Implement full slider API in REST and switch >> client to it" >> >> All the code for this exists. Except that POST/PUT support in the slider >> AM is disabled for security reasons. >> >> To ensure that only the kerberos-authenticated user can talk to slider, >> all HTTP requests must go through the YARN RM Proxy, which does the check >> and proxies access, also doing some HTML cleanup to prevent malicious >> scripts & things. >> >> We can't go through the proxy; see YARN-2031, YARN-2084. Someone needs to >> fix those bits of YARN. >> >> I've got a patch for a bit of it, but see it has aged. If someone could >> bring that up to sync with trunk we could try to get that in to Hadoop 2.8 >> (more specifically: if people other than I write these patches, I can >> commit it myself). >> >> >> 1. You can enable the REST API by changing the HTTP filters to not >> redirect HTTP requests sent to the ws/ bits of the REST API, which you can >> do with by setting "slider.feature.ws.insecure" to "true" (I've done this >> for testing). >> >> 2. The API is documented at : >> http://slider.incubator.apache.org/docs/api/slider_REST_api_v2.html >> >> 3. The read-side of the REST model works everywhere today; we define the >> various JSON structures in the slider JAR. >> >> Before rushing to turn on a back door the current UK government would be >> proud of, know that we do have is secure API client designed to work with >> both Slider's existing IPC protocol and the REST API. : >> org.apache.slider.api.SliderApplicationApi >> >> >> There's an RPC client ( SliderApplicationIpcClient) and REST client >> (SliderApplicationApiRestClient); the test TestStandaloneREST shows how >> they are interchangeable from a client perspective, they both present the >> same rest model and operations. >> >> If you can put the slider JAR on your classpath, and deal with classpath >> version pain (JAX-RS has proven the troublespot), then you can load >> slider.jar in your app and use it as a library, rather than a command line >> tool. This is how the Apache Ambari integration works. >> >> >>
