[jira] [Commented] (PIO-26) Integrate Apache RAT for license checking
[ https://issues.apache.org/jira/browse/PIO-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500297#comment-15500297 ] ASF GitHub Bot commented on PIO-26: --- Github user jj870la commented on the issue: https://github.com/apache/incubator-predictionio/pull/294 ;#298 > Integrate Apache RAT for license checking > - > > Key: PIO-26 > URL: https://issues.apache.org/jira/browse/PIO-26 > Project: PredictionIO > Issue Type: New Feature >Reporter: Chan > Fix For: 0.10.0 > > > http://creadur.apache.org/rat/ > Use this for Apache license checking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIO-26) Integrate Apache RAT for license checking
[ https://issues.apache.org/jira/browse/PIO-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500291#comment-15500291 ] ASF GitHub Bot commented on PIO-26: --- Github user jj870la commented on the issue: https://github.com/apache/incubator-predictionio/pull/294 / H2O1=COD5? + <3 commits/requestP !//EXE//! > Integrate Apache RAT for license checking > - > > Key: PIO-26 > URL: https://issues.apache.org/jira/browse/PIO-26 > Project: PredictionIO > Issue Type: New Feature >Reporter: Chan > Fix For: 0.10.0 > > > http://creadur.apache.org/rat/ > Use this for Apache license checking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIO-26) Integrate Apache RAT for license checking
[ https://issues.apache.org/jira/browse/PIO-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500281#comment-15500281 ] ASF GitHub Bot commented on PIO-26: --- Github user jj870la commented on the issue: https://github.com/apache/incubator-predictionio/pull/294 !//EXE!.:Jet/boat/ski//H2O1 > Integrate Apache RAT for license checking > - > > Key: PIO-26 > URL: https://issues.apache.org/jira/browse/PIO-26 > Project: PredictionIO > Issue Type: New Feature >Reporter: Chan > Fix For: 0.10.0 > > > http://creadur.apache.org/rat/ > Use this for Apache license checking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Remove engine registration
Hello folks, Great to hear about this possibility. I've been working on running PredictionIO on Heroku https://www.heroku.com Heroku's 12-factor architecture https://12factor.net prefers "stateless builds" to ensure that compiled artifacts result in processes which may be cheaply restarted, replaced, and scaled via process count & size. I imagine this stateless property would be valuable for others as well. The fact that `pio build` inserts stateful metadata into a database causes ripples throughout the lifecycle of PIO engines on Heroku: * An engine cannot be built for production without the production database available. When a production database contains PII (personally identifiable information) which has security compliance requirements, the build system may not be privileged to access that PII data. This also affects CI (continuous integration/testing), where engines would need to be rebuilt in production, defeating assurances CI is supposed to provide. * The build artifacts cannot be reliably reused. "Slugs" at Heroku are intended to be stateless, so that you can rollback to a previous version during the lifetime of an app. With `pio build` causing database side-effects, there's a greater-than-zero probability of slug-to-metadata inconsistencies eventually surfacing in a long-running system. From my user-perspective, a few changes to the CLI would fix it: 1. add a "skip registration" option, `pio build --without-engine-registration` 2. a new command `pio app register` that could be run separately in the built engine (before training) Alas, I do not know PredictionIO internals, so I can only offer a suggestion for how this might be solved. Donald, one specific note, Regarding "No automatic version matching of PIO binary distribution and artifacts version used in the engine template": The Heroku slug contains the PredictionIO binary distribution used to build the engine, so there's never a version matching issue. I guess some systems might deploy only the engine artifacts to production where a pre-existing PIO binary is available, but that seems like a risky practice for long-running systems. Thanks for listening, *Mars Hall Customer Facing Architect Salesforce App Cloud / Heroku San Francisco, California > On Sep 16, 2016, at 10:42, Donald Szeto wrote: > > Hi all, > > I want to start the discussion of removing engine registration. How many > people actually take advantage of being able to run pio commands everywhere > outside of an engine template directory? This will be a nontrivial change on > the operational side so I want to gauge the potential impact to existing > users. > > Pros: > - Stateless build. This would work well with many PaaS. > - Eliminate the "pio build" command once and for all. > - Ability to use your own build system, i.e. Maven, Ant, Gradle, etc. > - Potentially better experience with IDE since engine templates no longer > depends on an SBT plugin. > > Cons: > - Inability to run pio engine training and deployment commands outside of > engine template directory. > - No automatic version matching of PIO binary distribution and artifacts > version used in the engine template. > - A less unified user experience: from pio-build-train-deploy to build, then > pio-train-deploy. > > Regards, > Donald
Re: [VOTE]: Apache PredictionIO (incubating) 0.10.0 Release
Hate to make more work for you but the new name would also maybe avoid some confusion for older migrating users. On Sep 16, 2016, at 9:02 AM, Donald Szeto wrote: If everyone agrees that the artifacts should have an "apache-" prefix, I will roll an RC2 shortly. Since the namespace change is unavoidable, might as well get all artifact name changes done in one shot for good. On Friday, September 16, 2016, Andrew Purtell wrote: > Let me double check. > >> On Sep 16, 2016, at 7:33 AM, Alex Merritt > wrote: >> >> I believe it depends on which of the two votes you mean. For the podling >> vote, PPMC votes are binding, for the incubator vote, IPMC votes are, > no? >> >>> On Sep 15, 2016 9:42 PM, "Andrew Purtell" > wrote: >>> >>> I believe 'binding' only applies to IPMC. >>> > On Sep 15, 2016, at 12:49 PM, Suneel Marthi > wrote: Folks, When u vote please specify "+1 Binding" if u r a PMC member. Its only the PMC votes that count for a release to pass. > On Thu, Sep 15, 2016 at 2:11 PM, Robert Lu > >>> wrote: > > +1 > >> On Sep 15, 2016, at 01:13, Matthew Tovbin > wrote: >> >> +1 >> >>> On Wed, Sep 14, 2016 at 10:12 AM, Pat Ferrel > >> wrote: >> >>> +1 >>> >>> >>> On Sep 13, 2016, at 11:55 AM, Donald Szeto > > wrote: >>> >>> This is the vote for 0.10.0 of Apache PredictionIO (incubating). >>> >>> The vote will run for at least 72 hours and will close on Sept 16th, > 2016. >>> >>> The artifacts can be downloaded here: >>> https://dist.apache.org/repos/dist/dev/incubator/predictioni >>> o/0.10.0-incubating-rc1/ >>> or >>> from the Maven staging repo here: >>> https://repository.apache.org/content/repositories/orgapache >>> predictionio-1001/ >>> >>> All JIRAs completed for this release are tagged with 'FixVersion = > 0.10.0'. >>> You can view them here: >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje >>> ctId=12320420&version=12337844 >>> >>> The artifacts have been signed with Key : 8BF4ABEB >>> >>> Please vote accordingly: >>> >>> [ ] +1, accept RC as the official 0.10.0 release >>> [ ] -1, do not accept RC as the official 0.10.0 release because... >>> >
Re: Remove engine registration
Yes, a new thing though it might serve some of the same purposes. The idea is to only use engine instance information from the metadata store so the template commands will work from anywhere and mostly in any order. On a cluster machine if the engine instance data is in the metastore and the binary exists wherever it was on the machine registered, then `pio deploy ` would work without any other part of the workflow. Also `pio train ` would work from any cluster machine with no need of special folder layout or manifest.json Sorry to overload the term but though this new type of engine instance would have much of the same info, it would also have to contain the path to the binary and maybe other things. On Sep 16, 2016, at 7:51 PM, Kenneth Chan wrote: Pat, would you explain more about the 'instanceId' as in `pio register --variant path/to/some-engine.json --instanceId some-REST-compatible-resource-id` ? Currently PIO also has a concept of engineInstanceId, which is output of train. I think you are referring to different thing, right? Kenneth On Fri, Sep 16, 2016 at 12:58 PM, Pat Ferrel mailto:p...@occamsmachete.com>> wrote: This is a great discussion topic and a great idea. However the cons must also be addressed, we will need to do this before multi-tenant deploys can happen and the benefits are just as large as removing `pio build` It would be great to get rid of manifest.json and put all metadata in the store with an externally visible id so all parts of the workflow on all machines will get the right metadata and any template specific commands will run from anywhere on any cluster machine and in any order. All we need is a global engine-instance id. This will make engine-instances behave more like datasets, which are given permanent ids with `pio app new …` This might be a new form of `pio register` and it implies a new optional param to pio template specific commands (the instance id) but removes a lot of misunderstandings people have and easy mistakes in workflow. So workflow would be: 1) build with SBT/mvn 2) register any time engine.json changes so make the json file an optional param to `pio register --variant path/to/some-engine.json --instanceId some-REST-compatible-resource-id` the instance could also be auto-generated and output or optionally in the engine.json. `pio engine list` lists registered instances with instanceId. The path to the binary would be put in the instanceId and would be expected to be the same on all cluster machines that need it. 3) `pio train --instanceId` optional if it’s in engine.json 4) `pio deploy --instanceId` optional if it’s in engine.json 5) with easily recognized exceptions all the above can happen in any order on any cluster machine and from any directory. This takes one big step to multi-tenancy since the instance data has an externally visible id—call it a REST resource id… I bring this up not to confuse the issue but because if we change the workflow commands we should avoid doing it often because of the disruption it brings. On Sep 16, 2016, at 10:42 AM, Donald Szeto mailto:don...@apache.org>> wrote: Hi all, I want to start the discussion of removing engine registration. How many people actually take advantage of being able to run pio commands everywhere outside of an engine template directory? This will be a nontrivial change on the operational side so I want to gauge the potential impact to existing users. Pros: - Stateless build. This would work well with many PaaS. - Eliminate the "pio build" command once and for all. - Ability to use your own build system, i.e. Maven, Ant, Gradle, etc. - Potentially better experience with IDE since engine templates no longer depends on an SBT plugin. Cons: - Inability to run pio engine training and deployment commands outside of engine template directory. - No automatic version matching of PIO binary distribution and artifacts version used in the engine template. - A less unified user experience: from pio-build-train-deploy to build, then pio-train-deploy. Regards, Donald