Let me summarize this thread... 1) Default Build Targets
The current most popular combination is below: 0.12.0: - PIO_SCALA_VERSION=2.11.8 - PIO_SPARK_VERSION=2.1.1 - PIO_HADOOP_VERSION=2.6.5 - PIO_ELASTICSEARCH_VERSION=5.4.1 Please let us know if you have any concerns. Others: - Deprecating/deprecated is not included in this task. - If selecting the latest ES5 version, it will be able to work even if user deploys lower ES5.x. - We need to consider Python 2/3 support for templates. At the moment, I'll fix them to support both. 2) Dependencies Deprecation Process and Documentation Sara, could you handle this issue? (It might be better to also include upgrade policy..) 3) Binary distribution Please see/update https://issues.apache.org/jira/browse/PIO-69 4) HBase is old Filed: https://issues.apache.org/jira/browse/PIO-92 Regards, shinsuke 2017-06-09 2:45 GMT+09:00 Pat Ferrel <[email protected]>: > +1, the default build should not be the bleeding edge but rather the most > popular combination or even lagging it but we also need to have methods to > build the bleeding edge and deprecation so we minimize work to support all > versions of all deps. > > I like the poll idea but wonder how many will see the notice of polling. > Still it can’t hurt. > > BTW the implications of multiple binaries for these combinations means > re-checking all transitive deps with each binary combo. The part about the > ASF rules that make them particularly onerous is that licenses can change at > the whim of the copyright holder so my reading is that they need to be > re-checked every time a dep *version* is changed. We avoid this now and IMO > this makes any binary release of dubious value given the cost and give that > *all* Templates are build from source anyway. A tool to do this > automatically is about the only way to make this work. > > > On Jun 8, 2017, at 9:19 AM, Donald Szeto <[email protected]> wrote: > > Unfortunately with OSS it is not easy to survey accurately the user base > unless we provide some sort of opt-in feedback (automatically or by putting > up a poll on the web site). We have many users who don't subscribe to > mailing lists. > > I personally prefer updating source build defaults to a reasonably popular > combination. Note that this is independent from the process of deprecating > and eventually dropping support of old versions of dependencies. > > I feel that the major common discomfort here is the fear that some > old/stable versions of dependencies may get dropped without notice, instead > of reluctance to support new technologies. If so, I would propose we agree > on a deprecation process and document it. To enforce it, I would suggest > including all reasonable combinations of dependencies in our automatic > builds. This also has a benefit that once we are cleared on releasing > binaries, we can roll them all out to make it convenient for our users. > > On Wed, Jun 7, 2017 at 10:31 PM takako shimamoto <[email protected]> > wrote: >> >> I also agree with shinsuke. >> >> > Note that this change does not discard old version supports. >> > If you want to use old versions, you can build PIO with them. >> >> This means that the versions supported by PIO don't change. >> Users will be able to build PIO from source distribution as same as >> the current situation. >> >> As for supported versions (when and which versions to be deprecated),I >> think we should discuss it in another thread. >> In that case, it might be a good idea to decide it based on project >> policy since PIO is an open source project at the ASF. >> >> >> 2017-06-08 11:18 GMT+09:00 Naoki Takezoe <[email protected]>: >> >> For Hadoop 2.6 and Spark 2.1, our updated dependencies will work. >> > >> > +1 >> > >> > We should always catch up latest versions of Hadoop, Spark and so on, >> > but default build targets should cover existing popular environments >> > as much as possible. >> > >> > In addition, HBase version (0.98.5) looks much old. It's already EOM. >> > We should upgrade it to 1.2 at least. >> > >> > 2017-06-08 5:51 GMT+09:00 Pat Ferrel <[email protected]>: >> >> Supporting the latest and requiring them are 2 different things. >> >> Requiring them (except for ES) means PIO won’t run unless the clusters for >> >> every user are upgraded to match the client because only backward >> >> compatibility is supported. Last time I checked if you require HDFS 2.7, >> >> PIO >> >> won’t run on 2.6. If you require 2.6 PIO will run on 2.6 or 2.7 so >> >> immediate >> >> upgrades have no benefit. The last I checked there was no forward >> >> compatibility guarantee. Has this changed? >> >> >> >> If ES guarantees forward compatibility that is great. >> >> >> >> >> >> On Jun 7, 2017, at 11:08 AM, Mars Hall <[email protected]> wrote: >> >> >> >> These upgrades are very similar to the dependencies we support/provide >> >> for PredictionIO 0.11.0-incubating in the Heroku buildpack. >> >> >> >> If the framework is going to upgrade default dependencies, I >> >> wholeheartedly agree that moving to the most recent versions of everything >> >> is the way to go. >> >> >> >> Once PIO reaches 1.0 releases, I'd imagine that every major dependency >> >> upgrade would be taken together and increment the PredictionIO major >> >> version. >> >> >> >> *Mars >> >> >> >> ( <> .. <> ) >> >> >> >>> On Jun 4, 2017, at 22:25, Shinsuke Sugaya <[email protected]> >> >>> wrote: >> >>> >> >>> Hi all, >> >>> >> >>> We have a plan to change default build targets in PIO-83 and PIO-84. >> >>> Current versions look too old, so it will be better to support >> >>> newer versions as default. >> >>> >> >>> Current: >> >>> - PIO_SCALA_VERSION=2.10.6 >> >>> - PIO_SPARK_VERSION=1.6.3 >> >>> - PIO_ELASTICSEARCH_VERSION=1.7.6 >> >>> - PIO_HADOOP_VERSION=2.6.5 >> >>> >> >>> They will be changed to: >> >>> >> >>> 0.12.0: >> >>> - PIO_SCALA_VERSION=2.11.8 >> >>> - PIO_SPARK_VERSION=2.1.1 >> >>> - PIO_ELASTICSEARCH_VERSION=5.4.1 >> >>> - PIO_HADOOP_VERSION=2.7.3 >> >>> >> >>> Note that this change does not discard old version supports. >> >>> If you want to use old versions, you can build PIO with them. >> >>> >> >>> Please let us know if you have any concerns. >> >>> >> >>> https://issues.apache.org/jira/browse/PIO-83 >> >>> https://issues.apache.org/jira/browse/PIO-84 >> >>> >> >>> Regards, >> >>> shinsuke >> >> >> >> >> > >> > >> > >> > -- >> > Naoki Takezoe > >
