> What is the policy driving dependency upgrades? Although it might be difficult to define it, how about the following policy:
- Select newer dependencies - If the above one is not supported in major platforms(ex. AWS, Cloudera, Horton), change to the lower version - Review the version of dependencies at every releases if needed As for elasticsearch, I would like to keep a newer version since it will be released monthly. > I don’t run hadoop 2.7 locally and many users that have Cloudera or Horton > contracts may not either. Thank you for the info. CDH seems not to support hadoop 2.7... Updated as below: <UPDATE> 0.12.0: - PIO_SCALA_VERSION=2.11.8 - PIO_SPARK_VERSION=2.1.1 - PIO_ELASTICSEARCH_VERSION=5.4.1 - PIO_HADOOP_VERSION=2.6.5 For Python 2/3, I fixed some templates, such as recommender. Since I think template scripts can support both Python 2/3, I'll fix them. Regards, shinsuke 2017-06-06 4:57 GMT+09:00 Pat Ferrel <[email protected]>: > What is the policy driving dependency upgrades? > > I don’t run hadoop 2.7 locally and many users that have Cloudera or Horton > contracts may not either. Not sure why this should be the default until it’s > the most popular of we need some feature of it. > > I’d agree with most of what @Shinsuke suggests as long as there is an easy > way to build for any reasonable combination of deps. > > The hard one will be Python 3. All existing python scripts in templates will > need upgrading since it’s very difficult to support mixed py2 and py3 where > scala 2.10 and 2,11 are much easier. I still think it’s time to do this but > mention it because with each upgrade we need to consider how may templates > are left even further behind. Many now do not work with Apache PIO, this may > put them further behind. > > Though we work on PIO we must remember that PIO does nothing interesting > without templates and ask ourselves what pain we may cause for template > users. > > > > On Jun 5, 2017, at 11:06 AM, Donald Szeto <[email protected]> wrote: > > Hey all, this has a huge impact to the default build, so If you see any > issue with this, please let us know as soon as possible. > > On Sun, Jun 4, 2017 at 10:25 PM, Shinsuke Sugaya <[email protected]> > wrote: >> >> Hi all, >> >> We have a plan to change default build targets in PIO-83 and PIO-84. >> Current versions look too old, so it will be better to support >> newer versions as default. >> >> Current: >> - PIO_SCALA_VERSION=2.10.6 >> - PIO_SPARK_VERSION=1.6.3 >> - PIO_ELASTICSEARCH_VERSION=1.7.6 >> - PIO_HADOOP_VERSION=2.6.5 >> >> They will be changed to: >> >> 0.12.0: >> - PIO_SCALA_VERSION=2.11.8 >> - PIO_SPARK_VERSION=2.1.1 >> - PIO_ELASTICSEARCH_VERSION=5.4.1 >> - PIO_HADOOP_VERSION=2.7.3 >> >> Note that this change does not discard old version supports. >> If you want to use old versions, you can build PIO with them. >> >> Please let us know if you have any concerns. >> >> https://issues.apache.org/jira/browse/PIO-83 >> https://issues.apache.org/jira/browse/PIO-84 >> >> Regards, >> shinsuke > > >
