Re: [discuss] dropping Python 2.6 support

FWIW, RHEL 6 still uses Python 2.6, although 2.7.8 and 3.3.2 are available through Red Hat Software Collections. See: https://www.softwarecollections.org/en/ I run an academic compute cluster on RHEL 6. We do, however, provide Python 2.7.x and 3.5.x via modulefiles. On Tue, Jan 5, 2016 at 8:45

Re: [discuss] dropping Python 2.6 support

Sasha, it is more complicated than that: many RHEL 6 OS utilities rely on Python 2.6. Upgrading it to 2.7 breaks the system. For large enterprises migrating to another server OS means re-certifying (re-testing) hundreds of applications, so yes, they do prefer to stay where they are until the

Re: [discuss] dropping Python 2.6 support

+1 Companies that use stock python in redhat 2.6 will need to upgrade or install fresh version wich is total of 3.5 minutes so no issues ... On Tue, Jan 5, 2016 at 2:17 AM, Reynold Xin wrote: > Does anybody here care about us dropping support for Python 2.6 in Spark > 2.0?

Re: [discuss] dropping Python 2.6 support

Chiming in late, but my take on this line of argument is: these companies are welcome to keep using Spark 1.x. If anything the argument here is about how long to maintain 1.x, and indeed, it's going to go dormant quite soon. But using RHEL 6 (or any old-er version of any platform) and not wanting

Re: [discuss] dropping Python 2.6 support

On Sat, Jan 9, 2016 at 1:48 PM, Sean Owen wrote: > (For similar reasons I personally don't favor supporting Java 7 or > Scala 2.10 in Spark 2.x.) That reflects my sentiments as well. Thanks Sean for bringing that up! Jacek

Re: [discuss] dropping Python 2.6 support

+juliet for an additional opinion, but FWIW I think it's safe to say that future CDH will have a more consistent Python story and that story will support 2.7 rather than 2.6. On Tue, Jan 5, 2016 at 7:17 AM, Reynold Xin wrote: > Does anybody here care about us dropping

Re: [discuss] dropping Python 2.6 support

+1 Red Hat supports Python 2.6 on REHL 5 until 2020 , but otherwise yes, Python 2.6 is ancient history and the core Python developers stopped supporting it in 2013. REHL 5 is not a good enough reason to continue support for Python

Re: [discuss] dropping Python 2.6 support

rhel/centos 6 ships with python 2.6, doesnt it? if so, i still know plenty of large companies where python 2.6 is the only option. asking them for python 2.7 is not going to work so i think its a bad idea On Tue, Jan 5, 2016 at 1:52 PM, Juliet Hougland wrote: > I

Re: [discuss] dropping Python 2.6 support

+1 > On Jan 5, 2016, at 10:49 AM, Davies Liu wrote: > > +1 > > On Tue, Jan 5, 2016 at 5:45 AM, Nicholas Chammas > wrote: >> +1 >> >> Red Hat supports Python 2.6 on REHL 5 until 2020, but otherwise yes, Python >> 2.6 is ancient history and

Re: [discuss] dropping Python 2.6 support

+1 On Tue, Jan 5, 2016 at 5:45 AM, Nicholas Chammas wrote: > +1 > > Red Hat supports Python 2.6 on REHL 5 until 2020, but otherwise yes, Python > 2.6 is ancient history and the core Python developers stopped supporting it > in 2013. REHL 5 is not a good enough reason

Re: [discuss] dropping Python 2.6 support

Hey Python 2.6 don't let the door hit you on the way out! haha Drop It No Problem On 01/05/2016 12:17 AM, Reynold Xin wrote: Does anybody here care about us dropping support for Python 2.6 in Spark 2.0? Python 2.6 is ancient, and is pretty slow in many aspects (e.g. json parsing) when

Re: [discuss] dropping Python 2.6 support

As I pointed out in my earlier email, RHEL will support Python 2.6 until 2020. So I'm assuming these large companies will have the option of riding out Python 2.6 until then. Are we seriously saying that Spark should likewise support Python 2.6 for the next several years? Even though the core

Re: [discuss] dropping Python 2.6 support

yeah, the practical concern is that we have no control over java or python version on large company clusters. our current reality for the vast majority of them is java 7 and python 2.6, no matter how outdated that is. i dont like it either, but i cannot change it. we currently don't use pyspark

Re: [discuss] dropping Python 2.6 support

If users are able to install Spark 2.0 on their RHEL clusters, then I imagine that they're also capable of installing a standalone Python alongside that Spark version (without changing Python systemwide). For instance, Anaconda/Miniconda make it really easy to install Python 2.7.x/3.x without

Re: [discuss] dropping Python 2.6 support

i do not think so. does the python 2.7 need to be installed on all slaves? if so, we do not have direct access to those. also, spark is easy for us to ship with our software since its apache 2 licensed, and it only needs to be present on the machine that launches the app (thanks to yarn). even

Re: [discuss] dropping Python 2.6 support

+1 On Wed, Jan 6, 2016 at 9:18 AM, Juliet Hougland wrote: > Most admins I talk to about python and spark are already actively (or on > their way to) managing their cluster python installations. Even if people > begin using the system python with pyspark, there is

Re: [discuss] dropping Python 2.6 support

hey evil admin:) i think the bit about java was from me? if so, i meant to indicate that the reality for us is java is 1.7 on most (all?) clusters. i do not believe spark prefers java 1.8. my point was that even although java 1.7 is getting old as well it would be a major issue for me if spark

Re: [discuss] dropping Python 2.6 support

I don't think that we're planning to drop Java 7 support for Spark 2.0. Personally, I would recommend using Java 8 if you're running Spark 1.5.0+ and are using SQL/DataFrames so that you can benefit from improvements to code cache flushing in the Java 8 JVMs. Spark SQL's generated classes can

Re: [discuss] dropping Python 2.6 support

Created JIRA: https://issues.apache.org/jira/browse/SPARK-12661 On Tue, Jan 5, 2016 at 2:49 PM, Koert Kuipers wrote: > i do not think so. > > does the python 2.7 need to be installed on all slaves? if so, we do not > have direct access to those. > > also, spark is easy for us

Re: [discuss] dropping Python 2.6 support

interesting i didnt know that! On Tue, Jan 5, 2016 at 5:57 PM, Nicholas Chammas wrote: > even if python 2.7 was needed only on this one machine that launches the > app we can not ship it with our software because its gpl licensed > > Not to nitpick, but maybe this

Re: [discuss] dropping Python 2.6 support

I think all the slaves need the same (or a compatible) version of Python installed since they run Python code in PySpark jobs natively. On Tue, Jan 5, 2016 at 6:02 PM Koert Kuipers wrote: > interesting i didnt know that! > > On Tue, Jan 5, 2016 at 5:57 PM, Nicholas Chammas <

Re: [discuss] dropping Python 2.6 support

> > Note that you _can_ use a Python 2.7 `ipython` executable on the driver > while continuing to use a vanilla `python` executable on the executors Whoops, just to be clear, this should actually read "while continuing to use a vanilla `python` 2.7 executable". On Tue, Jan 5, 2016 at 3:07 PM,

Re: [discuss] dropping Python 2.6 support

even if python 2.7 was needed only on this one machine that launches the app we can not ship it with our software because its gpl licensed Not to nitpick, but maybe this is important. The Python license is GPL-compatible but not GPL : Note GPL-compatible

Re: [discuss] dropping Python 2.6 support

if python 2.7 only has to be present on the node that launches the app (does it?) than that could be important indeed. On Tue, Jan 5, 2016 at 6:02 PM, Koert Kuipers wrote: > interesting i didnt know that! > > On Tue, Jan 5, 2016 at 5:57 PM, Nicholas Chammas < >

Re: [discuss] dropping Python 2.6 support

Yep, the driver and executors need to have compatible Python versions. I think that there are some bytecode-level incompatibilities between 2.6 and 2.7 which would impact the deserialization of Python closures, so I think you need to be running the same 2.x version for all communicating Spark

[discuss] dropping Python 2.6 support

Does anybody here care about us dropping support for Python 2.6 in Spark 2.0? Python 2.6 is ancient, and is pretty slow in many aspects (e.g. json parsing) when compared with Python 2.7. Some libraries that Spark depend on stopped supporting 2.6. We can still convince the library maintainers to