Re: [discuss] ending support for Java 7 in Spark 2.0

I do agree w.r.t scala 2.10 as well; similar arguments apply (though there is a nuanced diff - source compatibility for scala vs binary compatibility wrt Java) Was there a proposal which did not go through ? Not sure if I missed it. Regards Mridul On Thursday, March 24, 2016, Koert Kuipers wrote

Re: Does SparkSql has official jdbc/odbc driver ?

No - it is too painful to develop a jdbc/odbc driver. On Thu, Mar 24, 2016 at 11:56 PM, sage wrote: > Hi all, >Does SparkSql has official jdbc/odbc driver? >I only found third-party's odbc/jdbc driver, like simba, and most of > third-party's odbc/jdbc driver are not free to use. > > > >

Does SparkSql has official jdbc/odbc driver ?

Hi all, Does SparkSql has official jdbc/odbc driver? I only found third-party's odbc/jdbc driver, like simba, and most of third-party's odbc/jdbc driver are not free to use. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Does-SparkSql-has-offi

Re: Can we remove private[spark] from Metrics Source and SInk traits?

+1 on exposing the source/sink interface, since MetricsSystem by natural support plugin source and sink, so supporting this don't require a big change of current code. Also there's lot of requirements to add custom sink and source to the MetricsSystem, it is not suitable to maintain these code in S

Re: [discuss] ending support for Java 7 in Spark 2.0

I do agree w.r.t scala 2.10 as well; similar arguments apply (though there is a nuanced diff - source compatibility for scala vs binary compatibility wrt Java) Was there a proposal which did not go through ? Not sure if I missed it. Regards Mridul On Thursday, March 24, 2016, Koert Kuipers wrote

Re: [discuss] ending support for Java 7 in Spark 2.0

Arguments are really convincing; new Dataset API as well as performance improvements is exiting, so I'm personally +1 on moving onto Java8. However, I'm afraid Tencent is one of "the organizations stuck with Java7" -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, and wou

Re: [discuss] ending support for Java 7 in Spark 2.0

i think that logic is reasonable, but then the same should also apply to scala 2.10, which is also unmaintained/unsupported at this point (basically has been since march 2015 except for one hotfix due to a license incompatibility) who wants to support scala 2.10 three years after they did the last

Re: [discuss] ending support for Java 7 in Spark 2.0

Removing compatibility (with jdk, etc) can be done with a major release- given that 7 has been EOLed a while back and is now unsupported, we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from now). Given the functionality & performance benefits of going to jdk8, future enhanceme

Re: [discuss] ending support for Java 7 in Spark 2.0

i think marcelo also pointed this out before. its very interesting to hear, i was not aware of that until today. it would mean we would only have to convince a group/client with a cluster to install jdk8 on the nodes, without actually transitioning to it, if i understand it correctly. that would de

Re: [discuss] ending support for Java 7 in Spark 2.0

Container Java version can be different from yarn Java version : we run jobs with jdk8 on jdk7 cluster without issues. Regards Mridul On Thursday, March 24, 2016, Koert Kuipers wrote: > i guess what i am saying is that in a yarn world the only hard > restrictions left are the the containers you

Re: [discuss] ending support for Java 7 in Spark 2.0

the good news is, that from an shared infrastructure perspective, most places have zero scala, so the upgrade is actually very easy. i can see how it would be different for say twitter On Thu, Mar 24, 2016 at 7:50 PM, Reynold Xin wrote: > If you want to go down that route, you should also as

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 4:54 PM, Mark Hamstra wrote: > It's a pain in the ass. Especially if some of your transitive > dependencies never upgraded from 2.10 to 2.11. > Yeah, I'm going to have to agree here. It is not as bad as it was in the 2.9 days, but its still non-trivial due to the eco-s

Re: [discuss] ending support for Java 7 in Spark 2.0

There aren't many such libraries, but there are a few. When faced with one of those dependencies that still doesn't go beyond 2.10, you essentially have the choice of taking on the maintenance burden to bring the library up to date, or you do what is potentially a fairly larger refactoring to use

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin wrote: > If you want to go down that route, you should also ask somebody who has had > experience managing a large organization's applications and try to update > Scala version. I understand both sides. But if you look at what I've been asking since th

Re: [discuss] ending support for Java 7 in Spark 2.0

It's a pain in the ass. Especially if some of your transitive dependencies never upgraded from 2.10 to 2.11. On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin wrote: > If you want to go down that route, you should also ask somebody who has > had experience managing a large organization's application

Re: [discuss] ending support for Java 7 in Spark 2.0

In addition, with Spark 2.0, we are throwing away binary compatibility anyways so user applications will have to be recompiled. The only argument I can see is for libraries that have already been built on Scala 2.10 that are no longer being maintained. How big of an issue do we think that is? Kos

Re: [discuss] ending support for Java 7 in Spark 2.0

If you want to go down that route, you should also ask somebody who has had experience managing a large organization's applications and try to update Scala version. On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin wrote: > On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin wrote: > > Actually it's *w

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin wrote: > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not > binary compatible, whereas JVM 7 and 8 are binary compatible except certain > esoteric cases. Tr

Re: [discuss] ending support for Java 7 in Spark 2.0

Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, than upgrading the JVM runtime from 7 to 8, because Scala 2.10 and 2.11 are not binary compatible, whereas JVM 7 and 8 are binary compatible except certain esoteric cases. On Thu, Mar 24, 2016 at 4:44 PM, Kostas Sakellis wrote: > If

Re: [discuss] ending support for Java 7 in Spark 2.0

If an argument here is the ongoing build/maintenance burden I think we should seriously consider dropping scala 2.10 in Spark 2.0. Supporting scala 2.10 is bigger build/infrastructure burden than supporting jdk7 since you actually have to build different artifacts and test them whereas you can targ

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky wrote: > You can, but since it's going to be a maintainability issue I would > argue it is in fact a problem. Every thing you choose to support generates a maintenance burden. Support 3 versions of Scala would be a huge maintenance burden, for exampl

Re: [discuss] ending support for Java 7 in Spark 2.0

I mean from the perspective of someone developing Spark, it makes things more complicated. It's just my point of view, people that actually support Spark deployments may have a different opinion ;) On Thu, Mar 24, 2016 at 2:41 PM, Jakob Odersky wrote: > You can, but since it's going to be a maint

Re: [discuss] ending support for Java 7 in Spark 2.0

You can, but since it's going to be a maintainability issue I would argue it is in fact a problem. On Thu, Mar 24, 2016 at 2:34 PM, Marcelo Vanzin wrote: > Hi Jakob, > > On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky wrote: >> Reynold's 3rd point is particularly strong in my opinion. Supporting

Re: [discuss] ending support for Java 7 in Spark 2.0

Hi Jakob, On Thu, Mar 24, 2016 at 2:29 PM, Jakob Odersky wrote: > Reynold's 3rd point is particularly strong in my opinion. Supporting > Consider what would happen if Spark 2.0 doesn't require Java 8 and > hence not support Scala 2.12. Will it be stuck on an older version > until 3.0 is out? Tha

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 for Java 8 only I think it will make it easier to make a unified API for Java and Scala, instead of the wrappers of Java over Scala. On Mar 24, 2016 11:46 AM, "Stephen Boesch" wrote: > +1 for java8 only +1 for 2.11+ only .At this point scala libraries > supporting only 2.10 are typicall

Re: [discuss] ending support for Java 7 in Spark 2.0

Reynold's 3rd point is particularly strong in my opinion. Supporting Scala 2.12 will require Java 8 anyway, and introducing such a change is probably best done in a major release. Consider what would happen if Spark 2.0 doesn't require Java 8 and hence not support Scala 2.12. Will it be stuck on an

Re: [discuss] ending support for Java 7 in Spark 2.0

Spark 2.x has to be the time for Java 8. I'd rather increase JVM major version on a Spark major version than on a Spark minor version, and I'd rather Spark do that upgrade for the 2.x series than the 3.x series (~2yr from now based on the lifetime of Spark 1.x). If we wait until the next opportun

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Patrick is investigating. On Thu, Mar 24, 2016 at 7:25 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Just checking in on this again as the builds on S3 are still broken. :/ > > Could it have something to do with us moving release-build.sh >

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 for java8 only +1 for 2.11+ only .At this point scala libraries supporting only 2.10 are typically less active and/or poorly maintained. That trend will only continue when considering the lifespan of spark 2.X. 2016-03-24 11:32 GMT-07:00 Steve Loughran : > > On 24 Mar 2016, at 15:27, Koe

Re: [discuss] ending support for Java 7 in Spark 2.0

On 24 Mar 2016, at 15:27, Koert Kuipers mailto:ko...@tresata.com>> wrote: i think the arguments are convincing, but it also makes me wonder if i live in some kind of alternate universe... we deploy on customers clusters, where the OS, python version, java version and hadoop distro are not chos

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 10:13 AM, Reynold Xin wrote: > Yes So is it safe to say the only hard requirements for Java 8 in your list is (4)? (1) and (3) are infrastructure issues. Yes, it sucks to maintain more testing infrastructure and potentially more complicated build scripts, but does that re

Re: [discuss] ending support for Java 7 in Spark 2.0

Yes On Thursday, March 24, 2016, Marcelo Vanzin wrote: > On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin > wrote: > > I actually talked quite a bit today with an engineer on the scala > compiler > > team tonight and the scala 2.10 + java 8 combo should be ok. The latest > > Scala 2.10 release shou

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 9:54 AM, Koert Kuipers wrote: > i guess what i am saying is that in a yarn world the only hard restrictions > left are the the containers you run in, which means the hadoop version, java > version and python version (if you use python). It is theoretically possible to run

Re: [discuss] ending support for Java 7 in Spark 2.0

On Thu, Mar 24, 2016 at 1:04 AM, Reynold Xin wrote: > I actually talked quite a bit today with an engineer on the scala compiler > team tonight and the scala 2.10 + java 8 combo should be ok. The latest > Scala 2.10 release should have all the important fixes that are needed for > Java 8. So, do

Re: [discuss] ending support for Java 7 in Spark 2.0

i guess what i am saying is that in a yarn world the only hard restrictions left are the the containers you run in, which means the hadoop version, java version and python version (if you use python). On Thu, Mar 24, 2016 at 12:39 PM, Koert Kuipers wrote: > The group will not upgrade to spark 2

Re: [discuss] ending support for Java 7 in Spark 2.0

Thank you for the context Jean... I appreciate it... On Thu, Mar 24, 2016 at 12:40 PM, Jean-Baptiste Onofré wrote: > Hi Al, > > Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will go > on Spark 2.0, but maintenance release can be performed on 1.x branch. > > Regards > JB > >

Re: [discuss] ending support for Java 7 in Spark 2.0

Hi Al, Spark 2.0 doesn't mean Spark 1.x will stop. Clearly, new features will go on Spark 2.0, but maintenance release can be performed on 1.x branch. Regards JB On 03/24/2016 05:38 PM, Al Pivonka wrote: As an end user (developer) and Cluster Admin. I would have to agree with Koet. To me th

Re: [discuss] ending support for Java 7 in Spark 2.0

The group will not upgrade to spark 2.0 themselves, but they are mostly fine with vendors like us deploying our application via yarn with whatever spark version we choose (and bundle, so they do not install it separately, they might not even be aware of what spark version we use). This all works be

Re: [discuss] ending support for Java 7 in Spark 2.0

As an end user (developer) and Cluster Admin. I would have to agree with Koet. To me the real question is timing, current version is 1.6.1, the question I have is how many more releases till 2.0 and what is the time frame? If you give people six to twelve months to plan and make sure they know (

Re: [discuss] ending support for Java 7 in Spark 2.0

(PS CDH5 runs fine with Java 8, but I understand your more general point.) This is a familiar context indeed, but in that context, would a group not wanting to update to Java 8 want to manually put Spark 2.0 into the mix? That is, if this is a context where the cluster is purposefully some stable

Re: [discuss] ending support for Java 7 in Spark 2.0

i think the arguments are convincing, but it also makes me wonder if i live in some kind of alternate universe... we deploy on customers clusters, where the OS, python version, java version and hadoop distro are not chosen by us. so think centos 6, cdh5 or hdp 2.3, java 7 and python 2.6. we simply

Re: Spark 1.6.1 Hadoop 2.6 package on S3 corrupt?

Just checking in on this again as the builds on S3 are still broken. :/ Could it have something to do with us moving release-build.sh ? ​ On Mon, Mar 21, 2016 at 1:43 PM Nicholas Chammas wrote: > Is someone goi

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 to support Java 8 (and future) *only* in Spark 2.0, and end support of Java 7. It makes sense. Regards JB On 03/24/2016 08:27 AM, Reynold Xin wrote: About a year ago we decided to drop Java 6 support in Spark 1.5. I am wondering if we should also just drop Java 7 support in Spark 2.0 (i.e.

Re: [discuss] ending support for Java 7 in Spark 2.0

> On 24 Mar 2016, at 07:27, Reynold Xin wrote: > > About a year ago we decided to drop Java 6 support in Spark 1.5. I am > wondering if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark > 2.0 would require Java 8 to run). > > Oracle ended public updates for JDK 7 in one year ag

Re: [discuss] ending support for Java 7 in Spark 2.0

Maybe so; I think we have a ticket open to update to 2.10.6, which maybe fixes it. It brings up a different point: supporting multiple Scala versions is much more painful than Java versions because of mutual incompatibility. Right now I get the sense there's an intent to keep supporting 2.10, and

Re: [discuss] ending support for Java 7 in Spark 2.0

I actually talked quite a bit today with an engineer on the scala compiler team tonight and the scala 2.10 + java 8 combo should be ok. The latest Scala 2.10 release should have all the important fixes that are needed for Java 8. On Thu, Mar 24, 2016 at 1:01 AM, Sean Owen wrote: > I generally fa

Re: [discuss] ending support for Java 7 in Spark 2.0

I generally favor this for the simplification. I didn't realize there were actually some performance wins and important bug fixes. I've had lots of trouble with scalac 2.10 + Java 8. I don't know if it's still a problem since 2.11 + 8 seems OK, but for a long time the sql/ modules would never comp

Re: [discuss] ending support for Java 7 in Spark 2.0

+1, yes Java 7 has been end of life for a year now, 2.0 is a good time to upgrade to Java 8 On Thu, Mar 24, 2016 at 12:42 AM, Raymond Honderdors < raymond.honderd...@sizmek.com> wrote: > Very good points > > > > Going to support java 8 looks like a good direction > > 2.0 would be a good release t

Re: [discuss] ending support for Java 7 in Spark 2.0

+1 Agree, dropping support for java 7 is long overdue - and 2.0 would be a logical release to do this on. Regards, Mridul On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin wrote: > About a year ago we decided to drop Java 6 support in Spark 1.5. I am > wondering if we should also just drop Java 7 s

RE: [discuss] ending support for Java 7 in Spark 2.0

Very good points Going to support java 8 looks like a good direction 2.0 would be a good release to start with that Raymond Honderdors Team Lead Analytics BI Business Intelligence Developer raymond.honderd...@sizmek.com T +972.7325.3569 Herzliya From: Reynol

Re: [discuss] ending support for Java 7 in Spark 2.0

One other benefit that I didn't mention is that we'd be able to use Java 8's Optional class to replace our built-in Optional. On Thu, Mar 24, 2016 at 12:27 AM, Reynold Xin wrote: > About a year ago we decided to drop Java 6 support in Spark 1.5. I am > wondering if we should also just drop Java

[discuss] ending support for Java 7 in Spark 2.0

About a year ago we decided to drop Java 6 support in Spark 1.5. I am wondering if we should also just drop Java 7 support in Spark 2.0 (i.e. Spark 2.0 would require Java 8 to run). Oracle ended public updates for JDK 7 in one year ago (Apr 2015), and removed public downloads for JDK 7 in July 201

答复: MLPC model can not be saved

Hi Alexander, Thanks for your reply. The pull request shows that MultilayerPerceptronClassifier implement default params writable interface. I will try that. Thanks Pan 发件人: Ulanov, Alexander [mailto:alexander.ula...@hpe.com] 发送时间: 2016年3月22日 1:38 收件人: HanPan; dev@spark.apach