Re: Spark Improvement Proposals

Let us continue to improve Apache Spark! I volunteer to go through all the SQL-related open JIRAs. Xiao Li 2016-10-06 21:14 GMT-07:00 Matei Zaharia <matei.zaha...@gmail.com>: > Hey Cody, > > Thanks for bringing these things up. You're talking about quite a few > different th

Re: Welcoming Felix Cheung as a committer

Congrats Felix! 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier : > Congrats Felix! > > On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar wrote: >> >> Congrats Felix! >> >> On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar

Re: [VOTE] Release Apache Spark 2.0.0 (RC5)

+1 2016-07-22 19:32 GMT-07:00 Kousuke Saruta : > +1 (non-binding) > > Tested on my cluster with three slave nodes. > > On 2016/07/23 10:25, Suresh Thalamati wrote: > > +1 (non-binding) > > Tested data source api , and jdbc data sources. > > > On Jul 19, 2016, at 7:35

Re: Does dataframe write append mode work with text format

ot;) .mode(SaveMode.Append).text(dir.getCanonicalPath) val newDF2 = spark.read.text(dir.getCanonicalPath) newDF2.show() I tried it. It works well. Thanks, Xiao Li 2016-06-18 8:57 GMT-07:00 Yash Sharma <yash...@gmail.com>: > Hi All, > I have been using the parquet append mode

Re: Welcoming Yanbo Liang as a committer

Congratulations, Yanbo! 2016-06-03 19:54 GMT-07:00 Nan Zhu : > Congratulations ! > > -- > Nan Zhu > > On June 3, 2016 at 10:50:33 PM, Ted Yu (yuzhih...@gmail.com) wrote: > > Congratulations, Yanbo. > > On Fri, Jun 3, 2016 at 7:48 PM, Matei Zaharia

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Changed my vote to +1. Thanks! 2016-05-19 13:28 GMT-07:00 Xiao Li <gatorsm...@gmail.com>: > Will do. Thanks! > > 2016-05-19 13:26 GMT-07:00 Reynold Xin <r...@databricks.com>: > >> Xiao thanks for posting. Please file a bug in JIRA. Again as I said in

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

Will do. Thanks! 2016-05-19 13:26 GMT-07:00 Reynold Xin <r...@databricks.com>: > Xiao thanks for posting. Please file a bug in JIRA. Again as I said in the > email this is not meant to be a functional release and will contain bugs. > > On Thu, May 19, 2016 at 1:20 PM

Re: [vote] Apache Spark 2.0.0-preview release (rc1)

-1 Unable to use Hive meta-store in pyspark shell. Tried both HiveContext and SparkSession. Both failed. It always uses in-memory catalog. Anybody else hit the same issue? Method 1: SparkSession >>> from pyspark.sql import SparkSession >>> spark =

Re: [discuss] using deep learning to improve Spark

April 1st... : ) 2016-04-01 0:33 GMT-07:00 Michael Malak : > I see you've been burning the midnight oil. > > > -- > *From:* Reynold Xin > *To:* "dev@spark.apache.org" > *Sent:* Friday, April

Re: Welcoming two new committers

Congratulations! Herman and Wenchen! I am just so happy for you! You absolutely deserve it! 2016-02-08 9:35 GMT-08:00 Corey Nolet : > Congrats guys! > > On Mon, Feb 8, 2016 at 12:23 PM, Ted Yu wrote: > >> Congratulations, Herman and Wenchen. >> >> On

Re: SQL language vs DataFrame API

Hi, Michael, Does that mean SqlContext will be built on HiveQL in the near future? Thanks, Xiao Li 2015-12-09 10:36 GMT-08:00 Michael Armbrust <mich...@databricks.com>: > I think that it is generally good to have parity when the functionality is > useful. However, in some c

Re: SQL language vs DataFrame API

That sounds great! When it is decided, please let us know and we can add more features and make it ANSI SQL compliant. Thank you! Xiao Li 2015-12-09 11:31 GMT-08:00 Michael Armbrust <mich...@databricks.com>: > I don't plan to abandon HiveQL compatibility, but I'd like to see us move

Re: Proposal for SQL join optimization

Hi, Zhan, That sounds really interesting! Please at me when you submit the PR. If possible, please also posted the performance difference. Thanks, Xiao Li 2015-11-11 14:45 GMT-08:00 Zhan Zhang <zzh...@hortonworks.com>: > Hi Folks, > > I did some performance measurement

Spark SQL: what does an exclamation mark mean in the plan?

] LogicalRDD [col1#6,col2#7,col3#8], MapPartitionsRDD[1] at createDataFrame at SimpleApp.scala:32 Could anybody give me a hint why there exists a !(exclamation mark) before the node name (Project)? This ! mark does not disappear in the subsequent query plan. Thank you! Xiao Li

Re: Spark SQL: what does an exclamation mark mean in the plan?

issingInput.nonEmpty && children.nonEmpty) "!" else "" override def simpleString: String = statePrefix + super.simpleString Xiao Li 2015-10-19 11:16 GMT-07:00 Michael Armbrust <mich...@databricks.com>: > It means that there is an invalid attribute reference (

Re: Should enforce the uniqueness of field name in DataFrame ?

True. As long as we can ensure the correct message are printed out, users can correct their app easily. For example, Reference 'name' is ambiguous, could be: name#1, name#5.; Thanks, Xiao Li 2015-10-14 23:58 GMT-07:00 Reynold Xin <r...@databricks.com>: > That could break a lot of app

Re: sbt test error -- "Could not reserve enough space"

Hi, Robert, Please check the following link. It might help you. http://stackoverflow.com/questions/18155325/scala-error-occurred-during-initialization-of-vm-on-ubuntu-12-04 Good luck, Xiao Li 2015-10-09 9:41 GMT-07:00 Robert Dodier <robert.dod...@gmail.com>: > Hi, > > I am t

Re: Build spark 1.5.1 branch fails

Hi, Chester, Please check your pom.xml. Your java.version and maven.version might not match your build environment. Or using -Denforcer.skip=true from the command line to skip it. Good luck, Xiao Li 2015-10-08 10:35 GMT-07:00 Chester Chen <ches...@alpinenow.com>: > Question regardi

Re: Understanding code/closure shipment to Spark workers‏

hod.invoke(null, childArgs.toArray) in the same file Function Invoke() is provided by JAVA Reflection for invoking the main function of your JAR. Hopefully, it can help you understand the problem. Thanks, Xiao Li 2015-10-07 16:47 GMT-07:00 Arijit <arij...@live.com>: > Hi, > > I wa

<    1   2   3   4