Re: Shared memory between C++ process and Spark

18:50, Annabel Melongo <melongo_anna...@yahoo.com> wrote: Jia, I'm so confused on this. The architecture of Spark is to run on top of HDFS. What you're requesting, reading and writing to a C++ process, is not part of that requirement. On Monday, December 7, 2015 1:42 PM, Jia

Re: Shared memory between C++ process and Spark

ibrary. Robin Sent from my iPhone On 7 Dec 2015, at 18:50, Annabel Melongo <melongo_anna...@yahoo.com.INVALID> wrote: Jia, I'm so confused on this. The architecture of Spark is to run on top of HDFS. What you're requesting, reading and writing to a C++ process, is not part of that

Re: Shared memory between C++ process and Spark

My guess is that Jia wants to run C++ on top of Spark. If that's the case, I'm afraid this is not possible. Spark has support for Java, Python, Scala and R. The best way to achieve this is to run your application in C++ and used the data created by said application to do manipulation within

Re: Shared memory between C++ process and Spark

y need to clarify that I have no intention to write and run Spark UDF in C++, I'm just wondering whether Spark can read and write data to a C++ process with zero copy. Best Regards,Jia  On Dec 7, 2015, at 12:26 PM, Annabel Melongo <melongo_anna...@yahoo.com> wrote: My guess is that Jia wa

Re: Is Spark 1.6 released?

[1] http://spark.apache.org/releases/spark-release-1-6-0.html[2]  http://spark.apache.org/downloads.html On Monday, January 4, 2016 2:59 PM, "saif.a.ell...@wellsfargo.com" wrote: Where can I read more about the dataset api on a user layer? I am failing to

Re: Stuck with DataFrame df.select("select * from table");

don't compile =)If your example is in Scala, then, I believe, semicolon is not required. -- Be well! Jean Morozov On Mon, Dec 28, 2015 at 8:49 PM, Annabel Melongo <melongo_anna...@yahoo.com> wrote: Jean, Try this:df.select("""select * from tmptable where x1 = '3.0'"&qu

Re: java.io.FileNotFoundException(Too many open files) in Spark streaming

Vijay, Are you closing the fileinputstream at the end of each loop ( in.close())? My guess is those streams aren't close and thus the "too many open files" exception. On Tuesday, January 5, 2016 8:03 AM, Priya Ch wrote: Can some one throw light on this ?

Re: Spark job uses only one Worker

Michael, I don't know what's your environment but if it's Cloudera, you should be able to see the link to your master in the Hue. Thanks On Thursday, January 7, 2016 5:03 PM, Michael Pisula wrote: I had tried several parameters, including

Re: Date Time Regression as Feature

Or he can also transform the whole date into a string On Thursday, January 7, 2016 2:25 PM, Sujit Pal wrote: Hi Jorge, Maybe extract things like dd, mm, day of week, time of day from the datetime string and use them as features? -sujit On Thu, Jan 7, 2016 at

Re: Can't submit job to stand alone cluster

then we should update them. I have filed SPARK-12565 to track this. Please let me know if there's anything else I can help clarify. Cheers,-Andrew 2015-12-29 13:07 GMT-08:00 Annabel Melongo <melongo_anna...@yahoo.com>: Andrew, Now I see where the confusion lays. Standalone clus

Re: Can't submit job to stand alone cluster

Greg, The confusion here is the expression "standalone cluster mode". Either it's stand-alone or it's cluster mode but it can't be both.  With this in mind, here's how jars are uploaded:    1. Spark Stand-alone mode: client and driver run on the same machine; use --packages option to submit a

Re: Can't submit job to stand alone cluster

nt mode. 2015-12-29 11:32 GMT-08:00 Annabel Melongo <melongo_anna...@yahoo.com>: Greg, The confusion here is the expression "standalone cluster mode". Either it's stand-alone or it's cluster mode but it can't be both.  With this in mind, here's how jars are uploaded:    1.

Re: Can't submit job to stand alone cluster

. Please correct me, if I'm wrong. Thanks   On Tuesday, December 29, 2015 2:54 PM, Andrew Or <and...@databricks.com> wrote: http://spark.apache.org/docs/latest/spark-standalone.html#launching-spark-applications 2015-12-29 11:48 GMT-08:00 Annabel Melongo <melongo_anna...@

Re: DataFrame Vs RDDs ... Which one to use When ?

Additionally, if you already have some legal sql statements to process said data, instead of reinventing the wheel using rdd's functions, you can speed up implementation by using dataframes along with these existing sql statements. On Monday, December 28, 2015 5:37 PM, Darren Govoni

Re: Stuck with DataFrame df.select("select * from table");

Jean, Try this:df.select("""select * from tmptable where x1 = '3.0'""").show(); Note: you have to use 3 double quotes as marked On Friday, December 25, 2015 11:30 AM, Eugene Morozov wrote: Thanks for the comments, although the issue is not in limit()

Re: pre-install 3-party Python package on spark cluster

When you run spark submit in either client or cluster mode, you can either use the options --packages or -jars to automatically copy your packages to the worker machines. Thanks On Monday, January 11, 2016 12:52 PM, Andy Davidson wrote: I use 

Re: trouble using eclipse to view spark source code

Andy, This has nothing to do with Spark but I guess you don't have the proper Scala version. The version you're currently running doesn't recognize a method in Scala ArrayOps, namely:          scala.collection.mutable.ArrayOps.$colon$plus On Monday, January 18, 2016 7:53 PM, Andy Davidson

Re: DataFrame to read json and include raw Json in DataFrame

Richard, Below documentation will show you how to create a sparkSession and how to programmatically load data: Spark SQL and DataFrames - Spark 2.1.0 Documentation | | | Spark SQL and DataFrames - Spark 2.1.0 Documentation | | | On Thursday, December 29, 2016 5:16 PM,

Re: DataFrame to read json and include raw Json in DataFrame

of my dataframe. On Friday, December 30, 2016 10:23 AM, Annabel Melongo <melongo_anna...@yahoo.com.INVALID> wrote: Richard, Below documentation will show you how to create a sparkSession and how to programmatically load data: Spark SQL and DataFrames - Spark 2.1.0 Documentation

unsubscribe

unsubscribe

Unsubscribe

Please remove me from the mailing list

Re: Unsubscribe

ow you unsubscribe.  See here for instructions:  https://gist.github.com/jeff303/ba1906bb7bcb2f2501528a8bb1521b8e On Wed, Aug 26, 2020, 4:22 PM Annabel Melongo wrote: Please remove me from the mailing list This email contains confidential information of and is the copyright of Infomedia. I