Stopping StreamingContext

4-4153-a4c1-059676861581/userFiles-8a970eec-da41-442b-9ccf-1621b9e9e045 Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [emailsignature]

RE: [PySpark] - running processes and computing time

ow what happens here? The time difference is too big for it to be networking right? From: Sudev A C [mailto:sudev...@go-mmt.com] Sent: Monday, July 3, 2017 7:48 PM To: Sidney Feiner <sidney.fei...@startapp.com>; user@spark.apache.org Subject: Re: [PySpark] - running processes You might want to

[PySpark] - running processes

? And if they do, how can I make sure that my graph object will really only be initiated once? Thanks :) Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [emailsignature]

RE: Message getting lost in Kafka + Spark Streaming

Are you sure that every message gets processed? It could be that some messages failed passing the decoder. And during the processing, are you maybe putting the events into a map? That way, events with the same key could override each other and that way you'll have less final events.

RE: [Spark Streaming] - Killing application from within code

dnesday, May 3, 2017 10:25 PM To: Sidney Feiner <sidney.fei...@startapp.com> Cc: user@spark.apache.org Subject: Re: [Spark Streaming] - Killing application from within code There isnt a clean programmatic way to kill the application running in the driver from the executor. You will ha

[Spark Streaming] - Killing application from within code

nd no matter what way I try, the driver stays alive. Is there a way to programmatically kill the application from within one of the workers? Thanks a lot :) Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [emailsignature]

RE: How to run a spark on Pycharm

Hey, I once found an article about that: https://mengdong.github.io/2016/08/08/fully-armed-pyspark-with-ipython-and-jupyter/ And I once managed to set it up on Pycharm as well. What I had to do was to add /path/to/spark to a system variable called "PYTHTONPATH". Try that one, it might help ☺

RE: pyspark in intellij

) · To the environment variable, add the same path as above to the PATH variable Hope these work ☺ Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [emailsignature] From: Stephen Boesch [mailto:java...@gmail.com] Sent: Sunday, February 26, 2017 3:56 AM To: user <user@spark.apache.

RE: How to query a query with not contain, not start_with, not end_with condition effective?

Chanh wants to return user_id's that don't have any record with a url containing "sell". Without a subquery/join, it can only filter per record without knowing about the rest of the user_id's record Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp

RE: Jars directory in Spark 2.0

Ok, good to know ☺ Shading every spark app it is then… Thanks! Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/> From: Marcelo Vanzin [mailto:van...@cloudera.com] Sent: Wednesday, February 1, 2017 7:41 PM To: Sidney

RE: Jars directory in Spark 2.0

Is this done on purpose? Because it really makes it hard to deploy applications. Is there a reason they didn't shade the jars they use to begin with? Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/> From: Koert K

Jars directory in Spark 2.0

park which means I can't blindly replace them? Thanks :) Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/> [Meet Us at] <http://www.startapp.com/press/#events_press>

RE: [PySpark 2.1.0] - SparkContext not properly initialized by SparkConf

Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/> From: Sidney Feiner Sent: Thursday, January 26, 2017 9:26 AM To: user@spark.apache.org Subject: [PySpark 2.1.0] - SparkContext not properly initialized by SparkConf Hey, I'm p

[PySpark 2.1.0] - SparkContext not properly initialized by SparkConf

/path/to/config/file Does anybody know why my SparkContext doesn't get initialized with my SparkConf? Thanks :) Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/> [Meet Us at] <http://www.startapp.com/press/#events_press>

Re: Spark-submit: where do --files go?

Every executor creates a directory with your submitted files and you can access every file's absolute path them with the following: val fullFilePath = SparkFiles.get(fileName) On Jan 19, 2017 19:35, jeff saremi wrote: I'd like to know how -- From within Java/spark --

RE: [PySpark - 1.6] - Avoid object serialization

lp one day ☺ Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/> From: Holden Karau [mailto:hol...@pigscanfly.ca] Sent: Thursday, December 29, 2016 8:54 PM To: Chawla,Sumit <sumitkcha...@gmail.com>; Eike von Seggern <

[PySpark - 1.6] - Avoid object serialization

ect created once for each executor and then it could avoid the serialization process, gain time and actually have a working cache system? Thanks in advance :) Sidney Feiner / SW Developer M: +972.528197720 / Skype: sidney.feiner.startapp [StartApp]<http://www.startapp.com/>