repartition vs partitionby

Hi folks I need to reparation large set of data around(300G) as i see some portions have large data(data skew) i have pairRDDs [({},{}),({},{}),({},{})] what is the best way to solve the the problem - To unsubscribe, e-mail:

Build Failure

hi I tried to build latest master branch of spark build/mvn -DskipTests clean package Reactor Summary: [INFO] [INFO] Spark Project Parent POM ... SUCCESS [03:46 min] [INFO] Spark Project Test Tags SUCCESS [01:02 min] [INFO] Spark Project

API to run spark Jobs

Hi Folks How i can submit my spark app(python) to the cluster without using spark-submit, actually i need to invoke jobs from UI - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail:

Re: API to run spark Jobs

p > distros might, for example EMR in AWS has a job submit UI. > > Spark submit just calls a REST api, you could build any UI you want on top of > that... > > > On Tue, Oct 6, 2015 at 9:37 AM, shahid qadri <shahidashr...@icloud.com > <mailto:shahidashr...@icloud.com>

Custom Partitioner

Hi Sparkians How can we create a customer partition in pyspark - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: How to effieciently write sorted neighborhood in pyspark

> On Aug 25, 2015, at 10:43 PM, shahid qadri <shahidashr...@icloud.com> wrote: > > Any resources on this > >> On Aug 25, 2015, at 3:15 PM, shahid qadri <shahidashr...@icloud.com> wrote: >> >> I would like to implement sorted neighborhood approach i

Re: How to effieciently write sorted neighborhood in pyspark

Any resources on this On Aug 25, 2015, at 3:15 PM, shahid qadri shahidashr...@icloud.com wrote: I would like to implement sorted neighborhood approach in spark, what is the best way to write that in pyspark

How to effieciently write sorted neighborhood in pyspark

I would like to implement sorted neighborhood approach in spark, what is the best way to write that in pyspark. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail:

disabling dynamic date time formatting in python api or globally

guys getting this error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, u'MapperParsingException[failed to parse [SOURCES.DATE_COMP]]; nested: MapperParsingException[failed to parse date field [--], tried

Re: disabling dynamic date time formatting in python api or globally

/ @scrutmydocs Le 15 févr. 2015 à 14:44, Shahid Qadri sha...@trialx.com javascript: a écrit : guys getting this error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) RequestError: TransportError(400, u'MapperParsingException[failed