Re: Can not control bucket files number if it was speficed

I didn't follow all of this thread, but if you want to have exactly one bucket-output-file per RDD-partition, you have to repartition (shuffle) your data on the bucket-key. If you don't repartition (shuffle), you may have records with different bucket-keys in the same RDD-partition, leading to

Re: How to make the result of sortByKey distributed evenly?

Your data has only two keys, and basically all values are assigned to only one of them. There is no better way to distribute the keys, than the one Spark executes. What you have to do is to use different keys to sort and range-partition on. Try to invoke sortBy() on a non-pair-RDD. This will

Re: Splitting columns from a text file

Ask yourself how to access the third element in an array in Scala. Am 05.09.2016 um 16:14 schrieb Ashok Kumar: Hi, I want to filter them for values. This is what is in array 74,20160905-133143,98.11218069128827594148 I want to filter anything > 50.0 in the third column Thanks On

Re: any idea what this error could be?

_2.10' version: >'2.0.0' >on the executor side I don't know what jars are being used but I have >installed >using this zip filespark-2.0.0-bin-hadoop2.7.tgz > > > > > > >On Sat, Sep 3, 2016 4:20 AM, Fridtjof Sander >fridtjof.san...@googlemail.com >wrote: &g

Re: any idea what this error could be?

There is an InvalidClassException complaining about non-matching serialVersionUIDs. Shouldn't that be caused by different jars on executors and driver? Am 03.09.2016 1:04 nachm. schrieb "Tal Grynbaum" : > My guess is that you're running out of memory somewhere. Try to

Re: Grouping on bucketed and sorted columns

available in 2.0.0. I would highly appreciate some feedback to my thoughts and questions Am 31.08.2016 um 14:45 schrieb Fridtjof Sander: Hi Spark users, I'm currently investigating spark's bucketing and partitioning capabilities and I have some questions: Let /T/ be a table that is bucketed

Grouping on bucketed and sorted columns

Hi Spark users, I'm currently investigating spark's bucketing and partitioning capabilities and I have some questions: Let /T/ be a table that is bucketed and sorted by /T.id/ and partitioned by /T.date/. Before persisting, /T/ has been repartitioned by /T.id/ to get only one file per

Re: Isotonic Regression, run method overloaded Error

sion().setIsotonic(true) val model = ir.fit(dataset) val predictions = model .transform(dataset) .select("prediction").rdd.map { case Row(pred) => pred }.collect() assert(predictions === Array(1, 2, 2, 2, 6, 16.5, 16.5, 17, 18)) | Thanks Yanbo 2016-07-11 6:14 GMT-07:00 Fridtjof Sand

Re: Isotonic Regression, run method overloaded Error

Hi Swaroop, from my understanding, Isotonic Regression is currently limited to data with 1 feature plus weight and label. Also the entire data is required to fit into memory of a single machine. I did some work on the latter issue but discontinued the project, because I felt no one really