Re: LICENSE and NOTICE file content

2018-06-21 Thread Justin Mclean
Hi, The PR was just for the LICENSE and NOTICE you still may want to look at the jar issue. Thanks, Justin - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: LICENSE and NOTICE file content

2018-06-21 Thread Justin Mclean
Hi, Here you go [1]. That is however only for the source, re the connivance binary (which I’ve not checked) the LICENSE and NOTICE is very likely to be different. It turns out the Android project does have a NOTICE file and that had an effect on the spark one. Thanks, Justin 1.

Re: LICENSE and NOTICE file content

2018-06-21 Thread Reynold Xin
Thanks Justin. Can you submit a pull request? On Thu, Jun 21, 2018 at 8:10 PM Justin Mclean wrote: > Hi, > > We’ve recently had a number of incubating projects copy your LICENSE and > NOTICE files as they see Spark as a popular project and they are a little > sad when the IPMC votes -1 on their

LICENSE and NOTICE file content

2018-06-21 Thread Justin Mclean
Hi, We’ve recently had a number of incubating projects copy your LICENSE and NOTICE files as they see Spark as a popular project and they are a little sad when the IPMC votes -1 on their releases. Now I'm not on your PMC, don’t know your projects history and there may be valid reasons for the

Re: RepartitionByKey Behavior

2018-06-21 Thread Jungtaek Lim
It is not possible because the cardinality of the partitioning key is non-deterministic, while partition count should be fixed. There's a chance that cardinality > partition count and then the system can't ensure the requirement. Thanks, Jungtaek Lim (HeartSaVioR) 2018년 6월 22일 (금) 오전 8:55,

Re: RepartitionByKey Behavior

2018-06-21 Thread Chawla,Sumit
Based on code read it looks like Spark does modulo of key for partition. Keys of c and b end up pointing to same value. Whats the best partitioning scheme to deal with this? Regards Sumit Chawla On Thu, Jun 21, 2018 at 4:51 PM, Chawla,Sumit wrote: > Hi > > I have been trying to this simple

RepartitionByKey Behavior

2018-06-21 Thread Chawla,Sumit
Hi I have been trying to this simple operation. I want to land all values with one key in same partition, and not have any different key in the same partition. Is this possible? I am getting b and c always getting mixed up in the same partition. rdd = sc.parallelize([('a', 5), ('d', 8),

Re: Scala 2.12 support

2018-06-21 Thread Stavros Kontopoulos
Hi all, Scala team @Lightbend (Lukas, Adriaan, Jason) and I, have worked for a couple of days now on this. We have captured current status and possible solutions for the remaining two issues here: https://docs.google.com/document/d/ 1fbkjEL878witxVQpOCbjlvOvadHtVjYXeB-2mgzDTvk Please review the

insertinto command fails if the location of the table is file

2018-06-21 Thread sandeep_katta
let's say I create table using below command create table csvTable . using CSV options (path "/user/data/customer.csv"); Crate Table command executes successfully irrespective of the presence of file(/user/data/customer.csv) If I try to insert some rows into this table it fails with below