Re: Mesos Spark Fine Grained Execution - CPU count

We will be interested by the results if you give a try to Dynamic allocation with mesos ! - Mail Original - De: "Michael Gummelt" À: "Sumit Chawla" Cc: u...@mesos.apache.org, d...@mesos.apache.org, "User" ,

Re: Mesos Spark Fine Grained Execution - CPU count

I think that what you are looking for is Dynamic resource allocation: http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your

Re: [SQL] Reading from hive table is listing all files in S3

) STORED AS PARQUET LOCATION 's3a://bucketname/' Do you think the definition can be the source of the problem ? Thanks - Mail Original - De: "Mich Talebzadeh" <mich.talebza...@gmail.com> À: "Mehdi Meziane" <mehdi.mezi...@ldmobile.net> Cc: "user @s

[SQL] Reading from hive table is listing all files in S3

Hi all, We have a hive table stored in S3 and registered in a hive metastore. This table is partitionned with a key "day". So we access this table through the spark dataframe API as : sqlContext.read() .table("tablename) .where(col("day").between("2016-08-01","2016-08-02")) When

Spark ML - Java implementation of custom Transformer

Hi all, We have some problems while implementing custom Transformers in JAVA (SPARK 1.6.1). We do override the method copy, but it crashes with an AbstractMethodError. If we extends the UnaryTransformer, and do not override the copy method, it works without any error. We tried to