unsubscribe

[VJ Anand]

-- 
*VJ Anand*
*Founder *
*Sankia*
vjan...@sankia.com
925-640-1340
www.sankia.com

*Confidentiality Notice*: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies
of the original message

No suitable Constructor found while compiling

[VJ Anand]

I am trying to extend RDD in java, and when I call the parent constructor,
it gives the error: no suitable constructor found for RDD (SparkContext,
Seq, ClassTag).

 Here is the snippet of the code:

class QueryShard extends RDD {



sc (sc, (Seq)new ArrayBuffer,
ClassTag$.MODULE$.apply(Tuple.class);

}

Compiler error: no suitable constructor found for RDD (SparkContext, Seq,
ClassTag)

Any thoughts? pointers..

Thanks
VJ

Re: Join Order Optimization

[VJ Anand]

Hi - Is there a design document for those operations that have been
implemented in 1.4.0? if so,where can I find them
-VJ

On Sun, Oct 11, 2015 at 7:27 PM, Cheng, Hao <hao.ch...@intel.com> wrote:

> Yes, I think the SPARK-2211 should be the right place to follow the CBO
> stuff, but probably that will not happen right away.
>
>
>
> The jira issue introduce the statistic info can be found at:
>
> https://issues.apache.org/jira/browse/SPARK-2393
>
>
>
> Hao
>
>
>
> *From:* Raajay [mailto:raaja...@gmail.com]
> *Sent:* Monday, October 12, 2015 10:17 AM
> *To:* Cheng, Hao
> *Cc:* user@spark.apache.org
> *Subject:* Re: Join Order Optimization
>
>
>
> Hi Cheng,
>
> Could you point me to the JIRA that introduced this change ?
>
>
> Also, is this SPARK-2211 the right issue to follow for cost-based
> optimization?
>
> Thanks
>
> Raajay
>
>
>
>
>
> On Sun, Oct 11, 2015 at 7:57 PM, Cheng, Hao <hao.ch...@intel.com> wrote:
>
> Spark SQL supports very basic join reordering optimization, based on the
> raw table data size, this was added couple major releases back.
>
>
>
> And the “EXPLAIN EXTENDED query” command is a very informative tool to
> verify whether the optimization taking effect.
>
>
>
> *From:* Raajay [mailto:raaja...@gmail.com]
> *Sent:* Sunday, October 11, 2015 9:22 AM
> *To:* user@spark.apache.org
> *Subject:* Join Order Optimization
>
>
>
> Hello,
>
> Does Spark-SQL support join order optimization as of the 1.5.1 release ?
> From the release notes, I did not see support for this feature, but figured
> will ask the users-list to be sure.
>
> Thanks
>
> Raajay
>
>
>



-- 
*VJ Anand*
*Founder *
*Sankia*
vjan...@sankia.com
925-640-1340
www.sankia.com

*Confidentiality Notice*: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies
of the original message

Custom RDD for Proprietary MPP database

[VJ Anand]

Hi,

I need to build a RDD that supports a custom built Database (Which is
sharded) across several nodes. I need to build an RDD that can support and
provide the partitions specific to this database.
I would like to do this in Java - I see there are JavaRDD, and other
specific RDD available - my question, is if I subclass or extend this RDD -
can I override the getPartitions, and other methods? Or is there any other
alternative? Any help or pointers much appreciated

Thanks
VJ