Ok thanks. It seems that --jars is not behaving as expected - getting class
not found for even the most simple object from my lib. But anyways, I have
to do at least a filter transformation before collecting the HBaseRDD into
R so will have to go the route of using scala spark shell to transform and
collect and save into local filesystem and the visualise the file with R
until custom RDD transformations are exposed in the SparkR API.

On 13 July 2015 at 10:27, Sun, Rui <rui....@intel.com> wrote:

> Hi, Michal,
>
> SparkR comes with a JVM backend that supports Java object instantiation,
> calling Java instance and static methods from R side. As defined in
> https://github.com/apache/spark/blob/master/R/pkg/R/backend.R,
> newJObject() is to create an instance of a Java class;
> callJMethod() is to call an instance method of a Java object;
> callJStatic() is to call a static method of a Java class.
>
> If the thing is as simple as data visualization, you can use the above
> low-level functions to create an instance of your HBASE RDD in JVM side,
> collect the data to R side, and visualize it.
>
> However, if you want to do HBASE RDD transformation and HBASE table
> update, things are quite complex now. SparkR supports majority of RDD API
> (though not exposed publicly in 1.4 release) allowing transformation
> functions in R code, but currently it only supports RDD source from text
> files and SparkR Data Frames, so your HBASE RDDs can't be used by SparkR
> RDD API for further processing.
>
> You can use --jars to include your scala library to be accessed by the JVM
> backend.
>
> ________________________________
> From: Michal Haris [michal.ha...@visualdna.com]
> Sent: Sunday, July 12, 2015 6:39 PM
> To: user@spark.apache.org
> Subject: Including additional scala libraries in sparkR
>
> I have spark program with a custom optimised rdd for hbase scans and
> updates. I have a small library of objects in scala to support efficient
> serialisation, partitioning etc. I would like to use R as an analysis and
> visualisation front-end. I have tried to use rJava (i.e. not using sparkR)
> and I got as far as initialising the spark context but I have encountered
> problems with hbase dependencies (HBaseConfiguration : Unsupported
> major.minor version 51.0) so tried sparkR but I can't figure out how to
> make my custom scala classes available to sparkR other than re-implementing
> them in R. Is there a way to include and invoke additional scala objects
> and RDDs within sparkR shell/job ? Something similar to additional jars and
> init script in normal spark submit/shell..
>
> --
> Michal Haris
> Technical Architect
> direct line: +44 (0) 207 749 0229
> www.visualdna.com<http://www.visualdna.com> | t: +44 (0) 207 734 7033
> 31 Old Nichol Street
> London
> E2 7HR
>



-- 
Michal Haris
Technical Architect
direct line: +44 (0) 207 749 0229
www.visualdna.com | t: +44 (0) 207 734 7033
31 Old Nichol Street
London
E2 7HR

Reply via email to