Re: Spark SQL Tables on top of HBase Tables

2016-09-05 Thread Yan Zhou
There is a HSpark project, https://github.com/yzhou2001/HSpark, providing
native and fast access to HBase.
Currently it only supports Spark 1.4, but any suggestions and contributions
are more than welcome.

 Try it out to find its speedups!

On Sat, Sep 3, 2016 at 12:57 PM, Mich Talebzadeh 
wrote:

> Mine is Hbase-0.98,
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 3 September 2016 at 20:51, Benjamin Kim  wrote:
>
>> I’m using Spark 1.6 and HBase 1.2. Have you got it to work using these
>> versions?
>>
>> On Sep 3, 2016, at 12:49 PM, Mich Talebzadeh 
>> wrote:
>>
>> I am trying to find a solution for this
>>
>> ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class
>> org.apache.hadoop.hive.hbase.HBaseSerDe not found
>>
>> I am using Spark 2 and Hive 2!
>>
>> HTH
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 3 September 2016 at 20:31, Benjamin Kim  wrote:
>>
>>> Mich,
>>>
>>> I’m in the same boat. We can use Hive but not Spark.
>>>
>>> Cheers,
>>> Ben
>>>
>>> On Sep 2, 2016, at 3:37 PM, Mich Talebzadeh 
>>> wrote:
>>>
>>> Hi,
>>>
>>> You can create Hive external  tables on top of existing Hbase table
>>> using the property
>>>
>>> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>>>
>>> Example
>>>
>>> hive> show create table hbase_table;
>>> OK
>>> CREATE TABLE `hbase_table`(
>>>   `key` int COMMENT '',
>>>   `value1` string COMMENT '',
>>>   `value2` int COMMENT '',
>>>   `value3` int COMMENT '')
>>> ROW FORMAT SERDE
>>>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
>>> STORED BY
>>>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>>> WITH SERDEPROPERTIES (
>>>   'hbase.columns.mapping'=':key,a:b,a:c,d:e',
>>>   'serialization.format'='1')
>>> TBLPROPERTIES (
>>>   'transient_lastDdlTime'='1472370939')
>>>
>>>  Then try to access this Hive table from Spark which is giving me grief
>>> at the moment :(
>>>
>>> scala> HiveContext.sql("use test")
>>> res9: org.apache.spark.sql.DataFrame = []
>>> scala> val hbase_table= spark.table("hbase_table")
>>> 16/09/02 23:31:07 ERROR log: error in initSerDe:
>>> java.lang.ClassNotFoundException Class 
>>> org.apache.hadoop.hive.hbase.HBaseSerDe
>>> not found
>>>
>>> HTH
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 2 September 2016 at 23:08, KhajaAsmath Mohammed <
>>> mdkhajaasm...@gmail.com> wrote:
>>>
 Hi Kim,

 I am also looking for same information. Just got the same requirement
 today.

 Thanks,
 Asmath

 On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim 
 wrote:

> I was wondering if anyone has tried to create Spark SQL tables on top
> of HBase tables so that data in HBase can be accessed using Spark
> Thriftserver with SQL statements? This is similar what can be done using
> Hive.
>
> Thanks,
> Ben
>
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

>>>
>>>
>>
>>
>


Re: Spark SQL Tables on top of HBase Tables

2016-09-03 Thread Mich Talebzadeh
Mine is Hbase-0.98,

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 3 September 2016 at 20:51, Benjamin Kim  wrote:

> I’m using Spark 1.6 and HBase 1.2. Have you got it to work using these
> versions?
>
> On Sep 3, 2016, at 12:49 PM, Mich Talebzadeh 
> wrote:
>
> I am trying to find a solution for this
>
> ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class
> org.apache.hadoop.hive.hbase.HBaseSerDe not found
>
> I am using Spark 2 and Hive 2!
>
> HTH
>
>
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 3 September 2016 at 20:31, Benjamin Kim  wrote:
>
>> Mich,
>>
>> I’m in the same boat. We can use Hive but not Spark.
>>
>> Cheers,
>> Ben
>>
>> On Sep 2, 2016, at 3:37 PM, Mich Talebzadeh 
>> wrote:
>>
>> Hi,
>>
>> You can create Hive external  tables on top of existing Hbase table using
>> the property
>>
>> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>>
>> Example
>>
>> hive> show create table hbase_table;
>> OK
>> CREATE TABLE `hbase_table`(
>>   `key` int COMMENT '',
>>   `value1` string COMMENT '',
>>   `value2` int COMMENT '',
>>   `value3` int COMMENT '')
>> ROW FORMAT SERDE
>>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
>> STORED BY
>>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>> WITH SERDEPROPERTIES (
>>   'hbase.columns.mapping'=':key,a:b,a:c,d:e',
>>   'serialization.format'='1')
>> TBLPROPERTIES (
>>   'transient_lastDdlTime'='1472370939')
>>
>>  Then try to access this Hive table from Spark which is giving me grief
>> at the moment :(
>>
>> scala> HiveContext.sql("use test")
>> res9: org.apache.spark.sql.DataFrame = []
>> scala> val hbase_table= spark.table("hbase_table")
>> 16/09/02 23:31:07 ERROR log: error in initSerDe:
>> java.lang.ClassNotFoundException Class 
>> org.apache.hadoop.hive.hbase.HBaseSerDe
>> not found
>>
>> HTH
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 2 September 2016 at 23:08, KhajaAsmath Mohammed <
>> mdkhajaasm...@gmail.com> wrote:
>>
>>> Hi Kim,
>>>
>>> I am also looking for same information. Just got the same requirement
>>> today.
>>>
>>> Thanks,
>>> Asmath
>>>
>>> On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim  wrote:
>>>
 I was wondering if anyone has tried to create Spark SQL tables on top
 of HBase tables so that data in HBase can be accessed using Spark
 Thriftserver with SQL statements? This is similar what can be done using
 Hive.

 Thanks,
 Ben


 -
 To unsubscribe e-mail: user-unsubscr...@spark.apache.org


>>>
>>
>>
>
>


Re: Spark SQL Tables on top of HBase Tables

2016-09-03 Thread Benjamin Kim
I’m using Spark 1.6 and HBase 1.2. Have you got it to work using these versions?

> On Sep 3, 2016, at 12:49 PM, Mich Talebzadeh  
> wrote:
> 
> I am trying to find a solution for this
> 
> ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class 
> org.apache.hadoop.hive.hbase.HBaseSerDe not found
> 
> I am using Spark 2 and Hive 2!
> 
> HTH
> 
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> On 3 September 2016 at 20:31, Benjamin Kim  > wrote:
> Mich,
> 
> I’m in the same boat. We can use Hive but not Spark.
> 
> Cheers,
> Ben
> 
>> On Sep 2, 2016, at 3:37 PM, Mich Talebzadeh > > wrote:
>> 
>> Hi,
>> 
>> You can create Hive external  tables on top of existing Hbase table using 
>> the property
>> 
>> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>> 
>> Example
>> 
>> hive> show create table hbase_table;
>> OK
>> CREATE TABLE `hbase_table`(
>>   `key` int COMMENT '',
>>   `value1` string COMMENT '',
>>   `value2` int COMMENT '',
>>   `value3` int COMMENT '')
>> ROW FORMAT SERDE
>>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
>> STORED BY
>>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>> WITH SERDEPROPERTIES (
>>   'hbase.columns.mapping'=':key,a:b,a:c,d:e',
>>   'serialization.format'='1')
>> TBLPROPERTIES (
>>   'transient_lastDdlTime'='1472370939')
>> 
>>  Then try to access this Hive table from Spark which is giving me grief at 
>> the moment :(
>> 
>> scala> HiveContext.sql("use test")
>> res9: org.apache.spark.sql.DataFrame = []
>> scala> val hbase_table= spark.table("hbase_table")
>> 16/09/02 23:31:07 ERROR log: error in initSerDe: 
>> java.lang.ClassNotFoundException Class 
>> org.apache.hadoop.hive.hbase.HBaseSerDe not found
>> 
>> HTH
>> 
>> 
>> Dr Mich Talebzadeh
>>  
>> LinkedIn  
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>  
>> 
>>  
>> http://talebzadehmich.wordpress.com 
>> 
>> Disclaimer: Use it at your own risk. Any and all responsibility for any 
>> loss, damage or destruction of data or any other property which may arise 
>> from relying on this email's technical content is explicitly disclaimed. The 
>> author will in no case be liable for any monetary damages arising from such 
>> loss, damage or destruction.
>>  
>> 
>> On 2 September 2016 at 23:08, KhajaAsmath Mohammed > > wrote:
>> Hi Kim,
>> 
>> I am also looking for same information. Just got the same requirement today.
>> 
>> Thanks,
>> Asmath
>> 
>> On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim > > wrote:
>> I was wondering if anyone has tried to create Spark SQL tables on top of 
>> HBase tables so that data in HBase can be accessed using Spark Thriftserver 
>> with SQL statements? This is similar what can be done using Hive.
>> 
>> Thanks,
>> Ben
>> 
>> 
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org 
>> 
>> 
>> 
>> 
> 
> 



Re: Spark SQL Tables on top of HBase Tables

2016-09-03 Thread Mich Talebzadeh
I am trying to find a solution for this

ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class
org.apache.hadoop.hive.hbase.HBaseSerDe not found

I am using Spark 2 and Hive 2!

HTH



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 3 September 2016 at 20:31, Benjamin Kim  wrote:

> Mich,
>
> I’m in the same boat. We can use Hive but not Spark.
>
> Cheers,
> Ben
>
> On Sep 2, 2016, at 3:37 PM, Mich Talebzadeh 
> wrote:
>
> Hi,
>
> You can create Hive external  tables on top of existing Hbase table using
> the property
>
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>
> Example
>
> hive> show create table hbase_table;
> OK
> CREATE TABLE `hbase_table`(
>   `key` int COMMENT '',
>   `value1` string COMMENT '',
>   `value2` int COMMENT '',
>   `value3` int COMMENT '')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
>   'hbase.columns.mapping'=':key,a:b,a:c,d:e',
>   'serialization.format'='1')
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1472370939')
>
>  Then try to access this Hive table from Spark which is giving me grief at
> the moment :(
>
> scala> HiveContext.sql("use test")
> res9: org.apache.spark.sql.DataFrame = []
> scala> val hbase_table= spark.table("hbase_table")
> 16/09/02 23:31:07 ERROR log: error in initSerDe: 
> java.lang.ClassNotFoundException
> Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 2 September 2016 at 23:08, KhajaAsmath Mohammed <
> mdkhajaasm...@gmail.com> wrote:
>
>> Hi Kim,
>>
>> I am also looking for same information. Just got the same requirement
>> today.
>>
>> Thanks,
>> Asmath
>>
>> On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim  wrote:
>>
>>> I was wondering if anyone has tried to create Spark SQL tables on top of
>>> HBase tables so that data in HBase can be accessed using Spark Thriftserver
>>> with SQL statements? This is similar what can be done using Hive.
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> -
>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>
>>>
>>
>
>


Re: Spark SQL Tables on top of HBase Tables

2016-09-03 Thread Benjamin Kim
Mich,

I’m in the same boat. We can use Hive but not Spark.

Cheers,
Ben

> On Sep 2, 2016, at 3:37 PM, Mich Talebzadeh  wrote:
> 
> Hi,
> 
> You can create Hive external  tables on top of existing Hbase table using the 
> property
> 
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> 
> Example
> 
> hive> show create table hbase_table;
> OK
> CREATE TABLE `hbase_table`(
>   `key` int COMMENT '',
>   `value1` string COMMENT '',
>   `value2` int COMMENT '',
>   `value3` int COMMENT '')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
>   'hbase.columns.mapping'=':key,a:b,a:c,d:e',
>   'serialization.format'='1')
> TBLPROPERTIES (
>   'transient_lastDdlTime'='1472370939')
> 
>  Then try to access this Hive table from Spark which is giving me grief at 
> the moment :(
> 
> scala> HiveContext.sql("use test")
> res9: org.apache.spark.sql.DataFrame = []
> scala> val hbase_table= spark.table("hbase_table")
> 16/09/02 23:31:07 ERROR log: error in initSerDe: 
> java.lang.ClassNotFoundException Class 
> org.apache.hadoop.hive.hbase.HBaseSerDe not found
> 
> HTH
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> On 2 September 2016 at 23:08, KhajaAsmath Mohammed  > wrote:
> Hi Kim,
> 
> I am also looking for same information. Just got the same requirement today.
> 
> Thanks,
> Asmath
> 
> On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim  > wrote:
> I was wondering if anyone has tried to create Spark SQL tables on top of 
> HBase tables so that data in HBase can be accessed using Spark Thriftserver 
> with SQL statements? This is similar what can be done using Hive.
> 
> Thanks,
> Ben
> 
> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org 
> 
> 
> 
> 



Re: Spark SQL Tables on top of HBase Tables

2016-09-02 Thread Mich Talebzadeh
Hi,

You can create Hive external  tables on top of existing Hbase table using
the property

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

Example

hive> show create table hbase_table;
OK
CREATE TABLE `hbase_table`(
  `key` int COMMENT '',
  `value1` string COMMENT '',
  `value2` int COMMENT '',
  `value3` int COMMENT '')
ROW FORMAT SERDE
  'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY
  'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
  'hbase.columns.mapping'=':key,a:b,a:c,d:e',
  'serialization.format'='1')
TBLPROPERTIES (
  'transient_lastDdlTime'='1472370939')

 Then try to access this Hive table from Spark which is giving me grief at
the moment :(

scala> HiveContext.sql("use test")
res9: org.apache.spark.sql.DataFrame = []
scala> val hbase_table= spark.table("hbase_table")
16/09/02 23:31:07 ERROR log: error in initSerDe:
java.lang.ClassNotFoundException Class
org.apache.hadoop.hive.hbase.HBaseSerDe not found

HTH


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 2 September 2016 at 23:08, KhajaAsmath Mohammed 
wrote:

> Hi Kim,
>
> I am also looking for same information. Just got the same requirement
> today.
>
> Thanks,
> Asmath
>
> On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim  wrote:
>
>> I was wondering if anyone has tried to create Spark SQL tables on top of
>> HBase tables so that data in HBase can be accessed using Spark Thriftserver
>> with SQL statements? This is similar what can be done using Hive.
>>
>> Thanks,
>> Ben
>>
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>


Re: Spark SQL Tables on top of HBase Tables

2016-09-02 Thread ayan guha
You can either read hbase in rdd and then turn it to a df or expose hbase
tables using hive and read from hive or use phoenix
On 3 Sep 2016 08:08, "KhajaAsmath Mohammed"  wrote:

> Hi Kim,
>
> I am also looking for same information. Just got the same requirement
> today.
>
> Thanks,
> Asmath
>
> On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim  wrote:
>
>> I was wondering if anyone has tried to create Spark SQL tables on top of
>> HBase tables so that data in HBase can be accessed using Spark Thriftserver
>> with SQL statements? This is similar what can be done using Hive.
>>
>> Thanks,
>> Ben
>>
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>


Re: Spark SQL Tables on top of HBase Tables

2016-09-02 Thread KhajaAsmath Mohammed
Hi Kim,

I am also looking for same information. Just got the same requirement today.

Thanks,
Asmath

On Fri, Sep 2, 2016 at 4:46 PM, Benjamin Kim  wrote:

> I was wondering if anyone has tried to create Spark SQL tables on top of
> HBase tables so that data in HBase can be accessed using Spark Thriftserver
> with SQL statements? This is similar what can be done using Hive.
>
> Thanks,
> Ben
>
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>