Re: Save DataFrame to HBase
Please note:
The name of hbase table is specified in:
def writeCatalog = s"""{
|"table":{"namespace":"default", "name":"table1"},
not by the:
HBaseTableCatalog.newTable -> "5"
FYI
On Tue, May 10, 2016 at 3:11 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> I think so.
>
> Please refer to the table population tests in (master branch):
>
> hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/DefaultSourceSuite.scala
>
> Cheers
>
> On Tue, May 10, 2016 at 2:53 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>
>> Ted,
>>
>> Will the hbase-spark module allow for creating tables in Spark SQL that
>> reference the hbase tables underneath? In this way, users can query using
>> just SQL.
>>
>> Thanks,
>> Ben
>>
>> On Apr 28, 2016, at 3:09 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>> Hbase 2.0 release likely would come after Spark 2.0 release.
>>
>> There're other features being developed in hbase 2.0
>> I am not sure when hbase 2.0 would be released.
>>
>> The refguide is incomplete.
>> Zhan has assigned the doc JIRA to himself. The documentation would be
>> done after fixing bugs in hbase-spark module.
>>
>> Cheers
>>
>> On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>>
>> Hi Ted,
>>
>> Do you know when the release will be? I also see some documentation for
>> usage of the hbase-spark module at the hbase website. But, I don’t see an
>> example on how to save data. There is only one for reading/querying data.
>> Will this be added when the final version does get released?
>>
>> Thanks,
>> Ben
>>
>> On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>> The hbase-spark module in Apache HBase (coming with hbase 2.0 release)
>> can do this.
>>
>> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com> wrote:
>>
>>> Has anyone found an easy way to save a DataFrame into HBase?
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>
>>
>>
>
Re: Save DataFrame to HBase
I think so. Please refer to the table population tests in (master branch): hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/DefaultSourceSuite.scala Cheers On Tue, May 10, 2016 at 2:53 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > Ted, > > Will the hbase-spark module allow for creating tables in Spark SQL that > reference the hbase tables underneath? In this way, users can query using > just SQL. > > Thanks, > Ben > > On Apr 28, 2016, at 3:09 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > Hbase 2.0 release likely would come after Spark 2.0 release. > > There're other features being developed in hbase 2.0 > I am not sure when hbase 2.0 would be released. > > The refguide is incomplete. > Zhan has assigned the doc JIRA to himself. The documentation would be done > after fixing bugs in hbase-spark module. > > Cheers > > On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > > Hi Ted, > > Do you know when the release will be? I also see some documentation for > usage of the hbase-spark module at the hbase website. But, I don’t see an > example on how to save data. There is only one for reading/querying data. > Will this be added when the final version does get released? > > Thanks, > Ben > > On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can > do this. > > On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > >> Has anyone found an easy way to save a DataFrame into HBase? >> >> Thanks, >> Ben >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > >
Re: Save DataFrame to HBase
Ted, Will the hbase-spark module allow for creating tables in Spark SQL that reference the hbase tables underneath? In this way, users can query using just SQL. Thanks, Ben > On Apr 28, 2016, at 3:09 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > Hbase 2.0 release likely would come after Spark 2.0 release. > > There're other features being developed in hbase 2.0 > I am not sure when hbase 2.0 would be released. > > The refguide is incomplete. > Zhan has assigned the doc JIRA to himself. The documentation would be done > after fixing bugs in hbase-spark module. > > Cheers > > On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > >> Hi Ted, >> >> Do you know when the release will be? I also see some documentation for >> usage of the hbase-spark module at the hbase website. But, I don’t see an >> example on how to save data. There is only one for reading/querying data. >> Will this be added when the final version does get released? >> >> Thanks, >> Ben >> >>> On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com >>> <mailto:yuzhih...@gmail.com>> wrote: >>> >>> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can >>> do this. >>> >>> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com >>> <mailto:bbuil...@gmail.com>> wrote: >>> Has anyone found an easy way to save a DataFrame into HBase? >>> >>> Thanks, >>> Ben >>> >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> <mailto:user-unsubscr...@spark.apache.org> >>> For additional commands, e-mail: user-h...@spark.apache.org >>> <mailto:user-h...@spark.apache.org> >>> >>> >>
Re: Save DataFrame to HBase
Hbase 2.0 release likely would come after Spark 2.0 release. There're other features being developed in hbase 2.0 I am not sure when hbase 2.0 would be released. The refguide is incomplete. Zhan has assigned the doc JIRA to himself. The documentation would be done after fixing bugs in hbase-spark module. Cheers > On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > > Hi Ted, > > Do you know when the release will be? I also see some documentation for usage > of the hbase-spark module at the hbase website. But, I don’t see an example > on how to save data. There is only one for reading/querying data. Will this > be added when the final version does get released? > > Thanks, > Ben > >> On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can >> do this. >> >>> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com> wrote: >>> Has anyone found an easy way to save a DataFrame into HBase? >>> >>> Thanks, >>> Ben >>> >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >
Re: Save DataFrame to HBase
Hi Ted, Do you know when the release will be? I also see some documentation for usage of the hbase-spark module at the hbase website. But, I don’t see an example on how to save data. There is only one for reading/querying data. Will this be added when the final version does get released? Thanks, Ben > On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do > this. > > On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > Has anyone found an easy way to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> > >
Re: Save DataFrame to HBase
Daniel, If you can get the code snippet, that would be great! I’ve been trying to get it to work for me as well. The examples on the Phoenix website do not work for me. If you are willing to also, can you include your setup to make Phoenix work with Spark? Thanks, Ben > On Apr 27, 2016, at 11:46 AM, Paras sachdeva <paras.sachdeva11...@gmail.com> > wrote: > > Hi Daniel, > > Would you possibly be able to share the snipped to code you have used ? > > Thank you. > > On Wed, Apr 27, 2016 at 3:13 PM, Daniel Haviv > <daniel.ha...@veracity-group.com <mailto:daniel.ha...@veracity-group.com>> > wrote: > Hi Benjamin, > Yes it should work. > > Let me know if you need further assistance I might be able to get the code > I've used for that project. > > Thank you. > Daniel > > On 24 Apr 2016, at 17:35, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > >> Hi Daniel, >> >> How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which >> comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? >> >> Thanks, >> Ben >> >>> On Apr 24, 2016, at 1:43 AM, Daniel Haviv <daniel.ha...@veracity-group.com >>> <mailto:daniel.ha...@veracity-group.com>> wrote: >>> >>> Hi, >>> I tried saving DF to HBase using a hive table with hbase storage handler >>> and hiveContext but it failed due to a bug. >>> >>> I was able to persist the DF to hbase using Apache Pheonix which was pretty >>> simple. >>> >>> Thank you. >>> Daniel >>> >>> On 21 Apr 2016, at 16:52, Benjamin Kim <bbuil...@gmail.com >>> <mailto:bbuil...@gmail.com>> wrote: >>> >>>> Has anyone found an easy way to save a DataFrame into HBase? >>>> >>>> Thanks, >>>> Ben >>>> >>>> >>>> - >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> <mailto:user-unsubscr...@spark.apache.org> >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> <mailto:user-h...@spark.apache.org> >>>> >> >
Re: Save DataFrame to HBase
Hi Daniel, Would you possibly be able to share the snipped to code you have used ? Thank you. On Wed, Apr 27, 2016 at 3:13 PM, Daniel Haviv < daniel.ha...@veracity-group.com> wrote: > Hi Benjamin, > Yes it should work. > > Let me know if you need further assistance I might be able to get the code > I've used for that project. > > Thank you. > Daniel > > On 24 Apr 2016, at 17:35, Benjamin Kim <bbuil...@gmail.com> wrote: > > Hi Daniel, > > How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed > which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? > > Thanks, > Ben > > On Apr 24, 2016, at 1:43 AM, Daniel Haviv <daniel.ha...@veracity-group.com> > wrote: > > Hi, > I tried saving DF to HBase using a hive table with hbase storage handler > and hiveContext but it failed due to a bug. > > I was able to persist the DF to hbase using Apache Pheonix which was > pretty simple. > > Thank you. > Daniel > > On 21 Apr 2016, at 16:52, Benjamin Kim <bbuil...@gmail.com> wrote: > > Has anyone found an easy way to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > >
Re: Save DataFrame to HBase
Hi Benjamin, Yes it should work. Let me know if you need further assistance I might be able to get the code I've used for that project. Thank you. Daniel > On 24 Apr 2016, at 17:35, Benjamin Kim <bbuil...@gmail.com> wrote: > > Hi Daniel, > > How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which > comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? > > Thanks, > Ben > >> On Apr 24, 2016, at 1:43 AM, Daniel Haviv <daniel.ha...@veracity-group.com> >> wrote: >> >> Hi, >> I tried saving DF to HBase using a hive table with hbase storage handler and >> hiveContext but it failed due to a bug. >> >> I was able to persist the DF to hbase using Apache Pheonix which was pretty >> simple. >> >> Thank you. >> Daniel >> >>> On 21 Apr 2016, at 16:52, Benjamin Kim <bbuil...@gmail.com> wrote: >>> >>> Has anyone found an easy way to save a DataFrame into HBase? >>> >>> Thanks, >>> Ben >>> >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >
Re: Save DataFrame to HBase
Hi Daniel, How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work? Thanks, Ben > On Apr 24, 2016, at 1:43 AM, Daniel Haviv <daniel.ha...@veracity-group.com> > wrote: > > Hi, > I tried saving DF to HBase using a hive table with hbase storage handler and > hiveContext but it failed due to a bug. > > I was able to persist the DF to hbase using Apache Pheonix which was pretty > simple. > > Thank you. > Daniel > > On 21 Apr 2016, at 16:52, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > >> Has anyone found an easy way to save a DataFrame into HBase? >> >> Thanks, >> Ben >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> <mailto:user-unsubscr...@spark.apache.org> >> For additional commands, e-mail: user-h...@spark.apache.org >> <mailto:user-h...@spark.apache.org> >>
Re: Save DataFrame to HBase
Hi, I tried saving DF to HBase using a hive table with hbase storage handler and hiveContext but it failed due to a bug. I was able to persist the DF to hbase using Apache Pheonix which was pretty simple. Thank you. Daniel > On 21 Apr 2016, at 16:52, Benjamin Kim <bbuil...@gmail.com> wrote: > > Has anyone found an easy way to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
Re: Save DataFrame to HBase
You can try this https://github.com/hortonworks/shc.git or here http://spark-packages.org/package/zhzhan/shc Currently it is in the process of merging into HBase. Thanks. Zhan Zhang On Apr 21, 2016, at 8:44 AM, Benjamin Kim <bbuil...@gmail.com<mailto:bbuil...@gmail.com>> wrote: Hi Ted, Can this module be used with an older version of HBase, such as 1.0 or 1.1? Where can I get the module from? Thanks, Ben On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>> wrote: The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this. On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com<mailto:bbuil...@gmail.com>> wrote: Has anyone found an easy way to save a DataFrame into HBase? Thanks, Ben - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
Re: Save DataFrame to HBase
Hi Ted, Can this module be used with an older version of HBase, such as 1.0 or 1.1? Where can I get the module from? Thanks, Ben > On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do > this. > > On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > Has anyone found an easy way to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> > >
Re: Save DataFrame to HBase
The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this. On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuil...@gmail.com> wrote: > Has anyone found an easy way to save a DataFrame into HBase? > > Thanks, > Ben > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Save DataFrame to HBase
Has anyone found an easy way to save a DataFrame into HBase? Thanks, Ben - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Save dataframe into hbase
Hi What is the efficient way to save Dataframe into hbase? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Save-dataframe-into-hbase-tp24552.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Save dataframe into hbase
The following JIRA is close to integration: HBASE-14181 Add Spark DataFrame DataSource to HBase-Spark Module after which hbase would provide better support for DataFrame interaction. On Wed, Sep 2, 2015 at 1:21 PM, ALEX K <alex.ka...@gmail.com> wrote: > you can use Phoenix-Spark plugin: > https://phoenix.apache.org/phoenix_spark.html > > On Wed, Sep 2, 2015 at 4:04 AM, Hafiz Mujadid <hafizmujadi...@gmail.com> > wrote: > >> Hi >> >> What is the efficient way to save Dataframe into hbase? >> >> Thanks >> >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Save-dataframe-into-hbase-tp24552.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: Save dataframe into hbase
you can use Phoenix-Spark plugin: https://phoenix.apache.org/phoenix_spark.html On Wed, Sep 2, 2015 at 4:04 AM, Hafiz Mujadid <hafizmujadi...@gmail.com> wrote: > Hi > > What is the efficient way to save Dataframe into hbase? > > Thanks > > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Save-dataframe-into-hbase-tp24552.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
