Hi Philip,
I got this bit of code to work in the spark-shell using scala against our dev
hbase cluster.
-bash-4.1$export
SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/cloudera/parcels/CDH/lib/hbase/hbase.jar:/opt/cloudera/parcels/CDH/lib/hbase/conf:/opt/cloudera/parcels/CDH/lib/hadoop/conf
-bash-4.1$./spark-shellscala>import
org.apache.hadoop.hbase.HBaseConfigurationscala>import
org.apache.hadoop.hbase.client._scala>import
org.apache.hadoop.hbase.util.Bytesscala>val conf =
HBaseConfiguration.create()scala>val table = new HTable(conf,
"my_items")scala>val p = new
Put(Bytes.toBytes("strawberry-fruit"))scala>p.add(Bytes.toBytes("item"),Bytes.toBytes("item"),Bytes.toBytes("strawberry"))scala>p.add(Bytes.toBytes("item"),Bytes.toBytes("category"),Bytes.toBytes("fruit"))scala>p.add(Bytes.toBytes("item"),Bytes.toBytes("price"),Bytes.toBytes("0.35"))scala>table.put(p)
It put the new row "strawberry-fruit" into an hbase table.
Sorry, but I have another newbie question. How do I add those CLASSPATH
dependencies when I want to compile a streaming jar in sbt so that the hbase
configs are automatically used?
Thanks,Ben
Date: Thu, 5 Dec 2013 10:24:02 -0700
From: philip.og...@oracle.com
To: user@spark.incubator.apache.org
Subject: Re: Writing to HBase
Here's a good place to start:
http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201311.mbox/%3ccacyzca3askwd-tujhqi1805bn7sctguaoruhd5xtxcsul1a...@mail.gmail.com%3E
On 12/5/2013 10:18 AM, Benjamin Kim
wrote:
Does anyone have an example or some sort of
starting point code when writing from Spark Streaming into
HBase?
We currently stream ad server event log data using Flume-NG
to tail log entries, collect them, and put them directly into
a HBase table. We would like to do the same with Spark
Streaming. But, we would like to do the data massaging and
simple data analysis before. This will cut down the steps in
prepping data and the number of tables for our data scientists
and real-time feedback systems.
Thanks,
Ben
