spark hbase

Hi, I have tried multiple ways to use hbase-spark and none of them works as expected. SHC and hbase-spark library are loading all the data on executors and it is running for ever. https://ramottamado.dev/how-to-use-hbase-fuzzyrowfilter-in-spark/ Above link has the solution that I am looking for

Spark Hbase Hive error in EMR

Hi, I am trying to connect hbase which sits on top of hive as external table. I am getting below exception. Am I missing anything to pass here? 21/04/09 18:08:11 INFO ZooKeeper: Client environment:user.dir=/ 21/04/09 18:08:11 INFO ZooKeeper: Initiating client connection,

spark hbase connector

Hi, My spark jobs need to talk to hbase and I am not sure which spark hbase connector is recommended: https://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/ https://phoenix.apache.org/phoenix_spark.html Or there is any other better solutions. Appreciate any guidance.

Hortonworks Spark-Hbase-Connector does not read zookeeper configurations from spark session config ??(Spark on Yarn)

Hi I am trying to write a spark code that reads data from Hbase and store it in DataFrame. I am able to run it perfectly with hbase-site.xml in $spark-home/conf folder. But I am facing few issues Here. Issue 1: Passing hbase-site.xml location with --file parameter submitted through client

Financial fraud detection using streaming RDBMS data into Spark & Hbase

I am not talking about Credit Card fraud etc. In the complex fraud cases like that one in UBS , the rogue trader over a period of time manipulated the figures. Although there is a lot of talk about using elaborate set-ups to predict

Issue with Spark HBase connector streamBulkGet method

ser-list.1001560.n3.nabble.com/Issue-with-Spark-HBase-connector-streamBulkGet-method-tp27613.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark HBase bulk load using hfile format

Hi i am doing bulk load into HBase as HFileFormat, by using saveAsNewAPIHadoopFile when i try to write i am getting an exception java.io.IOException: Added a key not lexically larger than previous. following is the code snippet case class HBaseRow(rowKey: ImmutableBytesWritable, kv: KeyValue)

A problem involving Spark & HBase.

ark-user-list.1001560.n3.nabble.com/A-problem-involving-Spark-HBase-tp26602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional com

Re: Generalised Spark-HBase integration

Hi Ted, yes, cloudera blog and your code was my starting point - but I needed something more spark-centric rather than on hbase. Basically doing a lot of ad-hoc transformations with RDDs that were based on HBase tables and then mutating them after series of iterative (bsp-like) steps. On 28 July

Re: Generalised Spark-HBase integration

/classes at 1438099569598 [ERROR] /home/hbase/s-on-hbase/src/main/scala/org/apache/spark/hbase/examples/simple/HBaseTableSimple.scala:36: error: type mismatch; [INFO] found : Int [INFO] required: Short [INFO] while (scanner.advance) numCells += 1 [INFO

Re: Generalised Spark-HBase integration

Cool, will revisit, is your latest code visible publicly somewhere ? On 28 July 2015 at 17:14, Ted Malaska ted.mala...@cloudera.com wrote: Yup you should be able to do that with the APIs that are going into HBase. Let me know if you need to chat about the problem and how to implement it with

Generalised Spark-HBase integration

Hi all, last couple of months I've been working on a large graph analytics and along the way have written from scratch a HBase-Spark integration as none of the ones out there worked either in terms of scale or in the way they integrated with the RDD interface. This week I have generalised it into

Re: Generalised Spark-HBase integration

I got a compilation error: [INFO] /home/hbase/s-on-hbase/src/main/scala:-1: info: compiling [INFO] Compiling 18 source files to /home/hbase/s-on-hbase/target/classes at 1438099569598 [ERROR] /home/hbase/s-on-hbase/src/main/scala/org/apache/spark/hbase/examples/simple/HBaseTableSimple.scala:36

Re: Spark + HBase + Kerberos

:46 PM To: Eric Walk Cc: user@spark.apache.org;Bill Busch Subject: Re: Spark + HBase + Kerberos Are hbase config / keytab files deployed on executor machines ? Consider adding -Dsun.security.krb5.debug=true for debug purpose. Cheers On Wed, Mar 18, 2015 at 11:39 AM, Eric Walk eric.w

Re: Spark + HBase + Kerberos

Are hbase config / keytab files deployed on executor machines ? Consider adding -Dsun.security.krb5.debug=true for debug purpose. Cheers On Wed, Mar 18, 2015 at 11:39 AM, Eric Walk eric.w...@perficient.com wrote: Having an issue connecting to HBase from a Spark container in a Secure

Spark Hbase

Hi Team, Could you please point me the example program for Spark HBase to read columns and values Regards, Rajesh

Re: Spark Hbase

Cheers On Wed, Sep 24, 2014 at 9:39 AM, Madabhattula Rajesh Kumar mrajaf...@gmail.com wrote: Hi Team, Could you please point me the example program for Spark HBase to read columns and values Regards, Rajesh

Re: Spark Hbase

/HBaseConverters.scala Cheers On Wed, Sep 24, 2014 at 9:39 AM, Madabhattula Rajesh Kumar mrajaf...@gmail.com wrote: Hi Team, Could you please point me the example program for Spark HBase to read columns and values Regards, Rajesh

Re: Spark Hbase job taking long time

://apache-spark-user-list.1001560.n3.nabble.com/Spark-Hbase-job-taking-long-time-tp11541.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: [hidden email] http://user

Re: Spark Hbase job taking long time

start-all.sh script with 2 workers Any pointers will be of a great help Regards, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Hbase-job-taking-long-time-tp11541.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Need help on spark Hbase

Hi Team, Now i've changed my code and reading configuration from hbase-site.xml file(this file is in classpath). When i run this program using : mvn exec:java -Dexec.mainClass=com.cisco.ana.accessavailability.AccessAvailability. It is working fine. But when i run this program from spark-submit

Re: Need help on spark Hbase

Hi Rajesh, I saw : Warning: Local jar /home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase -client-0.96.1.1-hadoop2.jar, does not exist, skipping. in your log. I believe this jar contains the HBaseConfiguration. I'm not sure what went wrong in your case but can you try without spaces in --jars i.e.

Re: Need help on spark Hbase

Hi Rajesh, can you describe your spark cluster setup? I saw localhost:2181 for zookeeper. Best Regards, Jerry On Tue, Jul 15, 2014 at 9:47 AM, Madabhattula Rajesh Kumar mrajaf...@gmail.com wrote: Hi Team, Could you please help me to resolve the issue. *Issue *: I'm not able to connect

Re: Need help on spark Hbase

One vector to check is the HBase libraries in the --jars as in : spark-submit --class your class --master master url --jars

Re: Need help on spark Hbase

Hi Rajesh, I have a feeling that this is not directly related to spark but I might be wrong. The reason why is that when you do: Configuration configuration = HBaseConfiguration.create(); by default, it reads the configuration files hbase-site.xml in your classpath and ... (I don't remember

Re: Need help on spark Hbase

Also, it helps if you post us logs, stacktraces, exceptions, etc. TD On Tue, Jul 15, 2014 at 10:07 AM, Jerry Lam chiling...@gmail.com wrote: Hi Rajesh, I have a feeling that this is not directly related to spark but I might be wrong. The reason why is that when you do: Configuration