Re: Putting record in HBase with Spark - error get regions.

> I'm executing a load process into HBase with spark. (around 150M record). > At the end of the process there are a lot of fail tasks. > > I get this error: > > 19/05/28 11:02:31 ERROR client.AsyncProcess: Failed to get region location > org.apache.hadoop.hbase.Table

Putting record in HBase with Spark - error get regions.

I'm executing a load process into HBase with spark. (around 150M record). At the end of the process there are a lot of fail tasks. I get this error: 19/05/28 11:02:31 ERROR client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.TableNotFoundException: my_table

Write data from Hbase using Spark Failing with NPE

aI am using Spark to write data to Hbase, I can read data just fine but write is failing with following exception. I found simila issue that got resolved by adding *site.xml and hbase JARs. But it is npot working for me.       JavaPairRDD  tablePuts =

Getting Data From Hbase using Spark is Extremely Slow

I have written four lines of simple spark program to process data in Phoenix table: queryString = getQueryFullString( );// Get data from Phoenix table select col from table JavaPairRDD phRDD = jsc.newAPIHadoopRDD(

Spark with HBase on Spark Runtime 2.2.1

I wrote a simple program to read data from HBase, the program works find in Cloudera backed by HDFS. The program works fine on SPARK RUNTIME 1.6 on Cloudera. But does NOT work on EMR with Spark Runtime 2.2.1. But getting an exception while testing data on EMR with S3. // Spark conf

Re: Hbase and Spark

Hi Masf, Do try the official Hbase Spark. https://hbase.apache.org/book.html#spark I think you will have to build the jar from source and run your spark program with --packages . https://spark-packages.org/package/hortonworks-spark/shc says it's not yet published to Spark packages or Maven Repo

Hbase and Spark

I´m trying to build an application where is necessary to do bulkGets and bulkLoad on Hbase. I think that I could use this component https://github.com/hortonworks-spark/shc *Is it a good option??* But* I can't import it in my project*. Sbt cannot resolve hbase connector This is my build.sbt:

Re: Accessing HBase through Spark with Security enabled

>> hconf.set("hbase.zookeeper.quorum", "hadoop-master") >> hconf.set(TableInputFormat.INPUT_TABLE, tableName) >> hconf.set("hbase.zookeeper.property.clientPort", "2181") >> hconf.set("hbase.master", "hadoop-master:6

Re: Accessing HBase through Spark with Security enabled

f.set("hbase.zookeeper.quorum", "hadoop-master") > hconf.set(TableInputFormat.INPUT_TABLE, tableName) > hconf.set("hbase.zookeeper.property.clientPort", "2181") > hconf.set("hbase.master", "hadoop-master:60000") > hconf.set("hadoop.security.authentic

Re: Accessing HBase through Spark with Security enabled

0") hconf.set("hadoop.security.authentication", "kerberos") hconf.addResource(new Path("/etc/hbase/conf/core-site.xml")) hconf.addResource(new Path("/etc/hbase/conf/hbase-site.xml")) spark should be automatically picking those up from the classpath; adding them t

Re: Accessing HBase through Spark with Security enabled

; main/scala/org/apache/spark/deploy/yarn/Client.scala#L396 > > Pozdrawiam, > Jacek Laskowski > > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > >

Re: Accessing HBase through Spark with Security enabled

at https://twitter.com/jaceklaskowski On Fri, Aug 12, 2016 at 11:30 PM, Aneela Saleem <ane...@platalytics.com> wrote: > Thanks for your response Jacek! > > Here is the code, how spark accesses HBase: > System.setProperty("java.security.krb5.conf", "/etc/

Re: Accessing HBase through Spark with Security enabled

Thanks for your response Jacek! Here is the code, how spark accesses HBase: System.setProperty("java.security.krb5.conf", "/etc/krb5.conf"); System.setProperty("java.security.auth.login.config", "/etc/hbase/conf/zk-jaas.conf"); val hconf = HBaseCon

Re: Accessing HBase through Spark with Security enabled

Hi, How do you access HBase? What's the version of Spark? (I don't see spark packages in the stack trace) Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On

Accessing HBase through Spark with Security enabled

Hi all, I'm trying to run a spark job that accesses HBase with security enabled. When i run the following command: */usr/local/spark-2/bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal spark/hadoop-master@platalyticsrealm --class com.platalytics.example.spark.App --master yarn

Re: How to connect HBase and Spark using Python?

Solved, see: http://stackoverflow.com/questions/38470114/how-to-connect-hbase-and-spark-using-python/38575095 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-connect-HBase-and-Spark-using-Python-tp27372p27409.html Sent from the Apache Spark User List

Re: How to connect HBase and Spark using Python?

It is included in Cloudera’s CDH 5.8. > On Jul 22, 2016, at 6:13 PM, Mail.com <pradeep.mi...@mail.com> wrote: > > Hbase Spark module will be available with Hbase 2.0. Is that out yet? > >> On Jul 22, 2016, at 8:50 PM, Def_Os <njde...@gmail.com> wrote: >> &g

Re: How to connect HBase and Spark using Python?

Hbase Spark module will be available with Hbase 2.0. Is that out yet? > On Jul 22, 2016, at 8:50 PM, Def_Os <njde...@gmail.com> wrote: > > So it appears it should be possible to use HBase's new hbase-spark module, if > you follow this pattern: > https://hbase

Re: How to connect HBase and Spark using Python?

So it appears it should be possible to use HBase's new hbase-spark module, if you follow this pattern: https://hbase.apache.org/book.html#_sparksql_dataframes Unfortunately, when I run my example from PySpark, I get the following exception: > py4j.protocol.Py4JJavaError: An error occurred wh

How to connect HBase and Spark using Python?

I'd like to know whether there's any way to query HBase with Spark SQL via the PySpark interface. See my question on SO: http://stackoverflow.com/questions/38470114/how-to-connect-hbase-and-spark-using-python The new HBase-Spark module in HBase, which introduces the HBaseContext/JavaHBaseContext

Re: Recommended way to push data into HBase through Spark streaming

Forgot to add, I'm on HBase 1.0.0-cdh5.4.5, so can't use HBaseContext. And spark version is 1.6.1 [image: http://] Tariq, Mohammad about.me/mti [image: http://] On Thu, Jun 16, 2016 at 10:12 PM, Mohammad Tariq wrote: > Hi group, > > I have a

Recommended way to push data into HBase through Spark streaming

Hi group, I have a streaming job which reads data from Kafka, performs some computation and pushes the result into HBase. Actually the results are pushed into 3 different HBase tables. So I was wondering what could be the best way to achieve this. Since each executor will open its own HBase

Re: Hbase in spark

I know little about your use case. Did you mean that your data is relatively evenly distributed in Spark domain but showed skew in the bulk load phase ? On Fri, Feb 26, 2016 at 9:02 AM, Renu Yadav <yren...@gmail.com> wrote: > Hi Ted, > > Thanks for the reply. I am using spark hb

Re: Hbase in spark

In hbase, there is hbase-spark module which supports bulk load. This module is to be backported in the upcoming 1.3.0 release. There is some pending work, such as HBASE-15271 . FYI On Fri, Feb 26, 2016 at 8:50 AM, Renu Yadav <yren...@gmail.com> wrote: > Has anybody implemented

Hbase in spark

Has anybody implemented bulk load into hbase using spark? I need help to optimize its performance. Please help. Thanks & Regards, Renu Yadav

Re: is Hbase Scan really need thorough Get (Hbase+solr+spark)

= getFromHbaseWithRowKey(k)## This is just for illustration, there is no such function :) res.add(v) return res On Wed, Jan 20, 2016 at 10:09 AM, beeshma r <beeshm...@gmail.com> wrote: > Hi > > I trying to integrated Hbase-solr-spark. > Solr is indexing all

Re: is Hbase Scan really need thorough Get (Hbase+solr+spark)

t; On Wed, Jan 20, 2016 at 10:09 AM, beeshma r <beeshm...@gmail.com> wrote: > >> Hi >> >> I trying to integrated Hbase-solr-spark. >> Solr is indexing all the documents from Hbase through hbase-indexer . >> Through the Spark I am manipulating all datasets .Thing is

is Hbase Scan really need thorough Get (Hbase+solr+spark)

Hi I trying to integrated Hbase-solr-spark. Solr is indexing all the documents from Hbase through hbase-indexer . Through the Spark I am manipulating all datasets .Thing is after getting the solrdocuments from the solr query ,it has the rowkey and rowvalues .So directly i got the rowkeys

Re: is Hbase Scan really need thorough Get (Hbase+solr+spark)

res.add(v) >> return res >> >> On Wed, Jan 20, 2016 at 10:09 AM, beeshma r <beeshm...@gmail.com> wrote: >> >>> Hi >>> >>> I trying to integrated Hbase-solr-spark. >>> Solr is indexing all the documents from Hbase through hbase

Re:Re: Re: Re: Error in load hbase on spark

:16, "Ted Yu" <yuzhih...@gmail.com> wrote: One possibility was that hbase config, including hbase.zookeeper.quorum, was not passed to your job. hbase-site.xml should be on the classpath. Can you show snippet of your code ? Looks like you were running against hbase 1.x C

Re: Re: Re: Error in load hbase on spark

> >> Thanks. >> >> >> >> >> >> >> At 2015-10-09 10:35:16, "Ted Yu" <yuzhih...@gmail.com> wrote: >> >> One possibility was that hbase config, including hbase.zookeeper.quorum, >> was not passed to

Re: Error in load hbase on spark

Hi Roy, Here is a cloudera-labs project SparkOnHBase that makes it really simple to read HBase data into Spark. https://github.com/cloudera-labs/SparkOnHBase <https://github.com/cloudera-labs/SparkOnHBase> Link to blog that explains how to use the package. http://blog.cloudera.com/blo

Re: Error in load hbase on spark

Work for hbase-spark module is still ongoing https://issues.apache.org/jira/browse/HBASE-14406 > On Oct 9, 2015, at 6:18 AM, Guru Medasani <gdm...@gmail.com> wrote: > > Hi Roy, > > Here is a cloudera-labs project SparkOnHBase that makes it really simple to > re

Re: Re: Error in load hbase on spark

hould be on the classpath. > > Can you show snippet of your code ? > > Looks like you were running against hbase 1.x > > Cheers > > On Thu, Oct 8, 2015 at 7:29 PM, Roy Wang <roywang1...@163.com> wrote: > >> >> I want to

Error in load hbase on spark

I want to load hbase table into spark. JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD = sc.newAPIHadoopRDD(conf, TableInputFormat.class, ImmutableBytesWritable.class, Result.class); *when call hBaseRDD.count(),got error.* Caused by: java.lang.IllegalStateException: The input

Re: Error in load hbase on spark

com> wrote: > > I want to load hbase table into spark. > JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD = > sc.newAPIHadoopRDD(conf, TableInputFormat.class, > ImmutableBytesWritable.class, Result.class); > > *when call hBaseRDD.count(),got error.* > > Cau

How to access HBase on Spark SQL

I notice that databricks provides external datasource api for Spark SQL. But I can't find any reference documents to guide how to access HBase based on it directly. Who know it? Or please give me some related links. Thanks. ZhangYi (张逸) BigEye website:

Does anyone integrate HBASE on Spark

Hi Sparkers, How do i integrate hbase on spark !!! Appreciate for replies !! Regards, Sandeep.v

Re: Does anyone integrate HBASE on Spark

Hi, There are some examples in spark/example https://github.com/apache/spark/tree/master/examples and there are also some examples in spark package http://spark-packages.org/. And I find this blog http://www.abcn.net/2014/07/lighting-spark-with-hbase-full-edition.html is quite good. Hope

Re: How to integrate HBASE on Spark

, 2015 8:52 AM Subject: Re: How to integrate HBASE on Spark Installing hbase on hadoop cluster would allow hbase to utilize features provided by hdfs, such as short circuit read (See '90.2. Leveraging local data' under http://hbase.apache.org/book.html#perf.hdfs). Cheers On Sun, Feb 22, 2015

Re: How to integrate HBASE on Spark

PM, sandeep vura sandeepv...@gmail.com wrote: Hi I had installed spark on 3 node cluster. Spark services are up and running.But i want to integrate hbase on spark Do i need to install HBASE on hadoop cluster or spark cluster. Please let me know asap. Regards, Sandeep.v

Re: How to integrate HBASE on Spark

sandeepv...@gmail.com; user@spark.apache.org user@spark.apache.org *Sent:* Monday, February 23, 2015 8:52 AM *Subject:* Re: How to integrate HBASE on Spark Installing hbase on hadoop cluster would allow hbase to utilize features provided by hdfs, such as short circuit read (See '90.2

Re: How to integrate HBASE on Spark

. Spark services are up and running.But i want to integrate hbase on spark Do i need to install HBASE on hadoop cluster or spark cluster. Please let me know asap. Regards, Sandeep.v

How to integrate HBASE on Spark

Hi I had installed spark on 3 node cluster. Spark services are up and running.But i want to integrate hbase on spark Do i need to install HBASE on hadoop cluster or spark cluster. Please let me know asap. Regards, Sandeep.v

Re: How to integrate HBASE on Spark

PM, sandeep vura sandeepv...@gmail.com wrote: Hi I had installed spark on 3 node cluster. Spark services are up and running.But i want to integrate hbase on spark Do i need to install HBASE on hadoop cluster or spark cluster. Please let me know asap. Regards, Sandeep.v

Re:Re: Serialization issue when using HBase with Spark

this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Serialization-issue-when-using-HBase-with-Spark-tp20655.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe

Re: Serialization issue when using HBase with Spark

this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Serialization-issue-when-using-HBase-with-Spark-tp20655.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe

Re: Serialization issue when using HBase with Spark

2, but I prefer option 1. So is there any solution for option 1? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Serialization-issue-when-using-HBase-with-Spark-tp20655.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Serialization issue when using HBase with Spark

/Serialization-issue-when-using-HBase-with-Spark-tp20655.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user

Serialization issue when using HBase with Spark

/Serialization-issue-when-using-HBase-with-Spark-tp20655.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h

Re: Serialization issue when using HBase with Spark

Can you paste the complete code? it looks like at some point you are passing a hadoop's configuration which is not Serializable. You can look at this thread for similar discussion http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-into-HBase-td13378.html Thanks Best Regards

Re: Issue Connecting to HBase in spark shell

It looks like the issue I had is that I didn't pull in htrace-core jar into the spark class path. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issue-Connecting-to-HBase-in-spark-shell-tp12855p12924.html Sent from the Apache Spark User List mailing list

Re: Writing data to HBase using Spark

Is there anyone else who is facing this problem of writing to HBase when running Spark on YARN mode or Standalone mode using this example? If not, then do I need to explicitly, specify something in the classpath? Regards, Gaurav On Wed, Jun 11, 2014 at 1:53 PM, Gaurav Dasgupta gaurav.d

Re: Writing data to HBase using Spark

else who is facing this problem of writing to HBase when running Spark on YARN mode or Standalone mode using this example? If not, then do I need to explicitly, specify something in the classpath? Regards, Gaurav On Wed, Jun 11, 2014 at 1:53 PM, Gaurav Dasgupta [hidden email] http://user

Re: Writing data to HBase using Spark

. If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7305.html To start a new topic under Apache Spark User List, email [hidden email] (/user/SendEmail.jtp?type=nodenode

Writing data to HBase using Spark

Hi, I am reading data from a HBase table to RDD and then using foreach on that RDD I am doing some processing on every Result of HBase table. After this processing I want to store the processed data back to another HBase table. How can I do that ? If I use standard Hadoop and HBase classes to

Re: Writing data to HBase using Spark

Please see sample code attached at https://issues.apache.org/jira/browse/SPARK-944. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7305.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark on HBase vs. Spark on HDFS

Also I am unsure if Spark on Hbase leverages Locality. When you cache process data do you see node_local jobs in process list. Spark on HDFS leverages locality quite well can really boost performance by 3-4x in my experience. If you are loading all your data from HBase to spark then you

Spark on HBase vs. Spark on HDFS

on having to set up an additional set of components on top of HDFS instead of just writing to HDFS directly. Is there any overview regarding implications of doing that ? I mean except things like taking care of file structure and the like. What is the true advantage of Spark on HBase in favor

Re: Spark on HBase vs. Spark on HDFS

of file structure and the like. What is the true advantage of Spark on HBase in favor of Spark on HDFS? Best Philip Automic Software GmbH, Hauptstrasse 3C, 3012 Wolfsgraben Firmenbuchnummer/Commercial Register No. 275184h Firmenbuchgericht/Commercial Register Court: Landesgericht St