Re: Writing data to HBase using Spark
Is there anyone else who is facing this problem of writing to HBase when running Spark on YARN mode or Standalone mode using this example? If not, then do I need to explicitly, specify something in the classpath? Regards, Gaurav On Wed, Jun 11, 2014 at 1:53 PM, Gaurav Dasgupta gaurav.d...@gmail.com wrote: Hi Kanwaldeep, I have tried your code but arrived into a problem. The code is working fine in *local* mode. But if I run the same code in Spark stand alone mode or YARN mode, then it is continuously executing, but not saving anything in the HBase table. I guess, it is stopping data streaming once the *saveToHBase* method is called for the first time. This is strange. I just want to know whether you have tested the code on all Spark execution modes? Thanks, Gaurav On Tue, Jun 10, 2014 at 12:20 PM, Kanwaldeep [via Apache Spark User List] ml-node+s1001560n7305...@n3.nabble.com wrote: Please see sample code attached at https://issues.apache.org/jira/browse/SPARK-944. -- If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7305.html To start a new topic under Apache Spark User List, email ml-node+s1001560n1...@n3.nabble.com To unsubscribe from Apache Spark User List, click here http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=1code=Z2F1cmF2LmRnMTlAZ21haWwuY29tfDF8LTk5NzA0ODAy . NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7474.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Writing data to HBase using Spark
Are you able to use HadoopInputoutput reader for hbase in new hadoop Api reader? Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi https://twitter.com/mayur_rustagi On Thu, Jun 12, 2014 at 7:49 AM, gaurav.dasgupta gaurav.d...@gmail.com wrote: Is there anyone else who is facing this problem of writing to HBase when running Spark on YARN mode or Standalone mode using this example? If not, then do I need to explicitly, specify something in the classpath? Regards, Gaurav On Wed, Jun 11, 2014 at 1:53 PM, Gaurav Dasgupta [hidden email] http://user/SendEmail.jtp?type=nodenode=7474i=0 wrote: Hi Kanwaldeep, I have tried your code but arrived into a problem. The code is working fine in *local* mode. But if I run the same code in Spark stand alone mode or YARN mode, then it is continuously executing, but not saving anything in the HBase table. I guess, it is stopping data streaming once the *saveToHBase* method is called for the first time. This is strange. I just want to know whether you have tested the code on all Spark execution modes? Thanks, Gaurav On Tue, Jun 10, 2014 at 12:20 PM, Kanwaldeep [via Apache Spark User List] [hidden email] http://user/SendEmail.jtp?type=nodenode=7474i=1 wrote: Please see sample code attached at https://issues.apache.org/jira/browse/SPARK-944. -- If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7305.html To start a new topic under Apache Spark User List, email [hidden email] http://user/SendEmail.jtp?type=nodenode=7474i=2 To unsubscribe from Apache Spark User List, click here. NAML http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: Re: Writing data to HBase using Spark http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7474.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Writing data to HBase using Spark
you are using spark streaming? master = “local[n]” where n 1? Best, -- Nan Zhu On Wednesday, June 11, 2014 at 4:23 AM, gaurav.dasgupta wrote: Hi Kanwaldeep, I have tried your code but arrived into a problem. The code is working fine in local mode. But if I run the same code in Spark stand alone mode or YARN mode, then it is continuously executing, but not saving anything in the HBase table. I guess, it is stopping data streaming once the saveToHBase method is called for the first time. This is strange. I just want to know whether you have tested the code on all Spark execution modes? Thanks, Gaurav On Tue, Jun 10, 2014 at 12:20 PM, Kanwaldeep [via Apache Spark User List] [hidden email] (/user/SendEmail.jtp?type=nodenode=7389i=0) wrote: Please see sample code attached at https://issues.apache.org/jira/browse/SPARK-944. If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7305.html To start a new topic under Apache Spark User List, email [hidden email] (/user/SendEmail.jtp?type=nodenode=7389i=1) To unsubscribe from Apache Spark User List, click here. NAML (http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml) View this message in context: Re: Writing data to HBase using Spark (http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7389.html) Sent from the Apache Spark User List mailing list archive (http://apache-spark-user-list.1001560.n3.nabble.com/) at Nabble.com (http://Nabble.com).
Writing data to HBase using Spark
Hi, I am reading data from a HBase table to RDD and then using foreach on that RDD I am doing some processing on every Result of HBase table. After this processing I want to store the processed data back to another HBase table. How can I do that ? If I use standard Hadoop and HBase classes to write data to HBase I fall into serialization issues. How should I write data to HBase in this case? Thanks, -Vibhor
Re: Writing data to HBase using Spark
Please see sample code attached at https://issues.apache.org/jira/browse/SPARK-944. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Writing-data-to-HBase-using-Spark-tp7304p7305.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
