Re: HDP 2.5 - Python - Spark-On-Hbase

[Weiqing Yang]

For SHC documentation, please refer the README in SHC github, which is kept
up-to-date.

On Mon, Jun 26, 2017 at 5:46 AM, ayan guha <guha.a...@gmail.com> wrote:

> Thanks all, I have found correct version of the package. Probably HDP
> documentation is little behind.
>
> Best
> Ayan
>
> On Mon, 26 Jun 2017 at 2:16 pm, Mahesh Sawaiker <
> mahesh_sawai...@persistent.com> wrote:
>
>> Ayan,
>>
>> The location of the logging class was moved from Spark 1.6 to Spark 2.0.
>>
>> Looks like you are trying to run 1.6 code on 2.0, I have ported some code
>> like this before and if you have access to the code you can recompile it by
>> changing reference to Logging class and directly use the slf4 Logger class,
>> most of the code tends to be easily portable.
>>
>>
>>
>> Following is the release note for Spark 2.0
>>
>>
>>
>> *Removals, Behavior Changes and Deprecations*
>>
>> *Removals*
>>
>> The following features have been removed in Spark 2.0:
>>
>>- Bagel
>>- Support for Hadoop 2.1 and earlier
>>- The ability to configure closure serializer
>>- HTTPBroadcast
>>- TTL-based metadata cleaning
>>- *Semi-private class org.apache.spark.Logging. We suggest you use
>>    slf4j directly.*
>>- SparkContext.metricsSystem
>>
>> Thanks,
>>
>> Mahesh
>>
>>
>>
>>
>>
>> *From:* ayan guha [mailto:guha.a...@gmail.com]
>> *Sent:* Monday, June 26, 2017 6:26 AM
>> *To:* Weiqing Yang
>> *Cc:* user
>> *Subject:* Re: HDP 2.5 - Python - Spark-On-Hbase
>>
>>
>>
>> Hi
>>
>>
>>
>> I am using following:
>>
>>
>>
>> --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories
>> http://repo.hortonworks.com/content/groups/public/
>>
>>
>>
>> Is it compatible with Spark 2.X? I would like to use it
>>
>>
>>
>> Best
>>
>> Ayan
>>
>>
>>
>> On Sat, Jun 24, 2017 at 2:09 AM, Weiqing Yang <yangweiqing...@gmail.com>
>> wrote:
>>
>> Yes.
>>
>> What SHC version you were using?
>>
>> If hitting any issues, you can post them in SHC github issues. There are
>> some threads about this.
>>
>>
>>
>> On Fri, Jun 23, 2017 at 5:46 AM, ayan guha <guha.a...@gmail.com> wrote:
>>
>> Hi
>>
>>
>>
>> Is it possible to use SHC from Hortonworks with pyspark? If so, any
>> working code sample available?
>>
>>
>>
>> Also, I faced an issue while running the samples with Spark 2.0
>>
>>
>>
>> "Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging"
>>
>>
>>
>> Any workaround?
>>
>>
>>
>> Thanks in advance
>>
>>
>>
>> --
>>
>> Best Regards,
>> Ayan Guha
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Best Regards,
>> Ayan Guha
>> DISCLAIMER
>> ==
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
> --
> Best Regards,
> Ayan Guha
>

Re: HDP 2.5 - Python - Spark-On-Hbase

[Weiqing Yang]

Yes.
What SHC version you were using?
If hitting any issues, you can post them in SHC github issues. There are
some threads about this.

On Fri, Jun 23, 2017 at 5:46 AM, ayan guha  wrote:

> Hi
>
> Is it possible to use SHC from Hortonworks with pyspark? If so, any
> working code sample available?
>
> Also, I faced an issue while running the samples with Spark 2.0
>
> "Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging"
>
> Any workaround?
>
> Thanks in advance
>
> --
> Best Regards,
> Ayan Guha
>

Re: Graph Analytics on HBase with HGraphDB and Spark GraphFrames

[Weiqing Yang]

Thanks for sharing this.


On Sun, Apr 2, 2017 at 7:08 PM, Irving Duran  wrote:

> Thanks for the share!
>
>
> Thank You,
>
> Irving Duran
>
> On Sun, Apr 2, 2017 at 7:19 PM, Felix Cheung 
> wrote:
>
>> Interesting!
>>
>> --
>> *From:* Robert Yokota 
>> *Sent:* Sunday, April 2, 2017 9:40:07 AM
>> *To:* user@spark.apache.org
>> *Subject:* Graph Analytics on HBase with HGraphDB and Spark GraphFrames
>>
>> Hi,
>>
>> In case anyone is interested in analyzing graphs in HBase with Apache
>> Spark GraphFrames, this might be helpful:
>>
>> https://yokota.blog/2017/04/02/graph-analytics-on-hbase-with
>> -hgraphdb-and-spark-graphframes/
>>
>
>

Re: Issues with Spark On Hbase Connector and versions

[Weiqing Yang]

The PR will be reviewed soon.

Thanks,
Weiqing

From: Sachin Jain >
Date: Sunday, August 28, 2016 at 11:12 PM
To: spats >
Cc: user >
Subject: Re: Issues with Spark On Hbase Connector and versions

There is connection leak problem with hortonworks hbase connector if you use 
hbase 1.2.0.
I tried to use hortonwork's connector and felt into the same problem.

Have a look at this Hbase issue HBASE-16017 [0]. The fix for this was 
backported to 1.3.0, 1.4.0 and 2.0.0
I have raised a ticket on their github repo [1] and also generated PR to get 
this fixed. Check this out [2].

But unfortunately no one has responded to it yet so it is not merged. But this 
fix works and I am currently using the same without problems.
So if you want to use this. You can use this one [3] where changes for pull 
request already there.

Hope it helps!!

[0]: https://issues.apache.org/jira/browse/HBASE-16017
[1]: https://github.com/hortonworks-spark/shc/issues/19
[2]: https://github.com/hortonworks-spark/shc/pull/20
[3]: https://github.com/sachinjain024/shc/tree/Issue-19-Connection-Leak

PS: Cross posting my answer from hbase user mailing list because I think it may 
be helpful to other readers.

On Sat, Aug 27, 2016 at 5:17 PM, spats 
> wrote:
Regarding hbase connector by hortonworks
https://github.com/hortonworks-spark/shc, it would be great if someone can
answer these

1. What versions of Hbase & Spark expected? I could not run examples
provided using spark 1.6.0 & hbase 1.2.0
2. I get error when i run example provided  here

, any pointers on what i am doing wrong?

looks like spark not reading hbase-site.xml, but passed it in --files while
spark-shell
e.g --files
/etc/hbase/conf/hbase-site.xml,/etc/hbase/conf/hdfs-site.xml,/etc/hbase/conf/core-site.xml

error
16/08/27 12:35:00 WARN zookeeper.ClientCnxn: Session 0x0 for server null,
unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Issues-with-Spark-On-Hbase-Connector-and-versions-tp27610.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: 
user-unsubscr...@spark.apache.org