Re: Spark Streaming checkpoint recovery failure for HBaseContext

2017-01-27 Thread Sean Busbey
HBaseContext makes use of broadcast variables, so to use it with checkpointing in a spark streaming application you have to wrap it in a singleton that can be recreated on restore from the checkpoint. I have an example of this I was working on but have not finished for the hbase-downstreamer

Re: Spark Streaming checkpoint recovery failure for HBaseContext

2017-01-27 Thread Chetan Khatri
The hbase-spark module is not production ready , having some issues there are TODO: patches - https://issues.apache.org/jira/browse/HBASE-16179 - https://issues.apache.org/jira/browse/HBASE-14160 But by changing according configuration and using build.sbt i got success to get run it. would you

Spark Streaming checkpoint recovery failure for HBaseContext

2017-01-26 Thread SivaRam Bollineni
hi, First of all, Thanks a ton for HBaseContext API, it's very useful and performant for our use-cases. We use HBaseContext in Spark Streaming with checkpoint. We have a problem in recovering from checkpoint. Below is the *NullPointerException* stack trace. I am sure it's because SparkContext