[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621073#comment-14621073 ]
Sean Busbey commented on HBASE-13992: ------------------------------------- Notes from call today where [~apurtell] and I helped walk through a plan for this with [~ted.m]. ---- h3. Overall Roadmap * start with current capabilities refactored into a module with hbase packages * initial support targets Spark 1.3 * later add RDD extension to expose hbase-specific functionality * later add data frame support, likely as additional module * start with master-only, use backport patch for branch-1 post refactor and docs h3. Minimums for Landing on Master * use an isolated top-level module to reduce impact on rest of project, modeled after the phoenix-spark module ** hbase-spark ** should help limit reach of dependency conflicts. look to phoenix-spark for any conflict resolution. * target spark 1.3 as upstream dependency ** already works with 1.3 ** 1.3 seen as able to reach a large existing user base and stable ** gap: data frames are missing key functionality pre-1.4. ** gap: unknown spark compatibility promises, history of scala version changes may limit how many future versions this can work with. * move existing implementation classes into an hbase subpackage ** org.apache.hadoop.hbase.spark ** there is currently a workaround to access a spark-package-private set of configs, needed for pre-1.3. since 1.3 is our minimum expected version, remove this workaround. * leave examples as a part of the single module definition for first landing * leave tests, including scala tests, as unit tests on the single module for first landing h3. Follow on for branch-1 * documentation - need to add a ref guide section on using the spark bindings * examples refactoring - examples should be moved out of the primary module into something like hbase-spark-examples * tests refactoring - java tests should be categorized to match rest of project. Scala tests should be moved to an IT ---- > Integrate SparkOnHBase into HBase > --------------------------------- > > Key: HBASE-13992 > URL: https://issues.apache.org/jira/browse/HBASE-13992 > Project: HBase > Issue Type: Bug > Reporter: Ted Malaska > Assignee: Ted Malaska > Labels: spark > Fix For: 2.0.0 > > > This Jira is to ask if SparkOnHBase can find a home in side HBase core. > Here is the github: > https://github.com/cloudera-labs/SparkOnHBase > I am the core author of this project and the license is Apache 2.0 > A blog explaining this project is here > http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ > A spark Streaming example is here > http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ > A real customer using this in produce is blogged here > http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ > Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)