Can you pastebin log snippet showing hbase related errors ? Please also consider posting the question on vendor's forum.
On Thu, Oct 8, 2015 at 10:17 PM, roywang1024 <roywang1...@163.com> wrote: > > I add hbase-conf-dir to spark/conf/classpath.txt,but still error. > > > > > > At 2015-10-09 11:04:35, "Ted Yu" <yuzhih...@gmail.com> wrote: > > The second code snippet is similar to: > examples//src/main/scala/org/apache/spark/examples/HBaseTest.scala > > See the comment in HBaseTest.scala : > // please ensure HBASE_CONF_DIR is on classpath of spark driver > // e.g: set it through spark.driver.extraClassPath property > // in spark-defaults.conf or through --driver-class-path > // command line option of spark-submit > > If during execution of TableInputFormatBase#initializeTable(), there was > exception, table field might not have been initialized. > > FYI > > On Thu, Oct 8, 2015 at 7:54 PM, roywang1024 <roywang1...@163.com> wrote: > >> >> I have try this >> >> SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark"); >> JavaSparkContext sc = new JavaSparkContext(sparkConf); >> Configuration conf = HBaseConfiguration.create(); >> Scan scan = new Scan(); >> scan.addFamily(Bytes.toBytes("InnerCode")); >> scan.addColumn(Bytes.toBytes("InnerCode"), Bytes.toBytes("")); >> conf.set(TableInputFormat.INPUT_TABLE, "SecuMain"); >> conf.set(TableInputFormat.SCAN, convertScanToString(scan)); >> >> and this >> >> SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark"); >> JavaSparkContext sc = new JavaSparkContext(sparkConf); >> Configuration conf = HBaseConfiguration.create(); >> String tableName = "SecuMain"; >> conf.set(TableInputFormat.INPUT_TABLE, tableName); >> >> also can't wok! >> >> Should I add hbase-site.xml to conf? >> >> Thanks. >> >> >> >> >> >> >> At 2015-10-09 10:35:16, "Ted Yu" <yuzhih...@gmail.com> wrote: >> >> One possibility was that hbase config, including hbase.zookeeper.quorum, >> was not passed to your job. >> hbase-site.xml should be on the classpath. >> >> Can you show snippet of your code ? >> >> Looks like you were running against hbase 1.x >> >> Cheers >> >> On Thu, Oct 8, 2015 at 7:29 PM, Roy Wang <roywang1...@163.com> wrote: >> >>> >>> I want to load hbase table into spark. >>> JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD = >>> sc.newAPIHadoopRDD(conf, TableInputFormat.class, >>> ImmutableBytesWritable.class, Result.class); >>> >>> *when call hBaseRDD.count(),got error.* >>> >>> Caused by: java.lang.IllegalStateException: The input format instance has >>> not been properly initialized. Ensure you call initializeTable either in >>> your constructor or initialize method >>> at >>> >>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:389) >>> at >>> >>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:158) >>> ... 11 more >>> >>> *But when job start,I can get these logs* >>> 2015-10-09 09:17:00[main] WARN TableInputFormatBase:447 - >>> initializeTable >>> called multiple times. Overwriting connection and table reference; >>> TableInputFormatBase will not close these old references when done. >>> >>> Does anyone know how does this happen? >>> >>> Thanks! >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Error-in-load-hbase-on-spark-tp24986.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >> >> >> > > > > >