Can you pastebin log snippet showing hbase related errors ?
Please also consider posting the question on vendor's forum.
On Thu, Oct 8, 2015 at 10:17 PM, roywang1024 <roywang1...@163.com> wrote:
>
> I add hbase-conf-dir to spark/conf/classpath.txt,but still error.
>
>
>
>
>
> At 2015-10-09 11:04:35, "Ted Yu" <yuzhih...@gmail.com> wrote:
>
> The second code snippet is similar to:
> examples//src/main/scala/org/apache/spark/examples/HBaseTest.scala
>
> See the comment in HBaseTest.scala :
> // please ensure HBASE_CONF_DIR is on classpath of spark driver
> // e.g: set it through spark.driver.extraClassPath property
> // in spark-defaults.conf or through --driver-class-path
> // command line option of spark-submit
>
> If during execution of TableInputFormatBase#initializeTable(), there was
> exception, table field might not have been initialized.
>
> FYI
>
> On Thu, Oct 8, 2015 at 7:54 PM, roywang1024 <roywang1...@163.com> wrote:
>
>>
>> I have try this
>>
>> SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark");
>> JavaSparkContext sc = new JavaSparkContext(sparkConf);
>> Configuration conf = HBaseConfiguration.create();
>> Scan scan = new Scan();
>> scan.addFamily(Bytes.toBytes("InnerCode"));
>> scan.addColumn(Bytes.toBytes("InnerCode"), Bytes.toBytes(""));
>> conf.set(TableInputFormat.INPUT_TABLE, "SecuMain");
>> conf.set(TableInputFormat.SCAN, convertScanToString(scan));
>>
>> and this
>>
>> SparkConf sparkConf = new SparkConf().setAppName("HBaseIntoSpark");
>> JavaSparkContext sc = new JavaSparkContext(sparkConf);
>> Configuration conf = HBaseConfiguration.create();
>> String tableName = "SecuMain";
>> conf.set(TableInputFormat.INPUT_TABLE, tableName);
>>
>> also can't wok!
>>
>> Should I add hbase-site.xml to conf?
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>> At 2015-10-09 10:35:16, "Ted Yu" <yuzhih...@gmail.com> wrote:
>>
>> One possibility was that hbase config, including hbase.zookeeper.quorum,
>> was not passed to your job.
>> hbase-site.xml should be on the classpath.
>>
>> Can you show snippet of your code ?
>>
>> Looks like you were running against hbase 1.x
>>
>> Cheers
>>
>> On Thu, Oct 8, 2015 at 7:29 PM, Roy Wang <roywang1...@163.com> wrote:
>>
>>>
>>> I want to load hbase table into spark.
>>> JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD =
>>> sc.newAPIHadoopRDD(conf, TableInputFormat.class,
>>> ImmutableBytesWritable.class, Result.class);
>>>
>>> *when call hBaseRDD.count(),got error.*
>>>
>>> Caused by: java.lang.IllegalStateException: The input format instance has
>>> not been properly initialized. Ensure you call initializeTable either in
>>> your constructor or initialize method
>>> at
>>>
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:389)
>>> at
>>>
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:158)
>>> ... 11 more
>>>
>>> *But when job start,I can get these logs*
>>> 2015-10-09 09:17:00[main] WARN TableInputFormatBase:447 -
>>> initializeTable
>>> called multiple times. Overwriting connection and table reference;
>>> TableInputFormatBase will not close these old references when done.
>>>
>>> Does anyone know how does this happen?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Error-in-load-hbase-on-spark-tp24986.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>
>>
>>
>>
>
>
>
>
>
