Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-03 Thread Serega Sheypak 
Ok, I got it. Thank you!

2014-11-03 2:20 GMT+03:00 Sean Busbey bus...@cloudera.com:

 On Sun, Nov 2, 2014 at 5:09 PM, Ted Yu yuzhih...@gmail.com wrote:

  bq. context.write(hbaseKey, put); //Exception here
 
  I am not mrunit expert. But as long as you call the following method
 prior
  to the above method invocation, you should be able to proceed:
 
  conf.setStrings(io.serializations, conf.get(io.serializations),
 
  MutationSerialization.class.getName(), ResultSerialization.class
  .getName(),
 
  KeyValueSerialization.class.getName());
 
 

 Those classes are not a part of the public HBase API, so directly
 referencing them is a bad idea. Doing so just sets them up to break on some
 future HBase upgrade.

 The OP needs a place in MRUnit to call one of
 HFileOutputFormat.configureIncrementalLoad,
 HFileOutputFormat2.configureIncrementalLoad, or
 TableMapReduceUtil.initTableReducerJob. Those are the only public API ways
 to configure the needed Serialization.

 --
 Sean

No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Serega Sheypak 
Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
I had a unit test for mapper used to create HFile and bulk load later.

I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
Now I've started to get exception

java.lang.IllegalStateException: No applicable class implementing
Serialization in conf at io.serializations: class
org.apache.hadoop.hbase.client.Put
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
at
org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at
org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
at
org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
at
org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
at
org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at
org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
at
org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)


And here is mapper code:



public class ItemRecommendationHBaseMapper extends MapperLongWritable,
BytesWritable, ImmutableBytesWritable, Put {

private final ImmutableBytesWritable hbaseKey = new
ImmutableBytesWritable();
private final DynamicObjectSerDeItemRecommendation serde = new
DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);

@Override
protected void map(LongWritable key, BytesWritable value, Context
context) throws IOException, InterruptedException {
checkPreconditions(key, value);
hbaseKey.set(Bytes.toBytes(key.get()));

ItemRecommendation item = serde.deserialize(value.getBytes());
checkPreconditions(item);
Put put = PutFactory.createPut(serde, item, getColumnFamily());

context.write(hbaseKey, put); //Exception here
}

Whatcan i do in order to make unit-test pass?

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Ted Yu 
bq. PutFactory.createPut(

Can you reveal how PutFactory creates the Put ?

Thanks

On Sun, Nov 2, 2014 at 1:02 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

 Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
 I had a unit test for mapper used to create HFile and bulk load later.

 I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
 Now I've started to get exception

 java.lang.IllegalStateException: No applicable class implementing
 Serialization in conf at io.serializations: class
 org.apache.hadoop.hbase.client.Put
 at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
 at

 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
 at

 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
 at

 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
 at

 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
 at

 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
 at

 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
 at

 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
 at

 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
 at

 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
 at

 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)


 And here is mapper code:



 public class ItemRecommendationHBaseMapper extends MapperLongWritable,
 BytesWritable, ImmutableBytesWritable, Put {

 private final ImmutableBytesWritable hbaseKey = new
 ImmutableBytesWritable();
 private final DynamicObjectSerDeItemRecommendation serde = new
 DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);

 @Override
 protected void map(LongWritable key, BytesWritable value, Context
 context) throws IOException, InterruptedException {
 checkPreconditions(key, value);
 hbaseKey.set(Bytes.toBytes(key.get()));

 ItemRecommendation item = serde.deserialize(value.getBytes());
 checkPreconditions(item);
 Put put = PutFactory.createPut(serde, item, getColumnFamily());

 context.write(hbaseKey, put); //Exception here
 }

 Whatcan i do in order to make unit-test pass?

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Serega Sheypak 
 public static Put createPut(DynamicObjectSerDeItemRecommendation serde,
ItemRecommendation item, String columnFamily){
Put put = new Put(Bytes.toBytes(Long.valueOf(item.getId(;
put.add(Bytes.toBytes(columnFamily), Bytes.toBytes(item.getRank()),
serde.serialize(item));
return put;
}

2014-11-03 0:12 GMT+03:00 Ted Yu yuzhih...@gmail.com:

 bq. PutFactory.createPut(

 Can you reveal how PutFactory creates the Put ?

 Thanks

 On Sun, Nov 2, 2014 at 1:02 PM, Serega Sheypak serega.shey...@gmail.com
 wrote:

  Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
  I had a unit test for mapper used to create HFile and bulk load later.
 
  I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
  Now I've started to get exception
 
  java.lang.IllegalStateException: No applicable class implementing
  Serialization in conf at io.serializations: class
  org.apache.hadoop.hbase.client.Put
  at
 com.google.common.base.Preconditions.checkState(Preconditions.java:149)
  at
 
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
  at
 
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
  at
 
 
 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
  at
 
 
 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
  at
 
 
 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
  at
 
 
 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
  at
 
 
 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
  at
 
 
 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
  at
 
 
 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
  at
 
 
 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)
 
 
  And here is mapper code:
 
 
 
  public class ItemRecommendationHBaseMapper extends MapperLongWritable,
  BytesWritable, ImmutableBytesWritable, Put {
 
  private final ImmutableBytesWritable hbaseKey = new
  ImmutableBytesWritable();
  private final DynamicObjectSerDeItemRecommendation serde = new
  DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);
 
  @Override
  protected void map(LongWritable key, BytesWritable value, Context
  context) throws IOException, InterruptedException {
  checkPreconditions(key, value);
  hbaseKey.set(Bytes.toBytes(key.get()));
 
  ItemRecommendation item = serde.deserialize(value.getBytes());
  checkPreconditions(item);
  Put put = PutFactory.createPut(serde, item, getColumnFamily());
 
  context.write(hbaseKey, put); //Exception here
  }
 
  Whatcan i do in order to make unit-test pass?

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Sean Busbey 
In the 0.94.x API, Put implemented Writable[1]. This meant that MR code,
like yours, could use it as a Key or Value between Mapper and Reducer.

In 0.96 and later APIs, Put no longer directly implements Writable[2].
Instead, HBase now includes a Hadoop Seriazliation implementation.
Normally, this would be configured via the TableMapReduceUtil class for
either a TableMapper or TableReducer.

Presuming that the intention of your MR job is to have all the Puts write
to some HBase table, you should be able to follow the write to HBase part
of the examples for reading and writing HBase via mapreduce in the
reference guide[3].

Specifically, you should have your Driver call one of the
initTableReducerJob methods on TableMapReduceUtil, where it currently sets
the Mapper class for your application[4].

-Sean

[1]:
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
[2]: http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
[3]: http://hbase.apache.org/book/mapreduce.example.html
[4]:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html


On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

 Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
 I had a unit test for mapper used to create HFile and bulk load later.

 I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
 Now I've started to get exception

 java.lang.IllegalStateException: No applicable class implementing
 Serialization in conf at io.serializations: class
 org.apache.hadoop.hbase.client.Put
 at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
 at

 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
 at

 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
 at

 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
 at

 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
 at

 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
 at

 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
 at

 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
 at

 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
 at

 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
 at

 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)


 And here is mapper code:



 public class ItemRecommendationHBaseMapper extends MapperLongWritable,
 BytesWritable, ImmutableBytesWritable, Put {

 private final ImmutableBytesWritable hbaseKey = new
 ImmutableBytesWritable();
 private final DynamicObjectSerDeItemRecommendation serde = new
 DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);

 @Override
 protected void map(LongWritable key, BytesWritable value, Context
 context) throws IOException, InterruptedException {
 checkPreconditions(key, value);
 hbaseKey.set(Bytes.toBytes(key.get()));

 ItemRecommendation item = serde.deserialize(value.getBytes());
 checkPreconditions(item);
 Put put = PutFactory.createPut(serde, item, getColumnFamily());

 context.write(hbaseKey, put); //Exception here
 }

 Whatcan i do in order to make unit-test pass?




-- 
Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Ted Yu 
Since your code uses mrunit which hbase 0.98 is not dependent on, have you
considered asking this question on:
mrunit-u...@incubator.apache.org

Cheers

On Sun, Nov 2, 2014 at 1:22 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

  public static Put createPut(DynamicObjectSerDeItemRecommendation serde,
 ItemRecommendation item, String columnFamily){
 Put put = new Put(Bytes.toBytes(Long.valueOf(item.getId(;
 put.add(Bytes.toBytes(columnFamily), Bytes.toBytes(item.getRank()),
 serde.serialize(item));
 return put;
 }

 2014-11-03 0:12 GMT+03:00 Ted Yu yuzhih...@gmail.com:

  bq. PutFactory.createPut(
 
  Can you reveal how PutFactory creates the Put ?
 
  Thanks
 
  On Sun, Nov 2, 2014 at 1:02 PM, Serega Sheypak serega.shey...@gmail.com
 
  wrote:
 
   Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
   I had a unit test for mapper used to create HFile and bulk load later.
  
   I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
   Now I've started to get exception
  
   java.lang.IllegalStateException: No applicable class implementing
   Serialization in conf at io.serializations: class
   org.apache.hadoop.hbase.client.Put
   at
  com.google.common.base.Preconditions.checkState(Preconditions.java:149)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
   at
  
  
 
 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
   at
  
  
 
 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
   at
  
  
 
 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
   at
  
  
 
 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
   at
  
  
 
 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
   at
  
  
 
 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)
  
  
   And here is mapper code:
  
  
  
   public class ItemRecommendationHBaseMapper extends MapperLongWritable,
   BytesWritable, ImmutableBytesWritable, Put {
  
   private final ImmutableBytesWritable hbaseKey = new
   ImmutableBytesWritable();
   private final DynamicObjectSerDeItemRecommendation serde = new
   DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);
  
   @Override
   protected void map(LongWritable key, BytesWritable value, Context
   context) throws IOException, InterruptedException {
   checkPreconditions(key, value);
   hbaseKey.set(Bytes.toBytes(key.get()));
  
   ItemRecommendation item = serde.deserialize(value.getBytes());
   checkPreconditions(item);
   Put put = PutFactory.createPut(serde, item, getColumnFamily());
  
   context.write(hbaseKey, put); //Exception here
   }
  
   Whatcan i do in order to make unit-test pass?

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Serega Sheypak 
I use it to prepare HFile using my custom mapper emitting Put and
  HFileOutputFormat.configureIncrementalLoad(job, createHTable())
//connection to target table

and then bulk load data to table using LoadIncrementalHFiles

P.S.
HFileOutputFormat is also deprecated... so many changes... (((


2014-11-03 0:41 GMT+03:00 Sean Busbey bus...@cloudera.com:

 In the 0.94.x API, Put implemented Writable[1]. This meant that MR code,
 like yours, could use it as a Key or Value between Mapper and Reducer.

 In 0.96 and later APIs, Put no longer directly implements Writable[2].
 Instead, HBase now includes a Hadoop Seriazliation implementation.
 Normally, this would be configured via the TableMapReduceUtil class for
 either a TableMapper or TableReducer.

 Presuming that the intention of your MR job is to have all the Puts write
 to some HBase table, you should be able to follow the write to HBase part
 of the examples for reading and writing HBase via mapreduce in the
 reference guide[3].

 Specifically, you should have your Driver call one of the
 initTableReducerJob methods on TableMapReduceUtil, where it currently sets
 the Mapper class for your application[4].

 -Sean

 [1]:

 http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
 [2]:
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
 [3]: http://hbase.apache.org/book/mapreduce.example.html
 [4]:

 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html


 On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak serega.shey...@gmail.com
 wrote:

  Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
  I had a unit test for mapper used to create HFile and bulk load later.
 
  I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
  Now I've started to get exception
 
  java.lang.IllegalStateException: No applicable class implementing
  Serialization in conf at io.serializations: class
  org.apache.hadoop.hbase.client.Put
  at
 com.google.common.base.Preconditions.checkState(Preconditions.java:149)
  at
 
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
  at
 
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
  at
 
 
 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
  at
 
 
 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
  at
 
 
 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
  at
 
 
 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
  at
 
 
 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
  at
 
 
 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
  at
 
 
 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
  at
 
 
 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)
 
 
  And here is mapper code:
 
 
 
  public class ItemRecommendationHBaseMapper extends MapperLongWritable,
  BytesWritable, ImmutableBytesWritable, Put {
 
  private final ImmutableBytesWritable hbaseKey = new
  ImmutableBytesWritable();
  private final DynamicObjectSerDeItemRecommendation serde = new
  DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);
 
  @Override
  protected void map(LongWritable key, BytesWritable value, Context
  context) throws IOException, InterruptedException {
  checkPreconditions(key, value);
  hbaseKey.set(Bytes.toBytes(key.get()));
 
  ItemRecommendation item = serde.deserialize(value.getBytes());
  checkPreconditions(item);
  Put put = PutFactory.createPut(serde, item, getColumnFamily());
 
  context.write(hbaseKey, put); //Exception here
  }
 
  Whatcan i do in order to make unit-test pass?
 



 --
 Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Sean Busbey 
If you're calling HFileOutputFormat.configureIncrementalLoad, that should
be setting up the Serialization for you.

Can you look at the job configuration and see what's present for the key
io.serializations?

-Sean

On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

 I use it to prepare HFile using my custom mapper emitting Put and
   HFileOutputFormat.configureIncrementalLoad(job, createHTable())
 //connection to target table

 and then bulk load data to table using LoadIncrementalHFiles

 P.S.
 HFileOutputFormat is also deprecated... so many changes... (((


 2014-11-03 0:41 GMT+03:00 Sean Busbey bus...@cloudera.com:

  In the 0.94.x API, Put implemented Writable[1]. This meant that MR code,
  like yours, could use it as a Key or Value between Mapper and Reducer.
 
  In 0.96 and later APIs, Put no longer directly implements Writable[2].
  Instead, HBase now includes a Hadoop Seriazliation implementation.
  Normally, this would be configured via the TableMapReduceUtil class for
  either a TableMapper or TableReducer.
 
  Presuming that the intention of your MR job is to have all the Puts write
  to some HBase table, you should be able to follow the write to HBase
 part
  of the examples for reading and writing HBase via mapreduce in the
  reference guide[3].
 
  Specifically, you should have your Driver call one of the
  initTableReducerJob methods on TableMapReduceUtil, where it currently
 sets
  the Mapper class for your application[4].
 
  -Sean
 
  [1]:
 
 
 http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
  [2]:
  http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
  [3]: http://hbase.apache.org/book/mapreduce.example.html
  [4]:
 
 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
 
 
  On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak serega.shey...@gmail.com
 
  wrote:
 
   Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
   I had a unit test for mapper used to create HFile and bulk load later.
  
   I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
   Now I've started to get exception
  
   java.lang.IllegalStateException: No applicable class implementing
   Serialization in conf at io.serializations: class
   org.apache.hadoop.hbase.client.Put
   at
  com.google.common.base.Preconditions.checkState(Preconditions.java:149)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
   at
  
  
 
 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
   at
  
  
 
 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
   at
  
  
 
 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
   at
  
  
 
 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
   at
  
  
 
 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
   at
  
  
 
 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
   at
  
  
 
 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)
  
  
   And here is mapper code:
  
  
  
   public class ItemRecommendationHBaseMapper extends MapperLongWritable,
   BytesWritable, ImmutableBytesWritable, Put {
  
   private final ImmutableBytesWritable hbaseKey = new
   ImmutableBytesWritable();
   private final DynamicObjectSerDeItemRecommendation serde = new
   DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);
  
   @Override
   protected void map(LongWritable key, BytesWritable value, Context
   context) throws IOException, InterruptedException {
   checkPreconditions(key, value);
   hbaseKey.set(Bytes.toBytes(key.get()));
  
   ItemRecommendation item = serde.deserialize(value.getBytes());
   checkPreconditions(item);
   Put put = PutFactory.createPut(serde, item, getColumnFamily());
  
   context.write(hbaseKey, put); //Exception here
   }
  
   Whatcan i do in order to make unit-test pass?
  
 
 
 
  --
  Sean
 




-- 
Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Sean Busbey 
On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

 P.S.
 HFileOutputFormat is also deprecated... so many changes... (((



Incidentally, you should consider switching to HFileOutputFormat2, since
you rely on the version that has a Mapper outputting Put values instead of
KeyValue the impact on you should be negligible.


-- 
Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Serega Sheypak 
Sean, I've started to catch that serialization problem on unit-test level
while using mrunit.
I don't see any possibility to call HFileOutputFormat.configureIncrementalLoad
before mrunit mocking stuff.
I was workngi w/o any problem in 0.94 :)


2014-11-03 1:08 GMT+03:00 Sean Busbey bus...@cloudera.com:

 If you're calling HFileOutputFormat.configureIncrementalLoad, that should
 be setting up the Serialization for you.

 Can you look at the job configuration and see what's present for the key
 io.serializations?

 -Sean

 On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak serega.shey...@gmail.com
 wrote:

  I use it to prepare HFile using my custom mapper emitting Put and
HFileOutputFormat.configureIncrementalLoad(job, createHTable())
  //connection to target table
 
  and then bulk load data to table using LoadIncrementalHFiles
 
  P.S.
  HFileOutputFormat is also deprecated... so many changes... (((
 
 
  2014-11-03 0:41 GMT+03:00 Sean Busbey bus...@cloudera.com:
 
   In the 0.94.x API, Put implemented Writable[1]. This meant that MR
 code,
   like yours, could use it as a Key or Value between Mapper and Reducer.
  
   In 0.96 and later APIs, Put no longer directly implements Writable[2].
   Instead, HBase now includes a Hadoop Seriazliation implementation.
   Normally, this would be configured via the TableMapReduceUtil class for
   either a TableMapper or TableReducer.
  
   Presuming that the intention of your MR job is to have all the Puts
 write
   to some HBase table, you should be able to follow the write to HBase
  part
   of the examples for reading and writing HBase via mapreduce in the
   reference guide[3].
  
   Specifically, you should have your Driver call one of the
   initTableReducerJob methods on TableMapReduceUtil, where it currently
  sets
   the Mapper class for your application[4].
  
   -Sean
  
   [1]:
  
  
 
 http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
   [2]:
  
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
   [3]: http://hbase.apache.org/book/mapreduce.example.html
   [4]:
  
  
 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
  
  
   On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak 
 serega.shey...@gmail.com
  
   wrote:
  
Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
I had a unit test for mapper used to create HFile and bulk load
 later.
   
I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
Now I've started to get exception
   
java.lang.IllegalStateException: No applicable class implementing
Serialization in conf at io.serializations: class
org.apache.hadoop.hbase.client.Put
at
   com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
   
   
  
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
at
   
   
  
 
 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at
   
   
  
 
 org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
at
   
   
  
 
 org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
at
   
   
  
 
 org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
at
   
   
  
 
 org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
at
   
   
  
 
 org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at
   
   
  
 
 org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
at
   
   
  
 
 org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
at
   
   
  
 
 org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(generated)
   
   
And here is mapper code:
   
   
   
public class ItemRecommendationHBaseMapper extends
 MapperLongWritable,
BytesWritable, ImmutableBytesWritable, Put {
   
private final ImmutableBytesWritable hbaseKey = new
ImmutableBytesWritable();
private final DynamicObjectSerDeItemRecommendation serde = new
DynamicObjectSerDeItemRecommendation(ItemRecommendation.class);
   
@Override
protected void map(LongWritable key, BytesWritable value, Context
context) throws IOException, InterruptedException {
checkPreconditions(key, value);
hbaseKey.set(Bytes.toBytes(key.get()));
   
ItemRecommendation item =
 serde.deserialize(value.getBytes());
checkPreconditions(item);
Put put = PutFactory.createPut(serde, item,
 getColumnFamily());
   
context.write(hbaseKey, put); //Exception here
}
   
Whatcan i do in order to make unit-test pass?
   
  
  
  
   --
   Sean
  
 



 --
 Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Serega Sheypak 
Cool, is it this stuff?
http://hbase.apache.org/book/hfilev2.html

2014-11-03 1:10 GMT+03:00 Sean Busbey bus...@cloudera.com:

 On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak serega.shey...@gmail.com
 wrote:

  P.S.
  HFileOutputFormat is also deprecated... so many changes... (((
 
 

 Incidentally, you should consider switching to HFileOutputFormat2, since
 you rely on the version that has a Mapper outputting Put values instead of
 KeyValue the impact on you should be negligible.


 --
 Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Sean Busbey 
On Sun, Nov 2, 2014 at 4:11 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

 Sean, I've started to catch that serialization problem on unit-test level
 while using mrunit.
 I don't see any possibility to call
 HFileOutputFormat.configureIncrementalLoad
 before mrunit mocking stuff.
 I was workngi w/o any problem in 0.94 :)



Ah. Well, that sounds like a bug in MRUnit for dealing with HBase 0.96+.

As Ted mentioned, you'll have more luck on the mrunit mailing list for
figuring that bit out.

-- 
Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Sean Busbey 
On Sun, Nov 2, 2014 at 4:14 PM, Serega Sheypak serega.shey...@gmail.com
wrote:

 Cool, is it this stuff?
 http://hbase.apache.org/book/hfilev2.html


No, that's all details on the update to the backing HFile format that
started in HBase 0.92. The change in output format is detailed here:

http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.html

It's basically just that the output format changed from using the old
private KeyValue class to the public one for Cell.

-- 
Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Ted Yu 
bq. context.write(hbaseKey, put); //Exception here

I am not mrunit expert. But as long as you call the following method prior
to the above method invocation, you should be able to proceed:

conf.setStrings(io.serializations, conf.get(io.serializations),

MutationSerialization.class.getName(), ResultSerialization.class
.getName(),

KeyValueSerialization.class.getName());

Cheers

On Sun, Nov 2, 2014 at 2:24 PM, Sean Busbey bus...@cloudera.com wrote:

 On Sun, Nov 2, 2014 at 4:11 PM, Serega Sheypak serega.shey...@gmail.com
 wrote:

  Sean, I've started to catch that serialization problem on unit-test level
  while using mrunit.
  I don't see any possibility to call
  HFileOutputFormat.configureIncrementalLoad
  before mrunit mocking stuff.
  I was workngi w/o any problem in 0.94 :)
 
 
 
 Ah. Well, that sounds like a bug in MRUnit for dealing with HBase 0.96+.

 As Ted mentioned, you'll have more luck on the mrunit mailing list for
 figuring that bit out.

 --
 Sean

Re: No applicable class implementing Serialization in conf at io.serializations: class org.apache.hadoop.hbase.client.Put

2014-11-02 Thread Sean Busbey 
On Sun, Nov 2, 2014 at 5:09 PM, Ted Yu yuzhih...@gmail.com wrote:

 bq. context.write(hbaseKey, put); //Exception here

 I am not mrunit expert. But as long as you call the following method prior
 to the above method invocation, you should be able to proceed:

 conf.setStrings(io.serializations, conf.get(io.serializations),

 MutationSerialization.class.getName(), ResultSerialization.class
 .getName(),

 KeyValueSerialization.class.getName());



Those classes are not a part of the public HBase API, so directly
referencing them is a bad idea. Doing so just sets them up to break on some
future HBase upgrade.

The OP needs a place in MRUnit to call one of
HFileOutputFormat.configureIncrementalLoad,
HFileOutputFormat2.configureIncrementalLoad, or
TableMapReduceUtil.initTableReducerJob. Those are the only public API ways
to configure the needed Serialization.

-- 
Sean