Hi guys. Thanks for helping out.
We downgraded to HBase 0.98 and resolved some classpath issues and then it worked. /Palle ----- Original meddelelse ----- > Fra: Stephan Ewen <se...@apache.org> > Til: user@flink.apache.org > Dato: Ons, 11. maj 2016 17:19 > Emne: Re: HBase write problem > > Just to narrow down the problem:The insertion into HBase actually > works, but the job does not finish after that?And the same job (same > source of data) that writes to a file, or prints, finishes?If that is > the case, can you check what status each task is in, via the web > dashboard? Are all tasks still in "running"? > > On Wed, May 11, 2016 at 4:53 PM, Flavio Pompermaier < > pomperma...@okkam.it [mailto:pomperma...@okkam.it]> wrote: > > I can't help you with the choice of the db storage, as always the > answer is "it depends" on a lot of factors :) > > For what I can tell you the problem could be that Flink support > HBase 0.98, so it could worth to update Flink connectors to a > more recent version (that should be backward compatible > hopefully..) or maybe create two separte hbase connectors (one > for hbase-0.9x and one for 0.1x). Let me know about your attempts > :) > > On Wed, May 11, 2016 at 4:47 PM, Palle < pa...@sport.dk > [mailto:pa...@sport.dk]> > wrote: > > Hadoop 2.7.2 > HBase 1.2.1 > > I have this running from a Hadoop job, but just not from > Flink. > > I will look into your suggestions, but would I be better off > choosing another DB for storage? I can see that Cassandra > gets some attention in this mailing list. I need to store app > 2 bio key value pairs consisting of 100 bytes for each pair. > > ----- Original meddelelse ----- > > Fra: Flavio Pompermaier < pomperma...@okkam.it > [mailto:pomperma...@okkam.it]> > Til: user < user@flink.apache.org [mailto:user@flink.apache.org]> > Dato: Ons, 11. maj 2016 16:29 > Emne: Re: HBase write problem > > And which version of HBase and Hadoop are you running? > Did you try to put the hbase-site.xml in the jar?Moreover, > I don't know how much reliable is at the moment the web > client UI..my experience is that the command line client > is much more reliable.You just need to run from the flink > dir something like: bin/flink run -c xxx.yyy.MyMainClass > /path/to/shadedJar.jar > On Wed, May 11, 2016 at 4:19 PM, Palle < pa...@sport.dk > [mailto:pa...@sport.dk]> > wrote: > > I run the job from the cluster. I run it through the > web UI. > The jar file submitted does not contain the > hbase-site.xml file. > > ----- Original meddelelse ----- > > Fra: Flavio Pompermaier < pomperma...@okkam.it > [mailto:pomperma...@okkam.it]> > Til: user < user@flink.apache.org [mailto:user@flink.apache.org]> > Dato: Ons, 11. maj 2016 09:36 > Emne: Re: HBase write problem > > Do you run the job from your IDE or from the > cluster? > > On Wed, May 11, 2016 at 9:22 AM, Palle < > pa...@sport.dk [mailto:pa...@sport.dk]> wrote: > > Thanks for the response, but I don't think > the problem is the classpath - hbase-site.xml > should be added. This is what it looks like > (hbase conf is added at the end): > > 2016-05-11 09:16:45,831 INFO > org.apache.zookeeper.ZooKeeper - Client > > environment:java.class.path=C:\systems\packages\flink-1.0.2\lib\flink-dist_2.11-1.0.2.jar;C:\systems\packages\flink-1.0.2\lib\flink-python_2.11-1.0.2.jar;C:\systems\packages\flink-1.0.2\lib\guava-11.0.2.jar;C:\systems\packages\flink-1.0.2\lib\hbase-annotations-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-annotations-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-client-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-common-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-common-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-examples-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-external-blockcache-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-hadoop-compat-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-hadoop2-compat-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-it-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-it-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-prefix-tree-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-procedure-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-protocol-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-resource-bundle-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-rest-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-server-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-server-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-shell-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-thrift-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\log4j-1.2.17.jar;C:\systems\packages\flink-1.0.2\lib\slf4j-log4j12-1.7.7.jar;C:\systems\master_flink\bin;C:\systems\packages\flink-1.0.2\lib;;C:\systems\packages\hbase-1.2.1\lib;C:\systems\hbase\conf;C:\systems\hbase\conf\hbase-site.xml; > 2016-05-11 09:16:45,831 INFO > org.apache.zookeeper.ZooKeeper - Client > > environment:java.library.path=C:\systems\packages\jre-1.8.0_74_x64\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\systems\master_flink\bin;C:\systems\packages\appsync-1.0.6\bin;C:\systems\packages\flink-1.0.2\bin;C:\systems\packages\jre-1.8.0_74_x64\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program > Files\Microsoft SQL > Server\120\DTS\Binn\;C:\Program > Files\Microsoft SQL Server\Client > SDK\ODBC\110\Tools\Binn\;C:\Program Files > (x86)\Microsoft SQL > Server\120\Tools\Binn\;C:\Program > Files\Microsoft SQL > Server\120\Tools\Binn\;C:\Program Files > (x86)\Microsoft SQL > Server\120\Tools\Binn\ManagementStudio\;C:\Program > Files (x86)\Microsoft SQL > > Server\120\DTS\Binn\;C:\systems\packages\hadoop-2.7.2\bin;C:\systems\packages\jdk-1.8.0_74_x64\bin;C:\systems\packages\apache-maven-3.3.9\bin;C:\systems\packages\protoc-2.5.0-win32\;C:\systems\packages\cygwin64\bin\;C:\systems\packages\cmake-3.5.2-win32-x86\bin;C:\Program > Files\Microsoft Windows Performance > > Toolkit\;C:\systems\packages\perl-5.6.0-win\bin;C:\systems\hbase\conf;. > ----- Original meddelelse ----- > > Fra: Flavio Pompermaier < > pomperma...@okkam.it [mailto:pomperma...@okkam.it]> > Til: user < user@flink.apache.org > [mailto:user@flink.apache.org]> > Dato: Ons, 11. maj 2016 00:05 > Emne: Re: HBase write problem > > Do you have the hbase-site.xml available > in the classpath? > > On 10 May 2016 23:10, "Palle" < > pa...@sport.dk [mailto:pa...@sport.dk]> > wrote: > > HBase write problem > > Hi all. > > I have a problem writing to HBase. > > I am using a slightly modified > example of this class to proof the > concept: > > https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java > > [https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java] > > However all the HBase-specific stuff > is exactly the same as in the > HBaseWriteExample. > > The problem I see is that the job > never completes (been running for > more than an hour now) and it is only > 13 key/value pairs that is to be > written to HBase :-) > I have tested the map/reduce stuff > works if I replace the HBase > connection stuff with just a write to > a text file - works OK. I have also > tested that I can insert data in > HBase from a similar Hadoop MapReduce > job. > > Here is the part of the code where I > guess the problem is: > > @Override > public Tuple2<Text, Mutation> > map(Tuple2<String, Integer> t) throws > Exception { > LOG.info("Tuple2 map() called"); > reuse.f0 = new Text(t.f0); > Put put = new Put(t.f0.getBytes()); > put.add(MasterConstants.CF_SOME, > MasterConstants.COUNT, > Bytes.toBytes(t.f1)); > reuse.f1 = put; > return reuse; > } > }).output(new > HadoopOutputFormat<Text, > Mutation>(new > TableOutputFormat<Text>(), job)); > > env.execute("Flink HBase Event Count > Hello World Test"); > > This code matches the code in the > HBaseWriteExample.java I should > think. > > The "Tuple2" log line I see exactly > the 13 times I expect, and the last > log line I see is this: > 2016-05-10 21:48:42,715 INFO > org.apache.hadoop.hbase.mapreduce.TableOutputFormat > - Created table instance for > event_type_count > > Any suggestions to what the problem > could be? > > Thanks, > Palle
