Thanks, I tried briefly but maybe I didn't do quite the right search. In any case, thanks for the help.
---- Saad On Mon, Mar 19, 2018 at 2:50 PM, Huaxiang Sun <[email protected]> wrote: > You can google search the exception stack and mostly it will find the JIRA. > > Regards, > > Huaxiang > > > On Mar 19, 2018, at 10:52 AM, Saad Mufti <[email protected]> wrote: > > > > Thanks!!! Wish that was documented somewhere in the manual. > > > > Cheers. > > > > ---- > > Saad > > > > > > On Mon, Mar 19, 2018 at 1:38 PM, Huaxiang Sun <[email protected]> wrote: > > > >> Mostly it is due to HBASE-15430 <https://issues.apache.org/ > >> jira/browse/HBASE-15430>, “snapshot.manifest.size.limit” needs to be > >> configured as 64MB or 128MB. > >> > >> Regards, > >> > >> Huaxiang Sun > >> > >> > >>> On Mar 19, 2018, at 10:16 AM, Saad Mufti <[email protected]> wrote: > >>> > >>> Hi, > >>> > >>> We are running on HBase 1.4.0 on an AWS EMR/HBase cluster. > >>> > >>> We have started seeing the following stacktrace when trying to take a > >>> snapshot of a table with a very large number of files (12000 regions > and > >>> roughly 360000 - 400000 files). The number of files should go down as > we > >>> haven't been compacting for a while for other operational reasons and > are > >>> now running it. But I'd to understand why our snapshots are failing > with > >>> the following: > >>> > >>> 2018-03-19 16:05:56,948 ERROR > >>>> [MASTER_TABLE_OPERATIONS-ip-10-194-208-6:16000-0] > >>>> snapshot.TakeSnapshotHandler: Failed taking snapshot { > >>>> ss=pgs-device.03-19-2018-15 table=pgs-device type=SKIPFLUSH } due to > >>>> exception:unable to parse data manifest Protocol message was too > >> large. May > >>>> be malicious. Use CodedInputStream.setSizeLimit() to increase the > size > >>>> limit. > >>>> > >>>> org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: unable > to > >>>> parse data manifest Protocol message was too large. May be malicious. > >> Use > >>>> CodedInputStream.setSizeLimit() to increase the size limit. > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest( > >> SnapshotManifest.java:468) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest. > >> load(SnapshotManifest.java:297) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest. > >> open(SnapshotManifest.java:129) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier. > >> verifySnapshot(MasterSnapshotVerifier.java:108) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process( > >> TakeSnapshotHandler.java:203) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.executor.EventHandler.run( > >> EventHandler.java:129) > >>>> > >>>> at > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1149) > >>>> > >>>> at > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:624) > >>>> > >>>> at java.lang.Thread.run(Thread.java:748) > >>>> > >>>> Caused by: com.google.protobuf.InvalidProtocolBufferException: > Protocol > >>>> message was too large. May be malicious. Use > >>>> CodedInputStream.setSizeLimit() to increase the size limit. > >>>> > >>>> at > >>>> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded( > >> InvalidProtocolBufferException.java:110) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.refillBuffer( > >> CodedInputStream.java:755) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readRawBytes( > >> CodedInputStream.java:811) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readBytes( > >> CodedInputStream.java:329) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile.<init>(SnapshotProtos.java:1313) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile.<init>(SnapshotProtos.java:1263) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile$1.parsePartialFrom( > >> SnapshotProtos.java:1364) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$StoreFile$1.parsePartialFrom( > >> SnapshotProtos.java:1359) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readMessage( > >> CodedInputStream.java:309) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles.<init>(SnapshotProtos.java:2161) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles.<init>(SnapshotProtos.java:2103) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom( > >> SnapshotProtos.java:2197) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom( > >> SnapshotProtos.java:2192) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readMessage( > >> CodedInputStream.java:309) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest.<init>(SnapshotProtos.java:1165) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest.<init>(SnapshotProtos.java:1094) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196) > >>>> > >>>> at > >>>> com.google.protobuf.CodedInputStream.readMessage( > >> CodedInputStream.java:309) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest.<init>(SnapshotProtos.java:3858) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest.<init>(SnapshotProtos.java:3792) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889) > >>>> > >>>> at > >>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89) > >>>> > >>>> at > >>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95) > >>>> > >>>> at > >>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$ > >> SnapshotDataManifest.parseFrom(SnapshotProtos.java:4115) > >>>> > >>>> at > >>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest( > >> SnapshotManifest.java:464) > >>>> > >>>> : > >>>> > >>>> ... 8 more > >>>> > >>>> > >>> Thanks. > >>> > >>> ---- > >>> Saad > >> > >> > >
