Check HBase logs for any additional exceptions. The main problem of HBase
bulkload that for almost any kind of problem it reports that there was a
split. Sometimes it could be related to HA configuration (people are using
cluster name with port number) and it's visible from the logs (errors that
hdfs name is incorrect).
Thanks,
Sergey
On Fri, May 26, 2017 at 12:18 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> You can compare 27c0a905ee174c9898d324acf1554bf9 with region names of the
> target table and find out which region the hfile should go to.
>
> Then check the region server log where that region was hosted and see what
> the cause might be.
>
> Cheers
>
> On Thu, May 25, 2017 at 5:53 PM, anil gupta <anilgupt...@gmail.com> wrote:
>
> > Cross posting since this seems to be an HBase issue.
> > I think completeBulkLoad step is failing. Please refer to the mail below.
> >
> > ---------- Forwarded message ----------
> > From: anil gupta <anilgupt...@gmail.com>
> > Date: Thu, May 25, 2017 at 4:38 PM
> > Subject: [IndexTool NOT working] mapreduce.LoadIncrementalHFiles: Split
> > occured while grouping HFiles
> > To: "u...@phoenix.apache.org" <u...@phoenix.apache.org>
> >
> >
> > Hi,
> >
> > We are using HDP2.3.2(Phoenix 4.4 and HBase 1.1), we created a secondary
> > index on an already existing table. We paused all writes to Primary
> table.
> > Then we invoked IndexTool to populate secondary index table. We have
> tried
> > same steps many times but we keep on getting following error(we have also
> > tried drop the index and adding it again):
> >
> > 2017-05-24 18:00:10,281 WARN [LoadIncrementalHFiles-2]
> > util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit
> is
> > deprecated by hbase.regionserver.global.memstore.size
> > 2017-05-24 18:00:10,340 WARN [LoadIncrementalHFiles-12]
> > util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit
> is
> > deprecated by hbase.regionserver.global.memstore.size
> > 2017-05-24 18:00:10,342 INFO [LoadIncrementalHFiles-11]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > c79ae6d27824424f99523dad586e86b1 first=JF2GPADC8GH331037\x00\
> > x80\x00\x1A0\x80\x00\x01Wj\x03r1defc4d301e4ec172b49be4a7ea33c2f7
> > last=JTHBK1GG4E2122477\x00\x80\x00$\xE4\x80\x00\x01[\xAD`{\
> > x17901d036d588292854ac5b1d4c29d8e1e
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-14]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > f0e97b218aed4abf9949cf49a57e559b first=5NPEB4AC3DH620091\x00\
> > x80\x00\xE0\x16\x80\x00\x01X\xE5g\xD6\x0B81d210ac753ed281e8627e5edb7
> eb59f
> > last=JF2GPADC8GH331037\x00\x80\x00\x1A0\x80\x00\x01W]&\
> > xE54f37d636104f6cd916b2b07bf3aa94d3f
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-2]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 27c0a905ee174c9898d324acf1554bf9 first=WMWZP3C58FT708786\x00\
> > x80\x00\xE0\x16\x80\x00\x01Y\xB8\x95U\xA0d21d32aed18af976dd53735705c
> 728cd
> > last=`1GCRCPE05BZ430377\x00\x80\x00}\x05\x80\x00\x01[\xDEE\
> > x91L383768c6ac5f306fa99f68964b4f18aa
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-12]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > d0a6c4b727bb416f840ed254658f3982 first=1N4BZ0CP4GC308715\x00\
> > x80\x01T\xFC\x80\x00\x01U\xE3\x7FL\x9A37b77d47941e99e430fcb0e0657f5558
> > last=2GKALMEK7H6220949\x00\x80\x00!\x1A\x80\x00\x01Y\x18\xE6\xB3\
> > xB42e72036f7e7e03078f41fc82712c5de7
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-0]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 11627a2861e3446e9d6f684ab534563e first=3C7WRNAJ6GG313342\x00\
> > x80\x00NB\x80\x00\x01V}\xFD\xE4+65bbebdd06dedd8466a31ebd33841a51
> > last=3N1CE2CP2FL407481\x00\x80\x00\xE0\x16\x80\x00\x01W\x1B\x0A\x02\
> > xC1fc95d4114d5e91197a5e41bf37c9e8c7
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-1]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 23df78bafd304ff887385a2b6becf06d first=1C4RJFLT6HC742023\x00\
> > x80\x00x\xFF\x80\x00\x01[J8\x0Ac8b65a80fe1662fb25d80798a66cc83dc
> > last=1FMCU9J90EUB68140\x00\x80\x01X\xA4\x80\x00\x01[\x1B\xDD\xB2\
> > x1C577502512ec987844b0108738a9ec6ba
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-3]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 39dc73882bec49a0bdd5d787b06ac032 first=1G1JD5SB5H4136951\x00\
> > x80\x00!\x8A\x80\x00\x01Z%\xF6\x7Ffef0b8faeeeb4a10103e1a67ea5ebdbec
> > last=1GNKVHKD7HJ275239\x00\x80\x00$\x87\x80\x00\x01Z%\xF6s\
> > xDC0961566a370af3b7da440e9705bc4c8c
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-8]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > a37a2a56ff5c48399cf1abd92f99662f first=###########239824\x00\
> > x80\x01(\xFE\x80\x00\x01Z\xAE\xD6\xE0Xe5a45a2beab337228bdba90c06f34a12
> > last=1C4RJFLT6HC742023\x00\x80\x00x\xFF\x80\x00\x01[H\xF9w\
> > x8D60edb518c27ef80f8a751701926d9174
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-10]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > b5843a62a6bd47fbbfc29303bee158e3 first=5FNRL5H4XFB033259\x00\
> > x80\x00\x1EZ\x80\x00\x01\x5C"\x87s\xF5ce24ec7e2a3698836386bccabc1265af
> > last=5NPEB4AC3DH620091\x00\x80\x00\xE0\x16\x80\x00\x01X\xE4\x1B\
> > x9Dq95568f371c1ebd06c497df7129f248a2
> > 2017-05-24 18:00:10,343 INFO [LoadIncrementalHFiles-5]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 595e89b2fae8494d8a878bc6ba306e2f first=JTHBK1GG4E2122477\x00\
> > x80\x00$\xE4\x80\x00\x01[\xAF\xF0]%d306ddb81ea3bc093c40efe9f198f03a
> > last=KNDPMCAC1H7201793\x00\x80\x00\x1E\xC7\x80\x00\x01X\xD5\xF5\xEA\
> > xD183ff28e7e1aeeff315b167d3d869b797
> > 2017-05-24 18:00:10,348 INFO [LoadIncrementalHFiles-9]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > ac4edbfdf7394e88aaf84f5489511b7e first=1GNKVHKD7HJ275239\x00\
> > x80\x00$\x87\x80\x00\x01Z+\x1C\x87\xB65ab1c776dde09cf3257821e420288075
> > last=1N4BZ0CP4GC308715\x00\x80\x01T\xFC\x80\x00\x01U\xE2\xA1 \
> > xB743fe1c3b744fae3c2f9e6a896003869f
> > 2017-05-24 18:00:10,348 INFO [LoadIncrementalHFiles-4]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 5580694921e846949f49c1426162da6e first=2GKALMEK7H6220949\x00\
> > x80\x00!\x1A\x80\x00\x01Y!\xE5\xCF\xC110e1f23860ff6a21fe848bb85e2e0969
> > last=3C7WRNAJ6GG313342\x00\x80\x00NB\x80\x00\x01V}\x1D\xF5\
> > x10620faf6227fb4181a7ee99fa07a52f08
> > 2017-05-24 18:00:10,349 INFO [LoadIncrementalHFiles-7]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 7bc03c5b15304ec083e18c2376789562 first=1FMCU9J90EUB68140\x00\
> > x80\x01X\xA4\x80\x00\x01[\x1D&\x9D\x12f80b9cac0af584642ecbd6e1b1bbd617
> > last=1G1JD5SB5H4136951\x00\x80\x00!\x8A\x80\x00\x01Z$\xB0\xAC\
> > x1625dc1ae357cb3e9332d823210102de85
> > 2017-05-24 18:00:10,349 INFO [LoadIncrementalHFiles-13]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > d28c4d2668cc4169b201b349b2522f9a first=3N1CE2CP2FL407531\x00\
> > x80\x00\x0A\xC9\x80\x00\x01X\xD1\xBC\xD195d73fa7cd13082d2e6f590ca15
> 78b453
> > last=5FNRL5H4XFB033259\x00\x80\x00\x1EZ\x80\x00\x01\x5C!8\xA7.
> > dd776deeea36c2c5b132fafed7013b16
> > 2017-05-24 18:00:10,350 INFO [LoadIncrementalHFiles-6]
> > mapreduce.LoadIncrementalHFiles:
> > Trying to load hfile=hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 5d90de564af645f084319a03f64b0a3d first=KNDPMCAC1H7201793\x00\
> > x80\x00\x1E\xC7\x80\x00\x01X\xD7F#\xEBcd2872ac4f258a091a79a55d4dbfef9e
> > last=WMWZP3C58FT708786\x00\x80\x00\xE0\x16\x80\x00\x01Y\xB7?\xCA\
> > xADb28da871b5581e409561c813ed9c0805
> > 2017-05-24 18:00:11,780 WARN [LoadIncrementalHFiles-15]
> > mapreduce.LoadIncrementalHFiles:
> > Attempt to bulk load region containing into table BI.VIN_IDX with files
> > [family:I path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > d28c4d2668cc4169b201b349b2522f9a, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/23df78bafd304ff887385a2b6becf06d, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 39dc73882bec49a0bdd5d787b06ac032, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/a37a2a56ff5c48399cf1abd92f99662f, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 11627a2861e3446e9d6f684ab534563e, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/ac4edbfdf7394e88aaf84f5489511b7e, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > f0e97b218aed4abf9949cf49a57e559b, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/7bc03c5b15304ec083e18c2376789562, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 5d90de564af645f084319a03f64b0a3d, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/5580694921e846949f49c1426162da6e, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > 595e89b2fae8494d8a878bc6ba306e2f, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/27c0a905ee174c9898d324acf1554bf9, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > c79ae6d27824424f99523dad586e86b1, family:I path:hdfs://HDFS1/user/hbase/
> > idx_test_7/BI.VIN_IDX/I/b5843a62a6bd47fbbfc29303bee158e3, family:I
> > path:hdfs://HDFS1/user/hbase/idx_test_7/BI.VIN_IDX/I/
> > d0a6c4b727bb416f840ed254658f3982] failed. This is recoverable and they
> > will be retried.
> > 2017-05-24 18:00:11,793 INFO [main] mapreduce.LoadIncrementalHFiles:
> > Split
> > occured while grouping HFiles, retry attempt 1 with 15 files remaining to
> > group or split
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
