The 'No StoreFiles for' is logged at DEBUG level.
Was 9151f75eaa7d00a81e5001f4744b8b6a among the regions which didn't finish
split ?
Can you pastebin more of the master log during this period ?
Any other symptom that you observed ?
Cheers
On Thu, Apr 7, 2016 at 12:59 AM, Heng Chen <heng.chen.1...@gmail.com> wrote:
> This is log about one region i grep from master log
>
> 2016-04-07 12:20:53,984 INFO [AM.ZK.Worker-pool2-t145]
> master.RegionStates: Transition null to {9151f75eaa7d00a81e5001f4744b8b6a
> state=SPLITTING_NEW, ts=1460002853984,
> server=dx-ape-regionserver40-online,60020,1459998494013}
> 2016-04-07 12:20:55,326 INFO [AM.ZK.Worker-pool2-t147]
> master.RegionStates: Transition {9151f75eaa7d00a81e5001f4744b8b6a
> state=SPLITTING_NEW, ts=1460002855326,
> server=dx-ape-regionserver40-online,60020,1459998494013} to
> {9151f75eaa7d00a81e5001f4744b8b6a state=OPEN, ts=1460002855326,
> server=dx-ape-regionserver40-online,60020,1459998494013}
> 2016-04-07 12:20:55,326 INFO [AM.ZK.Worker-pool2-t147]
> master.RegionStates: Onlined 9151f75eaa7d00a81e5001f4744b8b6a on
> dx-ape-regionserver40-online,60020,1459998494013
> 2016-04-07 12:20:55,328 INFO [AM.ZK.Worker-pool2-t147]
> master.AssignmentManager: Handled SPLIT event;
>
> parent=apolo_pdf,\x00\x00\x00\x00\x01\x94\xC0\xA8,1457693428562.3410ea47a97d0aefb12ec62e8e89b605.,
> daughter
>
> a=apolo_pdf,\x00\x00\x00\x00\x01\x94\xC0\xA8,1460002853961.9151f75eaa7d00a81e5001f4744b8b6a.,
> daughter
>
> b=apolo_pdf,\x00\x00\x00\x00\x01\xA2\x0D\x96,1460002853961.a7d6d735ccbf47e0a9d3016b8fef181a.,
> on dx-ape-regionserver40-online,60020,1459998494013
> 2016-04-07 12:21:44,083 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/16b7f857eb6741a5bcaaa5516034929f.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=16b7f857eb6741a5bcaaa5516034929f
> 2016-04-07 12:21:44,123 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/1a3f77a7588a4ad38d34ed97f6c095be.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=1a3f77a7588a4ad38d34ed97f6c095be
> 2016-04-07 12:21:44,132 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/544552889a4c4de99d52814b3c229c30.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=544552889a4c4de99d52814b3c229c30
> 2016-04-07 12:21:44,138 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/fcf658954ff84e998eb71ee6477c2ebe.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=fcf658954ff84e998eb71ee6477c2ebe
> 2016-04-07 12:21:44,138 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/16b7f857eb6741a5bcaaa5516034929f.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=16b7f857eb6741a5bcaaa5516034929f
> 2016-04-07 12:21:44,177 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/1a3f77a7588a4ad38d34ed97f6c095be.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=1a3f77a7588a4ad38d34ed97f6c095be
> 2016-04-07 12:21:44,179 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/544552889a4c4de99d52814b3c229c30.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=544552889a4c4de99d52814b3c229c30
> 2016-04-07 12:21:44,181 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/fcf658954ff84e998eb71ee6477c2ebe.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=fcf658954ff84e998eb71ee6477c2ebe
> 2016-04-07 12:21:44,184 DEBUG
> [dx-ape-hmaster1-online,60000,1459998573658-ClusterStatusChore]
> regionserver.HRegionFileSystem: No StoreFiles for:
>
> hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/m
> 2016-04-07 12:23:54,259 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/16b7f857eb6741a5bcaaa5516034929f.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=16b7f857eb6741a5bcaaa5516034929f
> 2016-04-07 12:23:54,259 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/1a3f77a7588a4ad38d34ed97f6c095be.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=1a3f77a7588a4ad38d34ed97f6c095be
> 2016-04-07 12:23:54,261 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/544552889a4c4de99d52814b3c229c30.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=544552889a4c4de99d52814b3c229c30
> 2016-04-07 12:23:54,302 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/fcf658954ff84e998eb71ee6477c2ebe.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=fcf658954ff84e998eb71ee6477c2ebe
> 2016-04-07 12:23:54,302 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/16b7f857eb6741a5bcaaa5516034929f.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=16b7f857eb6741a5bcaaa5516034929f
> 2016-04-07 12:23:54,341 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/1a3f77a7588a4ad38d34ed97f6c095be.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=1a3f77a7588a4ad38d34ed97f6c095be
> 2016-04-07 12:23:54,382 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/544552889a4c4de99d52814b3c229c30.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=544552889a4c4de99d52814b3c229c30
> 2016-04-07 12:23:54,383 DEBUG [region-location-4]
> regionserver.StoreFileInfo: reference
>
> 'hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/d/fcf658954ff84e998eb71ee6477c2ebe.3410ea47a97d0aefb12ec62e8e89b605'
> to region=3410ea47a97d0aefb12ec62e8e89b605
> hfile=fcf658954ff84e998eb71ee6477c2ebe
> 2016-04-07 12:23:54,386 DEBUG [region-location-4]
> regionserver.HRegionFileSystem: No StoreFiles for:
>
> hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/m
> 2016-04-07 12:25:54,033 DEBUG [region-location-4]
> regionserver.HRegionFileSystem: No StoreFiles for:
>
> hdfs://hdfs-master:8020/hbase/data/default/apolo_pdf/9151f75eaa7d00a81e5001f4744b8b6a/m
>
> 2016-04-07 15:50 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>:
>
> > hi, guys:
> >
> > i upgrade our cluster recently, after upgrade, i found some wired
> > problems:
> >
> >
> > in Master.log, there some a lots of logs like below:
> >
> > 2016-04-07 11:57:00,597 DEBUG [region-location-0]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_tutor_user_resource/23a62ee5b1d64bd8f57f4e31c383e343/m
> > 2016-04-07 11:57:00,597 DEBUG [region-location-3]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_solar_resource/72c36fc12fcc95c66167c2199d8bcc36/m
> > 2016-04-07 11:57:02,297 DEBUG [region-location-4]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_solar_user_image/c2712ea3fbb58eaf5cb7ded2b40c1df5/d
> > 2016-04-07 11:57:02,337 DEBUG [region-location-4]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_solar_user_image/c2712ea3fbb58eaf5cb7ded2b40c1df5/m
> > 2016-04-07 11:57:58,335 DEBUG [region-location-3]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_tarzan_user_resource/f75f5acbd22266ea596d0f1ceb0130f9/d
> > 2016-04-07 11:57:58,335 DEBUG [region-location-3]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_tarzan_user_resource/f75f5acbd22266ea596d0f1ceb0130f9/m
> > 2016-04-07 11:57:58,826 DEBUG [region-location-3]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_apolo_user_resource/75dc4eb5143d28a25c287918c368f6c5/d
> > 2016-04-07 11:57:58,866 DEBUG [region-location-0]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_apolo_resource/08cffb3598c6f4446a68aabf53b1e498/d
> > 2016-04-07 11:57:58,867 DEBUG [region-location-3]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_apolo_user_resource/75dc4eb5143d28a25c287918c368f6c5/m
> > 2016-04-07 11:57:58,906 DEBUG [region-location-0]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_apolo_resource/08cffb3598c6f4446a68aabf53b1e498/m
> > 2016-04-07 11:57:59,504 DEBUG [region-location-0]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_solar_user_resource/2c9272fcde86a28b0cf6b479ef0b3745/d
> > 2016-04-07 11:57:59,544 DEBUG [region-location-0]
> > regionserver.HRegionFileSystem: No StoreFiles for:
> >
> hdfs://common-cluster:8020/hbase/data/default/gallery_solar_user_resource/2c9272fcde86a28b0cf6b479ef0b3745/m
> >
> >
> >
> > And in web-UI, there are alway some split regions in one table. The
> > split regions number seems never be changed after upgraded.
> >
> >
> > I am not sure the reason of this problem, have you meet this in
> practice?
> >
> >
> >
> >
> >
>
