[
https://issues.apache.org/jira/browse/HBASE-13337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545307#comment-14545307
]
Samir Ahmic commented on HBASE-13337:
-------------------------------------
I can confirm that this is still issue on master branch and here is what i have
concluded from testing (distributed cluster):
1. RS restart is enough to reproduce this issue.
2. As [~jxiang]] suggested this is not problem with AM. Issue is probably
related to RSRpcService on regionserver side.
3. I was able to trace this issue in case of assign operation to
(ServerManager.java):
{code}
734 public RegionOpeningState sendRegionOpen(final ServerName server,
HRegionInfo region, List<ServerName> favoredNodes)
{code}
master never gets response from regionserver because of
java.nio.channels.ClosedChannelException.
4. From logs i have notices that when RS is in this state
ServerManager#sendRegionWarmup is also failing with
java.nio.channels.ClosedChannelException.
5. When issue is present nothing about this requests is logged on RS side so my
assumption is that RPC server on rs side in some strange state.
I will try to dig some more info from rs side and post it here.
> Table regions are not assigning back, after restarting all regionservers at
> once.
> ---------------------------------------------------------------------------------
>
> Key: HBASE-13337
> URL: https://issues.apache.org/jira/browse/HBASE-13337
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Affects Versions: 2.0.0
> Reporter: Y. SREENIVASULU REDDY
> Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-13337.patch
>
>
> Regions of the table are continouly in state=FAILED_CLOSE.
> {noformat}
> Region State
>
> RIT time (ms)
> 8f62e819b356736053e06240f7f7c6fd
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM1,16040,1427362531818 113929
> caf59209ae65ea80fca6bdc6996a7d68
> t1,dddddddd,1427362431330.caf59209ae65ea80fca6bdc6996a7d68.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM2,16040,1427362533691 113929
> db52a74988f71e5cf257bbabf31f26f3
> t1,44444444,1427362431330.db52a74988f71e5cf257bbabf31f26f3.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM3,16040,1427362533691 113920
> 43f3a65b9f9ff283f598c5450feab1f8
> t1,88888888,1427362431330.43f3a65b9f9ff283f598c5450feab1f8.
> state=FAILED_CLOSE, ts=Thu Mar 26 15:05:36 IST 2015 (113s ago),
> server=VM1,16040,1427362531818 113920
> {noformat}
> *Steps to reproduce:*
> 1. Start HBase cluster with more than one regionserver.
> 2. Create a table with precreated regions. (lets say 15 regions)
> 3. Make sure the regions are well balanced.
> 4. Restart all the Regionservers process at once across the cluster, except
> HMaster process
> 5. After restarting the Regionservers, successfully will connect to the
> HMaster.
> *Bug:*
> But no regions are assigning back to the Regionservers.
> *Master log shows as follows:*
> {noformat}
> 2015-03-26 15:05:36,201 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Transition {8f62e819b356736053e06240f7f7c6fd
> state=OFFLINE, ts=1427362536106, server=VM2,16040,1427362242602} to
> {8f62e819b356736053e06240f7f7c6fd state=PENDING_OPEN, ts=1427362536201,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,202 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStateStore: Updating row
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd. with
> state=PENDING_OPEN&sn=VM1,16040,1427362531818
> 2015-03-26 15:05:36,244 DEBUG [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Force region state offline
> {8f62e819b356736053e06240f7f7c6fd state=PENDING_OPEN, ts=1427362536201,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,244 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Transition {8f62e819b356736053e06240f7f7c6fd
> state=PENDING_OPEN, ts=1427362536201, server=VM1,16040,1427362531818} to
> {8f62e819b356736053e06240f7f7c6fd state=PENDING_CLOSE, ts=1427362536244,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,244 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStateStore: Updating row
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd. with
> state=PENDING_CLOSE
> 2015-03-26 15:05:36,248 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=1 of 10
> 2015-03-26 15:05:36,248 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=2 of 10
> 2015-03-26 15:05:36,249 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=3 of 10
> 2015-03-26 15:05:36,249 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=4 of 10
> 2015-03-26 15:05:36,249 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=5 of 10
> 2015-03-26 15:05:36,250 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=6 of 10
> 2015-03-26 15:05:36,250 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=7 of 10
> 2015-03-26 15:05:36,250 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=8 of 10
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=9 of 10
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.AssignmentManager: Server VM1,16040,1427362531818 returned
> java.nio.channels.ClosedChannelException for
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd., try=10 of 10
> 2015-03-26 15:05:36,251 WARN [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Failed to open/close 8f62e819b356736053e06240f7f7c6fd on
> VM1,16040,1427362531818, set to FAILED_CLOSE
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStates: Transition {8f62e819b356736053e06240f7f7c6fd
> state=PENDING_CLOSE, ts=1427362536244, server=VM1,16040,1427362531818} to
> {8f62e819b356736053e06240f7f7c6fd state=FAILED_CLOSE, ts=1427362536251,
> server=VM1,16040,1427362531818}
> 2015-03-26 15:05:36,251 INFO [VM2,16020,1427362216887-GeneralBulkAssigner-0]
> master.RegionStateStore: Updating row
> t1,55555555,1427362431330.8f62e819b356736053e06240f7f7c6fd. with
> state=FAILED_CLOSE
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
