[
https://issues.apache.org/jira/browse/HBASE-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096809#comment-18096809
]
mazhengxuan commented on HBASE-30219:
-------------------------------------
I would like to investigate this issue. I will try to reproduce the DDL
behavior when hbase:meta is unavailable and prepare a fix if I can confirm the
root cause. If nobody is working on it, could someone assign it to me?
> Admin DDL silently returns success (no CreateTableProcedure, no table) when
> hbase:meta RegionServer is unreachable
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30219
> URL: https://issues.apache.org/jira/browse/HBASE-30219
> Project: HBase
> Issue Type: Bug
> Components: Admin, Client, proc-v2
> Affects Versions: 2.6.4, 2.5.13
> Reporter: rstest
> Priority: Major
>
> h1. Summary
> clone_table_schema / createTable / truncateTable return success (exitValue=0)
> when the issuing client's node is partitioned from the RegionServer hosting
> hbase:meta, but no CreateTableProcedure is ever registered on the master and
> the table is never created
> {*}Environment{*}: Three-node cluster (1 master + 2 RegionServers), single
> HBase version, ZooKeeper quorum on all three nodes. A network partition is
> active between the two RegionServers (so a client on the RegionServer that is
> cut off from the meta-hosting RegionServer cannot reach hbase:meta), while
> the master and the ZK quorum remain healthy. Found by differential
> fault-injection testing (the same command sequence run on independent
> clusters under the same partition, with results compared) and reproduced
> locally with full HMaster logs captured.
> h2. Description
> When a client issues a DDL operation (e.g. {{{}clone_table_schema{}}},
> {{{}create{}}}, {{{}truncate_preserve{}}}) from a node that cannot reach the
> RegionServer currently hosting {{{}hbase:meta{}}}, the operation:
> * blocks for ~60 seconds, then
> * returns *success* to the caller ({{{}exitValue = 0{}}} from the shell; no
> exception from {{{}HBaseAdmin{}}}),
> even though:
> * the HMaster log contains *no CreateTableProcedure* (or
> TruncateTableProcedure) for that table at all - the operation never reached
> the master as a procedure, and
> * the table is {*}never created{*}: {{exists 'T'}} returns false and {{scan
> 'hbase:meta'}} has zero rows for it.
> So a DDL call that reports success silently does nothing. There is no error
> surfaced to the client, no entry in {{{}hbase:meta{}}}, and no procedure on
> the master - the failure is completely invisible to the application, which
> believes the table now exists.
> The decisive condition is {*}reachability of the meta-hosting RegionServer
> from the client's node{*}, not the health of the master or ZooKeeper. In our
> reproduction the master and the ZK quorum were both reachable from the client
> throughout; only the path from the client's RegionServer to the RegionServer
> hosting {{hbase:meta}} was cut. A DDL issued from the other side (the node
> that CAN reach the meta host) completes normally in ~1 second with a clean
> {{{}CreateTableProcedure ... ADD_TO_META -> ASSIGN -> state=ENABLED ->
> SUCCESS{}}}.
> h2. Evidence
> In a run where {{hbase:meta}} was hosted on RegionServer R1 and the partition
> cut R1 from R2:
> * A {{clone_table_schema}} issued from a client on *R2* (cannot reach
> R1/meta): client returns success after ~60 s; the complete HMaster log over
> the whole window contains *zero* mentions of the target table; {{exists}} =
> false; 0 rows in {{{}hbase:meta{}}}. -> silent loss.
> * A {{clone_table_schema}} issued from a client on *R1* (meta is local):
> HMaster log shows {{Client=... create 'T'}} -> {{CreateTableProcedure
> pid=NNN: CREATE_TABLE_PRE_OPERATION -> WRITE_FS_LAYOUT -> ADD_TO_META ->
> ASSIGN -> state=ENABLED -> SUCCESS}} in ~1 s; table exists. -> normal.
> The two cases differ only in which side of the partition the issuing client
> sits on relative to the meta host. Whichever DDL is issued from the
> meta-unreachable side is silently lost with a success return code; whichever
> is issued from the meta-reachable side succeeds.
> h2. What is established vs. the one open detail
> Established (reproduced, with logs):
> * A DDL call returns success while no procedure is ever registered on the
> master and the table is never created.
> * The trigger is the issuing client's node being unable to reach the
> RegionServer hosting {{hbase:meta}} (a RegionServer-to-RegionServer
> partition), with master and ZK quorum healthy.
> * The ~60 s duration indicates the client retries an operation that needs
> meta access (unreachable) and ultimately returns success instead of surfacing
> the failure.
> Open detail (the only thing not yet pinned): the exact point in the
> client/Admin path that converts the meta-unreachable failure into a success
> return - e.g. the {{clone_table_schema}} source-descriptor read or the
> {{createTable}} pre-flight giving up after retries and returning normally
> rather than throwing. The observable contract violation (success returned,
> nothing created) does not depend on which it is.
> h2. Steps to Reproduce
> # Start a 3-node cluster (node0 = master, node1 + node2 = RegionServers),
> any single version (reproduced on 2.5.13 and 2.6.4).
> # Ensure {{hbase:meta}} is hosted on node1 (e.g. {{move}} it there if
> needed; confirm with {{scan 'hbase:meta'}} / the master UI).
> # Create a source table to clone: \{{create 'S', {NAME => 'cf'}}}.
> # Partition node1 <-> node2 only (e.g. {{iptables -A INPUT -s <peer> -j
> DROP}} both directions). Do NOT touch node0, so the master and the ZK quorum
> stay healthy.
> # From a client/shell running on *node2* (the side cut off from meta on
> node1): {{clone_table_schema 'S', 'T'}} (or \{{create 'T', {NAME => 'cf'}
> }}).
> Observe:
> * The command blocks ~60 s, then returns success / {{exitValue = 0}} (no
> exception).
> * The HMaster log (node0) contains *no* CreateTableProcedure for {{{}T{}}}.
> * {{exists 'T'}} => false; {{scan 'hbase:meta'}} has no rows for {{{}T{}}}.
> Control: issuing the same {{clone_table_schema 'S', 'T2'}} from a client on
> *node1* (meta is local) creates {{T2}} normally in ~1 s. Healing the
> partition does not retroactively create {{T}} - the loss is durable.
> Expected: a DDL operation must not return success unless the table was
> actually created and is durably present in {{{}hbase:meta{}}}. If it cannot
> reach the metadata it needs, it must fail visibly (timeout/exception), not
> return success.
> Actual: success is returned with no procedure registered and no table created.
> h2. Root cause pointers
> * Shell DDL is a thin wrapper over {{{}Admin{}}}: {{clone_table_schema}} ->
> {{HBaseAdmin.cloneTableSchema}} (reads the source descriptor, then calls
> {{{}createTable{}}}); {{create}} -> {{HBaseAdmin.createTableAsync}} ->
> {{MasterRpcServices.createTable}} -> {{HMaster.createTable}} ->
> {{{}CreateTableProcedure{}}}; {{truncate_preserve}} ->
> {{{}HBaseAdmin.truncateTable(name, true){}}}.
> * The {{{}createTable{}}}/{{{}truncateTable{}}} futures are specified to
> wait for the table to be enabled and all regions online before returning. In
> the failing case the future returns success though no procedure ran - so the
> success path is reachable without the procedure ever being
> submitted/acknowledged.
> * Suspect area: the client/Admin retry path for the meta-dependent step
> (source-descriptor read or createTable submission) when the meta-hosting
> RegionServer is unreachable - it appears to exhaust retries (~60 s) and
> return normally instead of throwing. A maintainer with the captured HMaster +
> client logs (available on request) can confirm the exact branch.
> h2. Suggested fixes
> # A DDL Admin call must surface failure (timeout/exception) when the
> underlying meta-dependent step cannot complete, rather than returning
> success. Returning success implies the table is durably created and visible
> in {{{}hbase:meta{}}}.
> # Verify the post-condition before returning success: confirm the table
> exists in {{hbase:meta}} (and reached the expected enabled/region state), not
> merely that the local call sequence returned.
> # Add a fault-injection regression test: 3-node cluster, partition a
> RegionServer from the meta-hosting RegionServer, issue create/clone/truncate
> from the cut-off side, and assert the call FAILS (or the table is durably
> created) - it must never return success with no table created.
> h2. Additional context
> * The bug requires no version change and no special configuration - it
> reproduces on a single-version cluster (Steps to Reproduce above). It was
> originally surfaced by a differential test harness that compares independent
> clusters running the same plan under the same partition; because the
> partition silently dropped DDL from one side, the clusters ended with
> divergent table sets (one cluster missing table A, another missing table B, a
> truncated table's enabled state differing) - all explained by the single rule
> above: a DDL from the meta-unreachable side is silently lost with a success
> return code.
> * We can provide the captured per-lane HMaster logs (meta-unreachable side:
> zero mentions of the table; meta-reachable side: clean CreateTableProcedure
> SUCCESS), the RegionServer log showing the meta location, and the client-side
> timing showing the ~60 s success return.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
