kpatel006 opened a new pull request, #19892:
URL: https://github.com/apache/pinot/pull/19892
## Background
The broker stops routing to a server when the server is disabled in Helix or
when the `FailureDetector` marks it unhealthy. If every replica of a segment is
excluded, `BaseInstanceSelector` marks the segment unavailable and queries
return partial results.
An unhealthy server often still has the data and may answer; the "unhealthy"
mark can come from a short network blip. "Fail-open" means routing to such a
server when no healthy replica is left. Disabled servers are never used.
This PR is part of a series:
- **Metrics (`upstream/fail-open-metrics`):** adds
`FAIL_OPEN_ELIGIBLE_SEGMENTS` and `FAIL_OPEN_ELIGIBLE_REQUESTS` to count how
often fail-open would apply.
- **Policy (this PR):** adds the config flag that turns fail-open on or off.
- **Routing (follow-up):** makes the instance selectors use the flag.
## Summary
This PR adds a new broker config, `pinot.broker.enable.fail.open`, which
defaults to `false`. The flag lives in a new class, `FailOpenPolicy`, that is
safe to read from multiple threads. A value in cluster config overrides the
broker's own value, both at startup and when the cluster config changes at
runtime. If the cluster value is deleted, the policy goes back to the broker's
own value, or to the default if the broker never set one.
Routing doesn't read the flag yet, because nothing calls `isEnabled()`; the
instance selectors will start using it in a follow-up PR. `FailOpenPolicyTest`
covers the policy with unit tests, and `BrokerFailOpenPolicyIntegrationTest`
checks the same behavior on a running broker through Helix.
## Motivation
Routing to an unhealthy server avoids partial results, but the server might
return errors or respond slowly. Operators need to control when the broker does
this. The flag is off by default, so existing deployments behave exactly as
before. It can be turned on for a single broker or for the whole cluster, so it
can be rolled out gradually.
Because cluster config changes apply without a restart, operators can switch
fail-open on or off during an incident. Removing a cluster-wide override puts
each broker back on its own setting instead of leaving an old value in place.
## Diagram
```mermaid
flowchart LR
B["Broker config<br/>pinot.broker.enable.fail.open<br/>(default: false)"]
C["Cluster config<br/>(ZK / Helix)"]
P["FailOpenPolicy<br/>enabled flag"]
R["Instance selectors<br/>(follow-up PR)"]
B -->|"startup value,<br/>used again if cluster value is deleted"| P
C -->|"overrides broker value<br/>at startup and on runtime updates"| P
P -->|"isEnabled()"| R
R -->|"if enabled: route to an unhealthy replica<br/>when no healthy one
is left"| S["Servers"]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
