nsivabalan commented on code in PR #20195: URL: https://github.com/apache/hudi/pull/20195#discussion_r4279671751
########## hudi-agent-gateway/skills/hudi-config-consultant/references/catalog-schema.md: ########## @@ -0,0 +1,181 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Catalog schema and how to query it + +## Envelope + +```json +{ + "schemaVersion": 1, + "hudiVersion": "1.3.0-SNAPSHOT", + "generatedFrom": "<git sha of the tree it was generated from>", + "configs": [ ... ] +} +``` + +`configs` is sorted by `key`, so regeneration produces a readable diff. Read `hudiVersion` and +`generatedFrom` on every query; they belong in your answer. + +## A config entry + +### Declared facts — extracted verbatim from the builder chain + +| Field | Meaning | +|---|---| +| `key` | The property key, e.g. `hoodie.compact.inline.max.delta.commits`. | +| `keyResolution` | `literal`, `resolvedFromConstants` (key built from a `static final String` prefix), or `derivedFromConfigKey` (key built from another config's key). | +| `keyExpression` | The source expression, when the key was not a plain literal. | +| `type` | The declared type parameter: `String`, `Boolean`, `Integer`, `Long`, … | +| `declaredIn` | `{class, constant, file, line}` — where the `ConfigProperty` is declared. | +| `configGroup` | The owning config class, e.g. `HoodieCompactionConfig`. Use this to group related configs. | +| `module` | Top-level Maven module, e.g. `hudi-client`. | +| `hasDefaultValue` | `false` for `noDefaultValue()` configs — these are required or inferred. | +| `defaultValue` | The default, or `null`. | +| `defaultValueIsLiteral` | `false` means `defaultValue` is a **source expression**, not a value — e.g. `String.valueOf(104857600)` or `KEEP_LATEST_COMMITS.name()`. | +| `defaultValueNote` | Present when the default is computed, to say so. | +| `documentation` | The `withDocumentation` text. | +| `documentationSource` | `literal`, `resolvedFromConstants`, or `enumClass:<Name>` when the docs live on an enum. | +| `enumDocumentation` | For `enumClass` docs: `{enum, file, description, values: [{value, description}]}`. **This is where valid values and their meanings live** for strategy-style configs. | +| `validValues` | From `withValidValues`, when present. Often empty even for enum-valued configs — check `enumDocumentation` too. | +| `alternatives` | From `withAlternatives`: older key names that still work. | +| `sinceVersion`, `deprecatedAfter`, `supportedVersions` | Version metadata, when declared. | +| `advanced` | `markAdvanced()` was called. A tuning knob, not a routine setting. | +| `hasInferFunction` | Hudi may derive this value from other configs when unset. | +| `builderSteps` | The builder methods seen, in order. Useful for spotting what the parser did and did not see. | +| `parseWarnings` | Non-empty means the generator could not fully read something. Check before answering. | +| `aliasConstants` | Other Java constants that re-export the same config. | +| `alsoDeclaredIn` | Present when the same key is declared in more than one class. | + +### Derived context — recovered from code, heuristic + +| Field | Meaning | +|---|---| +| `accessors` | Config-class methods that read this config and nothing else, e.g. `HoodieWriteConfig.getInlineCompactDeltaCommitMax`. These are what consumers actually call. | +| `readSites` | Up to 20 `{class, method, file, line, via, gates}`. `via` is `constant` (the config constant is named at that line) or `accessor:<name>` (the line calls a resolved accessor). | +| `coConfigs` | Up to 25 `{key, sharedMethods}` — other configs mentioned in the same method bodies. A **co-occurrence** measure, not a declared dependency. | +| `gatingConditions` | Up to 12 `{condition, at}` — enclosing `if`/`switch` conditions, with the `class.method (file:line)` they came from. | Review Comment: Both halves fixed, and the diagnosis was exactly right — it was file-walk order filling the cap. **Read sites** are now collected to 200 and *ranked* before truncating to 20, rather than taking the first 20 alphabetically. Core engine code (`hudi-client`, `hudi-common`) first, then engine integrations, then sync/cloud connectors, with CLI, examples and the integ-test harness last. Within that, a read inside the `Hoodie*Config` class that declares the key is plumbing and sorts below a real consumer, and an accessor call site sorts above a bare constant reference. Gating conditions roll up in the same order, since a gate inherits the standing of the site it came from. Your worked example, before and after. `hoodie.compact.inline.max.delta.commits` led with `TableCommand.changeTableType` and all three top gates came from there; it now reads: ``` TimelineArchiverV1.getCommitInstantsToArchive TimelineArchiverV1.java:267 TimelineArchiverV2.getCommitInstantsToArchive TimelineArchiverV2.java:224 ScheduleCompactionActionExecutor.needCompact ScheduleCompactionActionExecutor.java:208 ``` with the first gate now `if (!commitTimeline.empty() && commitTimeline.countInstants() > maxInstantsToKeep)` at `TimelineArchiverV1.java:267` — the archiver condition you said was missed. `hoodie.clean.commits.retained` leads with `ArchivalUtils`, `HoodieMetadataWriteUtils` and `CleanPlanner`, and the integ-test gate is gone. **`coConfigs`** I did not try to fix by filtering large methods, because the field has a deeper problem that filtering would hide: it cannot see a companion compared as a typed enum rather than named as a key, which is why `hoodie.write.concurrency.mode` never appeared beside `hoodie.write.lock.provider` despite being the config whose absence silently swaps a distributed lock provider for the in-process one. So instead: - `coConfigs` is demoted in the skill from a ranking to a **lead list** — the instruction now says never present the ordering as a recommendation, open the shared method first. - A new `gatedBy` field records configs named inside a config's own gating conditions, which is a strictly stronger relationship. 71 configs have one; every Datadog key correctly shows `hoodie.metrics.on` and `hoodie.metrics.reporter.type`. Filtering `createMetadataWriteConfig` would still leave the enum-comparison blind spot, and a reader trusting a tidier-looking list would be worse off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
