linslee75 opened a new pull request, #5194: URL: https://github.com/apache/solr/pull/5194
https://issues.apache.org/jira/browse/SOLR-18563 Dev list discussion: https://lists.apache.org/thread/3ogs0hjfw89h4w78mzm8rsfpzgdkx2r6 ## Description Every Schema API modification reloads the core. A reload swaps the `IndexWriter`, and closing the old writer waits for running merges while holding the lock that every update needs. As a result, adding a single field while a long merge is running (for example, merging large HNSW vector segments) blocks indexing on that core until the merge finishes. `add-field`, `add-dynamic-field`, and `add-copy-field` do not modify existing field types or similarity. The `IndexWriter`'s analyzer and codec resolve each field against `core.getLatestSchema()`. Similarity is the only schema component captured when the writer is created, and these commands do not change it. Schemaless mode (`AddSchemaFieldsUpdateProcessorFactory`) already adds fields and copy fields this way, without a reload. ## Solution In `SchemaManager.doOperations`, a request containing **only** `add-field`, `add-dynamic-field`, and `add-copy-field` persists the updated managed schema with `ManagedIndexSchema.persistManagedSchema(false)` and updates the core with `SolrCore.setLatestSchema()`, instead of reloading the core. - **Indexing:** Documents indexed after the command returns can use the new schema elements. - **Standalone:** After releasing the schema update lock, Solr opens a new searcher and waits for it. Queries can use the new field as soon as the command returns, which preserves the SOLR-13795 behavior. If the searcher cannot be reopened, the request fails with HTTP 500 and reports that the schema was already persisted. - **SolrCloud:** The existing schema ZooKeeper version agreement check still runs, and replicas pick up the schema through their existing `ZkIndexSchemaReader` watch. See the behavior changes below for query visibility. - **Unchanged:** Any other command, including a request that mixes other commands with these additions, still reloads the core. The Reference Guide (`schema-api.adoc`) and a changelog entry are updated. ## Behavior changes - **SolrCloud query visibility:** Previously, the reload opened new searchers, so queries could use the new field as soon as the command returned. Now existing searchers keep the old schema until the next commit or soft commit opens a new one. This is the situation SOLR-13795 fixed for standalone by reloading. - In practice, a newly added field has no searchable data until that commit anyway. - Opening a searcher only on the core that received the request would leave replicas inconsistent. Doing it on every replica means opening searchers from the ZooKeeper schema watcher, which I kept out of scope. - Happy to add it if reviewers prefer. - **Empty request:** An empty request (`{}`) previously reloaded the core; it now returns immediately without doing anything. I am not aware of anyone relying on an empty request as a reload trigger. AI disclosure: I used an AI coding assistant for code review, test design, and drafting this description. I reviewed and verified all changes. ## Tests - `TestSchemaChangeDuringMerge` reproduces the issue: - A custom `MergeScheduler` holds a merge open. While it is held, the test issues `add-field` and then indexes a document. - On `main`, the update times out because the reload is blocked while closing the `IndexWriter`; with this change, it completes. - `TestBulkSchemaAPI` (standalone): - `testAdditiveChangesDoNotReloadCore`: verifies that the core `uniqueId` is unchanged, the latest schema contains the new elements, the searcher uses the latest schema, and a facet query on the new fields works without a commit. - `testMixedCommandsStillReloadCore`: verifies that `add-field-type` plus `add-field` still reloads the core. - `testEmptyRequestDoesNotReloadCore` - `TestManagedSchemaAPI` (SolrCloud, 1 shard × 2 replicas): - `testAddFieldDoesNotReloadCore` and `testAddDynamicFieldAndCopyFieldDoNotReloadCore`: verify that replicas do not reload, every replica's latest schema contains the change, and indexing, committing, and querying with the new fields works. - `testDeleteFieldStillReloadsCore`: verifies that the non-additive path still reloads. Ran `./gradlew tidy`, `./gradlew :solr:core:check -x test`, and the test classes listed above. A full `./gradlew check` could not complete locally: `:solr:solr-ref-guide:buildLocalAntoraSite` fails because this checkout is a Git worktree (`Local content source must be a git repository`), unrelated to this change. ## Checklist - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (Optional but recommended; not available for branches on forks living under an organisation.) - [x] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. (`:solr:core:check -x test` passes; see note above about the Reference Guide task.) - [x] I have added tests for my changes. - [x] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide). - [x] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
