xiangfu0 opened a new pull request, #19090: URL: https://github.com/apache/pinot/pull/19090
## Motivation There are a lot of quickstart types, and many of them do the same thing. Ten of them bootstrapped **exactly the same cluster and the same table set** as `BATCH` or `REALTIME`, and differed only in which sample query they printed once setup finished. `QuickStartBase` already loads 10 batch tables and 12 stream tables by default, and any subclass that does not override the directory lists inherits all of them. So `-type UPSERT` today starts Kafka, creates all twelve stream tables — `meetupRsvp`, `fineFoodReviews`, all three upsert tables — and then prints one query. It is byte-for-byte the same cluster as `-type REALTIME`. ## What this does Folds those sample queries into `Quickstart` and `RealtimeQuickStart` and deletes the classes. **32 types → 21**, about 1050 lines removed. | Merged into `-type BATCH` | Merged into `-type REALTIME` | |---|---| | `MULTI_STAGE` | `UPSERT` | | `JOIN` | `PARTIAL_UPSERT` | | `TIMESTAMP` | `UPSERT_JSON_INDEX` | | `BATCH_JSON_INDEX` | `REALTIME_JSON_INDEX` | | `BATCH_COMPLEX_TYPE` | `REALTIME_COMPLEX_TYPE` | All 24 legacy `-type` spellings still resolve, now to the merged quickstart, and log a notice pointing at the canonical type. They are excluded from the `Valid types are:` help list so new users are not steered onto them. The six `bin/quick-start-*.sh` launcher scripts are kept as well, repointed at the merged classes. Instead of one flat `runSampleQueries`, each feature is a method guarded by `hasTables(...)`, which reports what the quickstart actually bootstrapped. A quickstart that narrows its table set now skips the queries it cannot answer. This also fixes existing quickstarts — `GEOSPATIAL` and `TPCH` currently print error responses for tables they never bootstrap. ## Star Schema Benchmark tables were silently empty While merging `MULTI_STAGE` I found its five SSB tables ship their ingestion spec as `IngestionJobSpec.yaml`, while `QuickStartBase.copyResourceTableToTmpDirectory` looks for `ingestionJobSpec.yaml`. With no `task` config to fall back on, `BootstrapTableTool` logged `Not found ingestionJobSpec.yaml [...] skipping data ingestion` and created all five tables **empty** on Linux and when running from the shaded jar. A case-insensitive filesystem masks this locally. TPCH's specs were already named correctly. Renamed the five specs. They now ingest, so they are back in the default batch table set and `BATCH` demos star-schema joins on the multi-stage engine. ## Testing Built the binary distribution and ran `bin/pinot-admin.sh QuickStart -type BATCH` on a real cluster: | Table | Rows | |---|---| | lineorder | 9,999 | | customer | 5,000 | | part | 5,000 | | dates | 2,556 | | supplier | 2,000 | Star-schema joins return correct results with zero exceptions — revenue by year across 1992–1998, and revenue by customer nation. All ten CI query checks were then extracted verbatim from the workflow script and run against that live cluster; all ten pass. CI switches from `-type MULTI_STAGE` to `-type BATCH` and grows from 3 query checks to 10, covering every merged feature: single-stage count, multi-stage count, multi-stage dim join, multi-stage self join, SSB population, SSB star-schema join, `lookup()` join, `json_match`, complex-type flattening, and timestamp-index columns. They run through a shared `wait_for_query` helper that fails fast if the cluster never comes up, so a dead cluster no longer burns the full polling budget ten times. Two assertions were also strengthened: the join checks now verify every row actually resolved a team name rather than only counting rows. The old `lookup()` check would have passed against an empty dimension table. New unit tests cover the alias resolution and the `hasTables` guard, including the case where a quickstart narrows its batch directories but inherits the full default stream map. ## Backward incompatibility Two things worth calling out for the release notes: 1. **Ten public classes are removed** from `pinot-tools`. `MultistageEngineQuickStart` was a supertype — this PR reparents `TPCHQuickStart`, `ColocatedJoinQuickStart` and `LookupJoinEngineQuickStart` off it. A downstream fork extending it, or a script invoking `java -cp pinot-tools.jar org.apache.pinot.tools.UpsertQuickStart` directly, will break. The `-type` aliases do not help there. No deprecated shim was added because an empty subclass declaring the same type would collide with the alias and make `selectQuickStart` non-deterministic. 2. **`-type BATCH` now starts 3 servers instead of 1**, which is what `MULTI_STAGE` and `JOIN` already did. Without it, CI would lose all cross-server multi-stage exchange coverage. `MultiDirQuickstart` is pinned back to 1 because its tier `dataDir`s are fixed absolute paths that multiple server instances would otherwise share, and `HybridQuickstart` is pinned back to 1 since it demos hybrid routing rather than distributed execution. ## Release Notes Consolidates ten redundant quickstart types into `-type BATCH` and `-type REALTIME`. The old type names continue to work as aliases but are deprecated. The `MultistageEngineQuickStart`, `UpsertQuickStart`, `PartialUpsertQuickStart`, `UpsertJsonQuickStart`, `RealtimeJsonIndexQuickStart`, `RealtimeComplexTypeHandlingQuickStart`, `JsonIndexQuickStart`, `OfflineComplexTypeHandlingQuickStart`, `TimestampIndexQuickstart` and `JoinQuickStart` classes are removed. Also fixes the Star Schema Benchmark example tables, which were being created empty due to a mis-cased ingestion job spec filename. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
