kfaraz commented on code in PR #19945: URL: https://github.com/apache/druid/pull/19945#discussion_r4155074016
########## docs/release-info/release-notes.md: ########## @@ -65,42 +135,389 @@ This section contains detailed release notes separated by areas. #### Other web console improvements +- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253) +- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533) +- Improved handling for new tabs [#19483](https://github.com/apache/druid/pull/19483) +- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481) + ### Ingestion +* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508) +* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any column besides `__time` [#19386](https://github.com/apache/druid/pull/19386) +* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558) +* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394) +* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536) + #### SQL-based ingestion +##### EXTERN for S3 now supports role ARN + +You can now include the role ARN when running an INSERT INTO EXTERN query. For example: + +``` +INSERT INTO +EXTERN( + s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520')) +AS CSV +SELECT ... +``` + +[#19317](https://github.com/apache/druid/pull/19317) + ##### Other SQL-based ingestion improvements +- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316) + #### Streaming ingestion +##### Scaling cool down + +You can now configure different cool down durations for scaling up and scaling down streaming task autoscalers. + +[#19286](https://github.com/apache/druid/pull/19286) + +##### Improved supervisor restarts + +Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur: + +- Druid persists the updated spec without a restart. +- Druid restarts the supervisor, but running tasks aren't impacted. +- Druid restarts the supervisor, terminates the current tasks and launches fresh tasks (default behavior in older Druid versions). + +For example, cosmetic changes to a supervisor spec no longer trigger a restart. + +[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720) + +Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled. + +[#19541](https://github.com/apache/druid/pull/19541) + +###### Latest offset and backfill + +For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, like alerting scenarios. + +Note the following requirements: + +- The supervisor's `useEarliestSequenceNumber` property must be `false`. +- The supervisor context must have `useConcurrentLocks` set to `true` to allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks. +- The supervisor must be in a `RUNNING` state. + +Use `POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill` or the web console to perform this action. + +[#19477](https://github.com/apache/druid/pull/19477) + +##### Prunable shard specs for streaming published segments + +Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records +the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match. + +The feature is opt-in, Kafka-only, and disabled by +default; when unset, behavior is unchanged. + +`dim_value_set` is a new core shard spec type with no fallback, so it is not forward-compatible. Upgrade all services before enabling `streamingPartitionsSpec`. Once `dim_value_set` segments are published, downgrade is unsupported +until they are compacted away or `streamingPartitionsSpec` is removed. + +Added `maxValuesPerDimension` (optional) to `streamingPartitionsSpec`. + +[#19571](https://github.com/apache/druid/pull/19571) [#19596](https://github.com/apache/druid/pull/19596) + ##### Other streaming ingestion improvements +- Added a property called `boundedStreamConfig` to the `SeekableStreamSupervisorIOConfig`, which allows operators to spin up a Supervisor that consumes only a specified offset range [#19372](https://github.com/apache/druid/pull/19372) +- Improved the cost-based autoscaler for better throughput [#19646](https://github.com/apache/druid/pull/19646) + ### Querying +#### New query laning strategy + +The `weighted` query laning strategy scores queries by how many thresholds they breach (segment count, interval duration, data age, segment range) and assigns them to configurable graduated lanes with different capacity limits, providing more nuanced lane assignment than the existing binary high/low strategy. + +The weighted query laning strategy supports optional per-threshold cost weights: + +- `periodWeight` +- `durationWeight` +- `segmentCountWeight` +- `segmentRangeWeight` + +These threshold weights default to 1. + +[#19225](https://github.com/apache/druid/pull/19225) [#19665](https://github.com/apache/druid/pull/19665) [#19696](https://github.com/apache/druid/pull/19696) + #### Other querying improvements +- Added nullable `minTime/maxTime` Long fields to `ProjectionMetadata` [#19398](https://github.com/apache/druid/pull/19398) +- Added `getDimensionRangeSet` support to `LikeDimFilter` for equality and prefix cases [#19524](https://github.com/apache/druid/pull/19524) +- Added support for aggregate projections with clustered segments [#19599](https://github.com/apache/druid/pull/19599) +- Optimized performance of aggregators for groupBy queries [#19423](https://github.com/apache/druid/pull/19423) + ### Cluster management +#### Improved `diskNormalized` balancer strategy + +The `diskNormalized` strategy is now more tunable. The `utilizationThreshold` configuration works more intuitively: increasing the threshold increases the "tolerance" of the strategy while decreasing the threshold pushes nodes' disk utilization closer together. + +[#19663](https://github.com/apache/druid/pull/19663) + +#### MiddleManager and Indexer restarts + +MiddleManagers and Indexers now persist their enabled or disabled state across restarts. If you prefer the old behavior, where the server re-enables itself after a restart, set `druid.worker.startAlwaysEnabled = true`. + +[#19373](https://github.com/apache/druid/pull/19373) + +#### Kafka idle signal Review Comment: ```suggestion #### Use processing rate for cost-based auto-scaler ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
