dependabot[bot] opened a new pull request, #3821: URL: https://github.com/apache/iceberg-python/pull/3821
Bumps [ray](https://github.com/ray-project/ray) from 2.56.1 to 2.57.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/ray-project/ray/releases";>ray's releases</a>.</em></p> <blockquote> <h2>Ray-2.57.0</h2> <h1>Highlights</h1> <ul> <li><strong>Ray Data:</strong> In this Ray release, we've enabled <code>DataSourceV2</code> by default (<a href="https://redirect.github.com/ray-project/ray/issues/64821";>#64821</a>), so <code>read_parquet</code> and friends use the new scan/listing infrastructure with row-group-aware chunking and predicate splitting. Hash Shuffle V2 eliminates the aggregator actor pool. V1 had to provision that pool up front from an estimate of the input size, and its actors accumulated partition shards in actor heap memory, invisible to Ray and unspillable, until finalization. V2 replaces it with two stateless task-based operators, ShuffleMapOp --> ShuffleReduceOp, that pass shards through the object store, so intermediate state spills under pressure and no capacity has to be reserved in advance. The map/reduce barrier itself remains in both designs.</li> <li>HashShuffleV2 supports <code>join</code> (<a href="https://redirect.github.com/ray-project/ray/pull/63598";>#63598</a>, <a href="https://redirect.github.com/ray-project/ray/pull/64538";>#64538</a>, <a href="https://redirect.github.com/ray-project/ray/pull/64687";>#64687</a>). This lets shuffles reuse standard map/reduce scheduling, backpressure, and resource accounting.</li> <li><strong>Ray Serve:</strong> The HAProxy ingress is now distributed as the <code>ray-haproxy</code> PyPI package instead of being compiled into images, and it is the default HAProxy binary (<a href="https://redirect.github.com/ray-project/ray/issues/64141";>#64141</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64163";>#64163</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64164";>#64164</a>). We've also added gRPC support to the HAProxy direct-ingress path, including streaming, metrics, and custom request IDs (<a href="https://redirect.github.com/ray-project/ray/issues/63735";>#63735</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64310";>#64310</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64166";>#64166</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64112";>#64112</a>). For Ray Serve LLM, we've added experimental KV-cache-aware request routing that tracks replica KV state through an event plane, tokenizes before routing, and routes on prefill/decode token load (<a href="https://redirect.github.com/ray-project/ray/issues/64084";>#64084</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64085";>#64085</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64097";>#64097</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64224";>#64224</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64327";>#64327</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64400";>#64400</a>). KV cache-aware routing’s complete support will land in 2.58.</li> <li><strong>Ray Core:</strong> We've added an embedded RocksDB storage backend for GCS fault tolerance (REP-64), selectable with <code>RAY_gcs_storage=rocksdb</code> and <code>RAY_gcs_storage_path</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63657";>#63657</a>). GCS fault tolerance no longer requires an external Redis instance. We've also added a public API for topology-aware scheduling (<a href="https://redirect.github.com/ray-project/ray/issues/63479";>#63479</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63740";>#63740</a>).</li> </ul> <h1>Ray Data</h1> <h3>🎉 New Features</h3> <ul> <li>Enable <code>DataSourceV2</code> by default via <code>DataContext.use_datasource_v2</code> (<a href="https://redirect.github.com/ray-project/ray/issues/64821";>#64821</a>)</li> <li>New task-based hash shuffle v2 (<code>ShuffleMapOp</code> → <code>ShuffleReduceOp</code>) with <code>join</code>, multi-input reduce, downstream map fusion, and reducer remote args, behind an env flag (<a href="https://redirect.github.com/ray-project/ray/issues/63598";>#63598</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64538";>#64538</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64687";>#64687</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64438";>#64438</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64302";>#64302</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64532";>#64532</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64481";>#64481</a>)</li> <li>Add a <code>Catalog</code> abstraction with a <code>UnityCatalog</code> implementation that can be passed to <code>read_*</code>, and Unity Catalog write support for Parquet and Iceberg (<a href="https://redirect.github.com/ray-project/ray/issues/64193";>#64193</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64519";>#64519</a>)</li> <li>Add <code>read_zarr</code> for Zarr datasets (<a href="https://redirect.github.com/ray-project/ray/issues/63003";>#63003</a>) and <code>read_lerobot</code> for LeRobot v3 datasets (<a href="https://redirect.github.com/ray-project/ray/issues/63821";>#63821</a>)</li> <li>Add <code>PushdownCountFiles</code> optimization to answer <code>count()</code> from Parquet footers (<a href="https://redirect.github.com/ray-project/ray/issues/64763";>#64763</a>)</li> <li>Add common subexpression elimination to the expression optimizer (<a href="https://redirect.github.com/ray-project/ray/issues/63974";>#63974</a>)</li> <li>Add GPU support for <code>Aggregate</code> (<a href="https://redirect.github.com/ray-project/ray/issues/63708";>#63708</a>)</li> <li>Make dataset iteration metrics queryable per split (<a href="https://redirect.github.com/ray-project/ray/issues/64608";>#64608</a>)</li> <li>Add custom operator stats to capture worker-side metrics during task execution (<a href="https://redirect.github.com/ray-project/ray/issues/64221";>#64221</a>)</li> <li>Refactor usage collection into an extensible <code>UsageCallback</code> (<a href="https://redirect.github.com/ray-project/ray/issues/64500";>#64500</a>)</li> <li>Export <code>from_blocks</code> from <code>ray.data</code> (<a href="https://redirect.github.com/ray-project/ray/issues/64127";>#64127</a>)</li> <li>Add support for <code>rapidsmpf-26.4.0</code> (<a href="https://redirect.github.com/ray-project/ray/issues/64324";>#64324</a>)</li> </ul> <h3>💫 Enhancements</h3> <ul> <li>Hide Ray Data internal frames from user-code error tracebacks (<a href="https://redirect.github.com/ray-project/ray/issues/64587";>#64587</a>)</li> <li>Run projection/predicate pushdown before limit pushdown, block optimization of non-deterministic expressions, and make fuse checks consistent between the logical and physical optimizers (<a href="https://redirect.github.com/ray-project/ray/issues/64651";>#64651</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64165";>#64165</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63936";>#63936</a>)</li> <li>Add a two-mode block metadata fetch behind a single <code>MetadataFetcher</code> interface (<a href="https://redirect.github.com/ray-project/ray/issues/64378";>#64378</a>)</li> <li>Use work stealing for file listing and a dynamic work queue for traversals (<a href="https://redirect.github.com/ray-project/ray/issues/64675";>#64675</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64388";>#64388</a>)</li> <li>Speed up <code>ExecutionResources</code> and the reservation/budget loops on the scheduling hot path (<a href="https://redirect.github.com/ray-project/ray/issues/63964";>#63964</a>)</li> <li>Fix <code>O(n^2)</code> schema reconciliation in <code>unify_schemas</code> and avoid per-column <code>Series</code> materialization in tensor-column casting (<a href="https://redirect.github.com/ray-project/ray/issues/64555";>#64555</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64038";>#64038</a>)</li> <li>Decouple collate and memory pinning in <code>iter_torch_batches</code>, add per-stage training-thread blocking attribution, bound in-flight <code>iter_threaded</code> items, and finalize after reordering under <code>preserve_order</code> (<a href="https://redirect.github.com/ray-project/ray/issues/64653";>#64653</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64183";>#64183</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64219";>#64219</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64282";>#64282</a>)</li> <li>Add default logical memory for map operators, wire the materializing-op filter into <code>OpResourceAllocator</code> budgeting, and move <code>estimate_object_store_usage</code> into the physical op (<a href="https://redirect.github.com/ray-project/ray/issues/63814";>#63814</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63665";>#63665</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63961";>#63961</a>)</li> <li>Decrease the downstream backpressure ratio to 2.0 (<a href="https://redirect.github.com/ray-project/ray/issues/64352";>#64352</a>)</li> <li>Route native <code>S3FileSystem</code> downloads to the PyArrow threaded path (<a href="https://redirect.github.com/ray-project/ray/issues/64089";>#64089</a>)</li> <li>Size hash-shuffle aggregators via bounded online sampling and add a timeout to the batched <code>ray.get</code> when fetching partitions (<a href="https://redirect.github.com/ray-project/ray/issues/63929";>#63929</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64256";>#64256</a>)</li> <li>Replace manual block reference accounting with Ray Core out-of-scope object callbacks (<a href="https://redirect.github.com/ray-project/ray/issues/64011";>#64011</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64157";>#64157</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64191";>#64191</a>)</li> <li>Default <code>read_numpy</code> to <code>allow_pickle=False</code> and make it manually configurable (<a href="https://redirect.github.com/ray-project/ray/issues/64684";>#64684</a>)</li> <li>Make <code>write_lance(mode=CREATE)</code> error instead of silently overwriting (<a href="https://redirect.github.com/ray-project/ray/issues/64364";>#64364</a>)</li> <li>Expand <code>DistributionTracker</code> with <code>merge()</code> and p25/p75, and add dead node counts and detected issues to usage collection (<a href="https://redirect.github.com/ray-project/ray/issues/64074";>#64074</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64459";>#64459</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64198";>#64198</a>)</li> <li>Remove cluster autoscaler v1, rename the subcluster label key to <code>ray-subcluster</code>, avoid scaling nodegroups dedicated to the head node, and quiet autoscaling coordinator logs (<a href="https://redirect.github.com/ray-project/ray/issues/64380";>#64380</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64003";>#64003</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63918";>#63918</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63534";>#63534</a>)</li> <li>Delay the "cluster resources not enough" warning until an operator is persistently starved, and lower the high-memory warning threshold (<a href="https://redirect.github.com/ray-project/ray/issues/63969";>#63969</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64124";>#64124</a>)</li> <li>Remove <code>ExecutionPlan</code>, <code>_num_outputs</code>, <code>batch_format</code> on <code>AllToAllOperators</code>, and <code>InheritBatchFormatRule</code>; use <code>input_dependencies</code> in logical operators (<a href="https://redirect.github.com/ray-project/ray/issues/63662";>#63662</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64167";>#64167</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64152";>#64152</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64149";>#64149</a>, <a href="https://redirect.github.com/ray-project/ray/issues/64148";>#64148</a>)</li> <li>Deprecate low-level scheduling APIs (<code>DataContext.scheduling_strategy</code>, <code>actor_locality_enabled</code>, <code>exclude_resources</code>, <code>local://</code>) ahead of the actor-only rearchitecture (<a href="https://redirect.github.com/ray-project/ray/issues/64632";>#64632</a>)</li> <li>Migrate the Daft dependency from <code>getdaft</code> to <code>daft</code> (<a href="https://redirect.github.com/ray-project/ray/issues/64240";>#64240</a>)</li> <li>Migrate <code>apply_chat_template</code>/<code>tokenize</code>/<code>detokenize</code> callers to <code>*_stage</code> form, and remove <code>PrepareImageStage</code> while deprecating the <code>image</code> row column (<a href="https://redirect.github.com/ray-project/ray/issues/63590";>#63590</a>, <a href="https://redirect.github.com/ray-project/ray/issues/63570";>#63570</a>)</li> </ul> <h3>🔨 Fixes</h3> <ul> <li>Fix <code>TensorDtype.__from_arrow__</code> crash on empty tensor columns (<a href="https://redirect.github.com/ray-project/ray/issues/64767";>#64767</a>)</li> <li>Fix Arrow-backed <code>to_pandas</code> regressions with an opt-out flag and int/float block overflow handling (<a href="https://redirect.github.com/ray-project/ray/issues/64768";>#64768</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ray-project/ray/commit/878455a3d8f5bcea569d8c40a39b7b5681a55125";><code>878455a</code></a> [docker] Update latest Docker dependencies for 2.57.0 release (<a href="https://redirect.github.com/ray-project/ray/issues/65345";>#65345</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/4464bbb6e21bce4893ed3396b2e33996188d511d";><code>4464bbb</code></a> [release/2.57.0][Data] Propagate <code>isolate_read_workers</code> to DatasourceV2 (<a href="https://redirect.github.com/ray-project/ray/issues/651";>#651</a>...</li> <li><a href="https://github.com/ray-project/ray/commit/8778646dbcf13095b5d4284fd9dd05e3ff1cb622";><code>8778646</code></a> [Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack … (<a href="https://redirect.github.com/ray-project/ray/issues/65177";>#65177</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/133ea78f4de7846ec37be0692553c8ff0b9112e8";><code>133ea78</code></a> [2.57.0][core][observability] Fix deadlock between metric registration and co...</li> <li><a href="https://github.com/ray-project/ray/commit/8d36c028c4e6021064c3b39dea7bfae56faf72ff";><code>8d36c02</code></a> [release/2.57.0][core] Import pytest lazily in test_utils to fix runtime_env_...</li> <li><a href="https://github.com/ray-project/ray/commit/d1fbb27e2bf2a74f169c477c66f18b4b4d1246ef";><code>d1fbb27</code></a> [cherry-pick][Observability] Add GPU uuid to the labels of GPU metrics (<a href="https://redirect.github.com/ray-project/ray/issues/6511";>#6511</a>...</li> <li><a href="https://github.com/ray-project/ray/commit/44e5ef3e607abb2d164088cf6c81ea8dca47e81c";><code>44e5ef3</code></a> [release-2.57.0][release] Fix huggingface_accelerate release test: floor peft...</li> <li><a href="https://github.com/ray-project/ray/commit/39713169c80869e17beba6741678bf24602b93f1";><code>3971316</code></a> [cherry-pick][llm] Add Ray Serve LLM SGLang metrics dashboard (<a href="https://redirect.github.com/ray-project/ray/issues/64797";>#64797</a>) (<a href="https://redirect.github.com/ray-project/ray/issues/65083";>#65083</a>)</li> <li><a href="https://github.com/ray-project/ray/commit/592b3f440fcc4c3f2bb2e09d53b4fe7403904f38";><code>592b3f4</code></a> [release/2.57.0][Data] Add tf-keras to <code>text_embedding</code> pip install packages ...</li> <li><a href="https://github.com/ray-project/ray/commit/e09992fa92d365d88acbdaaf1c4099d48eb915e1";><code>e09992f</code></a> [Cherry-pick][2.57.0][release] Add hello_world_py314 smoke release test; fix ...</li> <li>Additional commits viewable in <a href="https://github.com/ray-project/ray/compare/ray-2.56.1...ray-2.57.0";>compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
