Re: why advisoryPartitionSize <= maxShuffledHashJoinLocalMapThreshold

[XiDuo You]

CoalesceShufflePartitions will merge small partitions into bigger
ones. Say, if you set maxShuffledHashJoinLocalMapThreshold to 32MB,
but the advisoryPartitionSize is 64MB, then the final each reducer
partition size will be close to 64MB. It breaks the
maxShuffledHashJoinLocalMapThreshold. So we should make sure the size
of maxShuffledHashJoinLocalMapThreshold is not smaller than
advisoryPartitionSize.

盛森林 <434780...@qq.com> 于2023年8月16日周三 12:12写道：
>
> dear community,
> I want to set  maxShuffledHashJoinLocalMapThreshold to enable convert 
> smj to hash join,but I found maxShuffledHashJoinLocalMapThreshold must be 
> large than advisoryPartitionSize. I want to know what happend if 
> maxShuffledHashJoinLocalMapThreshold < advisoryPartitionSize

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

[ANNOUNCE] Release Apache Kyuubi (Incubating) 1.3.1-incubating

[XiDuo You]

Hi Spark Community,

The Apache Kyuubi (Incubating) community is pleased to announce that

Apache Kyuubi (Incubating) 1.3.1-incubating has been released!

Apache Kyuubi (Incubating) is a distributed multi-tenant JDBC server for

large-scale data processing and analytics, built on top of Apache Spark.

Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface

for end-users to manipulate large-scale data with pre-programmed and

extensible Spark SQL engines.

We are aiming to make Kyuubi an "out-of-the-box" tool for data warehouses

and data lakes.

This "out-of-the-box" model minimizes the barriers and costs for end-users

to use Spark at the client side.

At the server-side, Kyuubi server and engine's multi-tenant architecture

provides the administrators a way to achieve computing resource isolation,

data security, high availability, high client concurrency, etc.

The full release notes and download links are available at:

Release Notes: https://kyuubi.apache.org/release/1.3.1-incubating.html

To learn more about Apache Kyuubi (Incubating), please see

https://kyuubi.apache.org/

Kyuubi Resources:

- Issue: https://github.com/apache/incubator-kyuubi/issues

- Mailing list: d...@kyuubi.apache.org

We would like to thank all contributors of the Kyuubi community and
incubating

community who made this release possible!

Thanks,

On behalf of Apache Kyuubi (Incubating) community