Re: [ANNOUNCE] Apache Spark 3.1.3 released + Docker images

Hi, seems - [SPARK-35391] : Memory leak in ExecutorAllocationListener breaks dynamic allocation under high load Links to wrong jira ticket? Mich Talebzadeh 于2022年2月22日周二 15:49写道： > Well, that is pretty easy to do. > > However, a

Re: [Apache Spark Jenkins] build system shutting down Dec 23th, 2021

Thanks for your great work! Shane. Best Regards Angerszh Yikun Jiang 于2021年12月28日周二 10:29写道： > Thanks for your works! Shane! > > Regards, > Yikun > > > shane knapp ☠ 于2021年12月28日周二 06:48写道： > >> # sysctl stop jenkins >> # >> >>  >> >> goodbye jenkins!  >> >> On Mon, Dec 6, 2021 at

Hadoop profile change to hadoop-2 and hadoop-3 since Spark 3.3

Hi all, Since Spark 3.2, we have supported Hadoop 3.3.1 now, but its profile name is *hadoop-3.2* (and *hadoop-2.7*) that is not correct. So we made a change in https://github.com/apache/spark/pull/34715 Starting from Spark 3.3, we use hadoop profile *hadoop-2* and *hadoop-3 *, and default hadoop

Re: [DISCUSS] SPIP: Storage Partitioned Join for Data Source V2

ng sense to me. >> >> Would be great to have some feedback from people such as @Wenchen Fan >> @Cheng Su @angers zhu >> . >> >> >> On Tue, 26 Oct 2021 at 17:25, Dongjoon Hyun >> wrote: >> >>> +1 for this SPIP. >>> >>> On

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 (non-binding) Cheng Pan 于2021年10月9日周六 下午2:06写道： > +1 (non-binding) > > Integration test passed[1] with my project[2]. > > [1] > https://github.com/housepower/spark-clickhouse-connector/runs/3834335017 > [2] https://github.com/housepower/spark-clickhouse-connector > > Thanks, > Cheng Pan > >

Discuss about current yarn client mode problem

Hi devs, In current yarn-client mode, we have several problem, 1. When AM lost connection with driver, it will just finish application with final status of SUCCESS, then YarnClientSchedulerBackend.MonitorThread will got application status with SUCCESS final status and then call

Re: java.lang.ClassNotFoundException for custom hive authentication

Which version? Jason Jun 于2021年6月22日周二 下午4:19写道： > Hi there, > > I'm leveraging thriftserver to provide sql service, and using custom hive > authentication: > -- > > hive.server2.custom.authentication.class > com.abc.ABCAuthenticationProvider > > > > I've got this error when

Re: [Spark Core]: Adding support for size based partition coalescing

Hi all, you mean something like this https://github.com/apache/spark/pull/27248/files? If you need I can raise a pr add a SizeBasedCoaleaser mhawes 于2021年3月30日周二 下午9:06写道： > Hi Pol, I had considered repartitioning but the main issue for me there is > that it will trigger a shuffle and could

Re: Welcoming six new Apache Spark committers

Congratulations Prashant Sharma 于2021年3月27日周六 上午8:35写道： > Congratulations  all!! > > On Sat, Mar 27, 2021, 5:10 AM huaxin gao wrote: > >> Congratulations to you all!! >> >> On Fri, Mar 26, 2021 at 4:22 PM Yuming Wang wrote: >> >>> Congrats! >>> >>> On Sat, Mar 27, 2021 at 7:13 AM Takeshi

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Great work, Hyukjin ! Bests, Angers Wenchen Fan 于2021年3月3日周三 下午5:02写道： > Great work and congrats! > > On Wed, Mar 3, 2021 at 3:51 PM Kent Yao wrote: > >> Congrats, all! >> >> Bests, >> *Kent Yao * >> @ Data Science Center, Hangzhou Research Institute, NetEase Corp. >> *a spark enthusiast* >>

Chnage Restful API's default quanlites same as WebUI

Hi devs,These days I found that about taskSummary, the Restful API(/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary)'s default quantiles is `0.05,0.25,0.5,0.75,0.95`But in Spark Web UI’s stage page, the quantiles is `0.0,0.25,0.5,0.75,1.0`. And

Re: [VOTE] Standardize Spark Exception Messages SPIP

+1

Re: [VOTE][SPARK-30602] SPIP: Support push-based shuffle to improve shuffle efficiency

+1

Re: Welcoming some new Apache Spark committers

Congratulations !

Re: Spark Thrift Server java vm problem need help

If -Xmx is bigger then 32g, vm will not to use  UseCompressedOops as default, We can see a case, If we set spark.driver.memory is 64g, set -XX:+UseCompressedOops in spark.executor.extralJavaOptions, and set SPARK_DAEMON_MEMORY = 6g, Use current code , vm will got command like

Re: Spark Thrift Server java vm problem need help

Hi Sean, Yea,  I set  -XX:+UseCompressedOops in driver(you can see in command line) and these days, we have more user and I set spark.driver.memory to 64g, in Non-default VM flags it should be +XX:-UseCompressdOops , but it’s still +XX:-UseCompressdOops. I have find the

Spark Thrift Server java vm problem need help

Hi developers, These day I meet a strange problem and I can’t find whyWhen I start a spark thrift server with  spark.driver.memory 64g, then use jdk8/bin/jinfo pid to see vm flags got below information,In 64g vm, UseCompressedOops should be closed by default, why spark

Re: Fw: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Hi all,Blow is our working repohttps://github.com/spark-thriftserver/spark-thriftserver Hope for your good suggestions.

Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Hi all, I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. Hope for your suggestions and ideas.SPIP DOC : 

[VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

With the development of Spark and Hive，in current sql/hive-thriftserver module, we need to do a lot of work to solve code conflicts for different built-in hive versions.It's an annoying and unending work in current ways. And these issues have limited our

Re: A question about broadcast nest loop join

where not in ( query block)condition will been change to LeftSemi join in optimizer rule RewritePredicateSubquery. Then as cloud-fan said,  it will be change to a BroadCastNestLoopJoin

Re: Welcoming some new committers and PMC members

Congratulations

[Spark SQL] Any intersted in do SQL between two hive metastore.

Best RegardsAngers