Re: [ANNOUNCE] Apache Spark 3.2.1 released

https://spark.apache.org/downloads.html *2. Choose a package type:* menu shows that Pre-built for Hadoop 3.3 but download link is *spark-3.2.1-bin-hadoop3.2.tgz* need an update? L. C. Hsieh 于2022年1月29日周六 14:26写道： > Thanks Huaxin for the 3.2.1 release! > > On Fri, Jan 28, 2022 at 10:14 PM

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thanks Huaxin for the 3.2.1 release! On Fri, Jan 28, 2022 at 10:14 PM Dongjoon Hyun wrote: > > Thank you again, Huaxin! > > Dongjoon. > > On Fri, Jan 28, 2022 at 6:23 PM DB Tsai wrote: >> >> Thank you, Huaxin for the 3.2.1 release! >> >> Sent from my iPhone >> >> On Jan 28, 2022, at 5:45 PM,

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thank you again, Huaxin! Dongjoon. On Fri, Jan 28, 2022 at 6:23 PM DB Tsai wrote: > Thank you, Huaxin for the 3.2.1 release! > > Sent from my iPhone > > On Jan 28, 2022, at 5:45 PM, Chao Sun wrote: > >  > Thanks Huaxin for driving the release! > > On Fri, Jan 28, 2022 at 5:37 PM Ruifeng

[mongo-spark-connector] How can I improve the performance of Mongo spark write?

hello, I'm having a performance problem with writing with a Mongo spark connector. I want to write 0.3M data in the form of string and hex binary (4kB) through the Mongo spark connector. spark = create_c3s_spark_session(app_name, spark_config=[ ("spark.executor.cores", "1"),

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Is there a guide for upgrading from 3.2.0 to 3.2.1? thanks On Sat, Jan 29, 2022 at 9:14 AM huaxin gao wrote: > We are happy to announce the availability of Spark 3.2.1! > > Spark 3.2.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.2 maintenance

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thank you, Huaxin for the 3.2.1 release! Sent from my iPhone > On Jan 28, 2022, at 5:45 PM, Chao Sun wrote: > >  > Thanks Huaxin for driving the release! > >> On Fri, Jan 28, 2022 at 5:37 PM Ruifeng Zheng wrote: >> It's Great! >> Congrats and thanks, huaxin! >> >> >> --

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thanks Huaxin for driving the release! On Fri, Jan 28, 2022 at 5:37 PM Ruifeng Zheng wrote: > It's Great! > Congrats and thanks, huaxin! > > > -- 原始邮件 -- > *发件人:* "huaxin gao" ; > *发送时间:* 2022年1月29日(星期六) 上午9:07 > *收件人:* "dev";"user"; > *主题:* [ANNOUNCE] Apache

回复：[ANNOUNCE] Apache Spark 3.2.1 released

It's Great! Congrats and thanks, huaxin! -- 原始邮件 -- 发件人: "huaxin gao"

Re: [ANNOUNCE] Apache Spark 3.2.1 released

Thank you Huaxin. On Sat, Jan 29, 2022 at 9:08 AM huaxin gao wrote: > We are happy to announce the availability of Spark 3.2.1! > > Spark 3.2.1 is a maintenance release containing stability fixes. This > release is based on the branch-3.2 maintenance branch of Spark. We strongly > recommend all

[ANNOUNCE] Apache Spark 3.2.1 released

We are happy to announce the availability of Spark 3.2.1! Spark 3.2.1 is a maintenance release containing stability fixes. This release is based on the branch-3.2 maintenance branch of Spark. We strongly recommend all 3.2 users to upgrade to this stable release. To download Spark 3.2.1, head

Kafka to spark streaming

Hello everyone, we have spark streaming application. We send request to stream through Akka actor using Kafka topic. We wait for response as it is real time. Just want a suggestion is there any better option like Livy where we can send and receive request to spark streaming. Thanks Amit

Re: Small optimization questions

Hi, have you looked at the function which says record number in SPARK SQL, or SPARK functions. Having more volume of data means that you may get into memory issues as well. Therefore you scale in and scale out rule has to be responsive, as mentioned by Sean, along with the number of core per

Re: how can I remove the warning message

you can try spark.sparkContext.setLogLevel("ERROR") HTH view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise

how can I remove the warning message

When I submitted the job from scala client, I got the warning messages: WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.2.0.jar) to constructor