Re: Missing data in spark output

Hi, We have observed similar behavior in older versions of spark. But we were are currently using 3.3.0 where we have not seen such issues. Which version of Spark and Hadoop are you using? On 18/10/2022 19:48, Sandeep Vinayak wrote: Hello Everyone, We are recently observing an intermittent

Re: Welcome Yikun Jiang as a Spark committer

Well deserved! Congrats! -Rui On Mon, Oct 10, 2022 at 9:07 AM Xinrong Meng wrote: > Congratulations, Yikun! Well deserved. > > On Sun, Oct 9, 2022 at 9:36 PM John Zhuge wrote: > >> Congratulations, Yikun! >> >> On Sun, Oct 9, 2022 at 8:52 PM Senthil Kumar wrote: >> >>> Congratulations Yikun

Re: [DISCUSS] Flip the default value of Kafka offset fetching config (spark.sql.streaming.kafka.useDeprecatedOffsetFetching)

No further voice so far. I'm going to submit a PR. Thanks again for the feedback! On Mon, Oct 17, 2022 at 9:30 AM Jungtaek Lim wrote: > Thanks Gabor and Dongjoon for supporting this! > > Bump to reach more eyes. If there is no further voice on this in a couple > of days, I'll consider it as a

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 (non-binding) - Passed Apache Kyuubi (Incubating) integration tests[1] - Run some jobs on our internal K8s cluster [1] https://github.com/apache/incubator-kyuubi/pull/3507 Thanks, Cheng Pan On Wed, Oct 19, 2022 at 9:13 AM Yikun Jiang wrote: > > +1, also test passed with spark-docker

Re: Apache Spark 3.2.3 Release?

+1 发件人: vaquar khan 日期: 2022年10月19日 星期三 10:08 收件人: "416161...@qq.com" 抄送: Yuming Wang , kazuyuki tanimura , Gengliang Wang , huaxin gao , Dongjoon Hyun , Sean Owen , Chao Sun , dev 主题: Re: Apache Spark 3.2.3 Release? +1 On Tue, Oct 18, 2022, 8:58 PM

Re: Apache Spark 3.2.3 Release?

+1 On Tue, Oct 18, 2022, 8:58 PM 416161...@qq.com wrote: > +1 > > -- > Ruifeng Zheng > ruife...@foxmail.com > >

Re: Apache Spark 3.2.3 Release?

+1 RuifengZheng ruife...@foxmail.com --Original-- From: "Yuming Wang"

Re: Apache Spark 3.2.3 Release?

+1 On Wed, Oct 19, 2022 at 4:17 AM kazuyuki tanimura wrote: > +1 Thanks Chao! > > > Kazu > > On Oct 18, 2022, at 11:48 AM, Gengliang Wang wrote: > > +1. Thanks Chao! > > On Tue, Oct 18, 2022 at 11:45 AM huaxin gao > wrote: > >> +1 Thanks Chao! >> >> Huaxin >> >> On Tue, Oct 18, 2022 at 11:29

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1, also test passed with spark-docker workflow (downloading rc4 tgz, extract, build image, run K8s IT) [1] https://github.com/Yikun/spark-docker/pull/9 Regards, Yikun On Wed, Oct 19, 2022 at 8:59 AM Wenchen Fan wrote: > +1 > > On Wed, Oct 19, 2022 at 4:59 AM Chao Sun wrote: > >> +1. Thanks

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 On Wed, Oct 19, 2022 at 4:59 AM Chao Sun wrote: > +1. Thanks Yuming! > > Chao > > On Tue, Oct 18, 2022 at 1:18 PM Thomas graves wrote: > > > > +1. Ran internal test suite. > > > > Tom > > > > On Sun, Oct 16, 2022 at 9:14 PM Yuming Wang wrote: > > > > > > Please vote on releasing the

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1. Thanks Yuming! Chao On Tue, Oct 18, 2022 at 1:18 PM Thomas graves wrote: > > +1. Ran internal test suite. > > Tom > > On Sun, Oct 16, 2022 at 9:14 PM Yuming Wang wrote: > > > > Please vote on releasing the following candidate as Apache Spark version > > 3.3.1. > > > > The vote is open

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1. Ran internal test suite. Tom On Sun, Oct 16, 2022 at 9:14 PM Yuming Wang wrote: > > Please vote on releasing the following candidate as Apache Spark version > 3.3.1. > > The vote is open until 11:59pm Pacific time October 21th and passes if a > majority +1 PMC votes are cast, with a

Re: Apache Spark 3.2.3 Release?

+1 Thanks Chao! Kazu > On Oct 18, 2022, at 11:48 AM, Gengliang Wang wrote: > > +1. Thanks Chao! > > On Tue, Oct 18, 2022 at 11:45 AM huaxin gao > wrote: > +1 Thanks Chao! > > Huaxin > > On Tue, Oct 18, 2022 at 11:29 AM Dongjoon Hyun

Re: Apache Spark 3.2.3 Release?

+1. Thanks Chao! On Tue, Oct 18, 2022 at 11:45 AM huaxin gao wrote: > +1 Thanks Chao! > > Huaxin > > On Tue, Oct 18, 2022 at 11:29 AM Dongjoon Hyun > wrote: > >> +1 >> >> Thank you for volunteering, Chao! >> >> Dongjoon. >> >> >> On Tue, Oct 18, 2022 at 9:55 AM Sean Owen wrote: >> >>> OK by

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 from me, same as last time. On Tue, Oct 18, 2022 at 11:45 AM L. C. Hsieh wrote: > +1 > > Thanks Yuming! > > On Tue, Oct 18, 2022 at 11:28 AM Dongjoon Hyun > wrote: > > > > +1 > > > > Thank you, Yuming and all! > > > > Dongjoon. > > > > > > On Tue, Oct 18, 2022 at 9:22 AM Yang,Jie(INF) >

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 Thanks Yuming! On Tue, Oct 18, 2022 at 11:28 AM Dongjoon Hyun wrote: > > +1 > > Thank you, Yuming and all! > > Dongjoon. > > > On Tue, Oct 18, 2022 at 9:22 AM Yang,Jie(INF) wrote: >> >> Use maven to test Java 17 + Scala 2.13 and test passed, +1 for me >> >> >> >> 发件人: Sean Owen >> 日期:

Re: Apache Spark 3.2.3 Release?

+1 Thanks Chao! Huaxin On Tue, Oct 18, 2022 at 11:29 AM Dongjoon Hyun wrote: > +1 > > Thank you for volunteering, Chao! > > Dongjoon. > > > On Tue, Oct 18, 2022 at 9:55 AM Sean Owen wrote: > >> OK by me, if someone is willing to drive it. >> >> On Tue, Oct 18, 2022 at 11:47 AM Chao Sun

Re: Apache Spark 3.2.3 Release?

+1 Thanks Chao! On Tue, Oct 18, 2022 at 11:30 AM Dongjoon Hyun wrote: > > +1 > > Thank you for volunteering, Chao! > > Dongjoon. > > > On Tue, Oct 18, 2022 at 9:55 AM Sean Owen wrote: >> >> OK by me, if someone is willing to drive it. >> >> On Tue, Oct 18, 2022 at 11:47 AM Chao Sun wrote: >>>

Re: Apache Spark 3.2.3 Release?

+1 Thank you for volunteering, Chao! Dongjoon. On Tue, Oct 18, 2022 at 9:55 AM Sean Owen wrote: > OK by me, if someone is willing to drive it. > > On Tue, Oct 18, 2022 at 11:47 AM Chao Sun wrote: > >> Hi All, >> >> It's been more than 3 months since 3.2.2 (tagged at Jul 11) was >> released

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 Thank you, Yuming and all! Dongjoon. On Tue, Oct 18, 2022 at 9:22 AM Yang,Jie(INF) wrote: > Use maven to test Java 17 + Scala 2.13 and test passed, +1 for me > > > > *发件人**: *Sean Owen > *日期**: *2022年10月17日 星期一 21:34 > *收件人**: *Yuming Wang > *抄送**: *dev > *主题**: *Re: [VOTE] Release

Missing data in spark output

Hello Everyone, We are recently observing an intermittent data loss in the spark with output to GCS (google cloud storage). When there are missing rows, they are accompanied by duplicate rows. The re-run of the job doesn't have any duplicate or missing rows. Since it's hard to debug, we are first

Re: Apache Spark 3.2.3 Release?

OK by me, if someone is willing to drive it. On Tue, Oct 18, 2022 at 11:47 AM Chao Sun wrote: > Hi All, > > It's been more than 3 months since 3.2.2 (tagged at Jul 11) was > released There are now 66 patches accumulated in branch-3.2, including > 2 correctness issues. > > Is it a good time to

Apache Spark 3.2.3 Release?

Hi All, It's been more than 3 months since 3.2.2 (tagged at Jul 11) was released There are now 66 patches accumulated in branch-3.2, including 2 correctness issues. Is it a good time to start a new release? If there's no objection, I'd like to volunteer as the release manager for the 3.2.3

Re: [VOTE] Release Spark 3.3.1 (RC4)

Use maven to test Java 17 + Scala 2.13 and test passed, +1 for me 发件人: Sean Owen 日期: 2022年10月17日 星期一 21:34 收件人: Yuming Wang 抄送: dev 主题: Re: [VOTE] Release Spark 3.3.1 (RC4) +1 from me, same as last time On Sun, Oct 16, 2022 at 9:14 PM Yuming Wang mailto:wgy...@gmail.com>> wrote: Please