Re: [ANNOUNCE] Apache Spark 3.5.1 released

Congratulations and thanks Jungtaek for driving this! Xinrong Meng ezt írta (időpont: 2024. márc. 1., P, 5:24): > Congratulations! > > Thanks, > Xinrong > > On Thu, Feb 29, 2024 at 11:16 AM Dongjoon Hyun > wrote: > >> Congratulations! >> >> Bests, >> Dongjoon. >> >> On Wed, Feb 28, 2024 at

Re: [ANNOUNCE] Apache Spark 3.5.1 released

Thanks for reporting - this is odd - the dropdown did not exist in other recent releases. https://spark.apache.org/docs/3.5.0/api/python/index.html https://spark.apache.org/docs/3.4.2/api/python/index.html https://spark.apache.org/docs/3.3.4/api/python/index.html Looks like the dropdown feature

Re: [ANNOUNCE] Apache Spark 3.5.1 released

BTW, Jungtaek. PySpark document seems to show a wrong branch. At this time, `master`. https://spark.apache.org/docs/3.5.1/api/python/index.html PySpark Overview Date: Feb 24, 2024 Version: master

[DISCUSS] SPIP: Structured Spark Logging

Hi All, I propose to enhance our logging system by transitioning to structured logs. This initiative is designed to tackle the challenges of analyzing distributed logs from drivers, workers, and executors by allowing them to be queried using a fixed schema. The goal is to improve the

Re: When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

Please use the url as thr full string including '()' part. Or you can seach directly at ASF Jira with 'Spark' project and three labels, 'Correctness', 'correctness' and 'data-loss'. Dongjoon On Thu, Feb 29, 2024 at 11:54 Prem Sahoo wrote: > Hello Dongjoon, > Thanks for emailing me. > Could

Re: [ANNOUNCE] Apache Spark 3.5.1 released

Excellent work, congratulations! On Wed, Feb 28, 2024 at 10:12 PM Dongjoon Hyun wrote: > Congratulations! > > Bests, > Dongjoon. > > On Wed, Feb 28, 2024 at 11:43 AM beliefer wrote: > >> Congratulations! >> >> >> >> At 2024-02-28 17:43:25, "Jungtaek Lim" >> wrote: >> >> Hi everyone, >> >> We

Re: [ANNOUNCE] Apache Spark 3.5.1 released

Congratulations Sent from my iPhoneOn Feb 29, 2024, at 4:54 PM, Xinrong Meng wrote:Congratulations!Thanks,XinrongOn Thu, Feb 29, 2024 at 11:16 AM Dongjoon Hyun wrote:Congratulations!Bests,Dongjoon.On Wed, Feb 28, 2024 at 11:43 AM beliefer

Re: When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

Hello Dongjoon, Thanks for emailing me. Could you please share a list of fixes as the link provided by you is not working. On Thu, Feb 29, 2024 at 11:27 AM Dongjoon Hyun wrote: > Hi, > > If you are observing correctness issues, you may hit some old (and fixed) > correctness issues. > > For

Re: [ANNOUNCE] Apache Spark 3.5.1 released

Congratulations! Thanks, Xinrong On Thu, Feb 29, 2024 at 11:16 AM Dongjoon Hyun wrote: > Congratulations! > > Bests, > Dongjoon. > > On Wed, Feb 28, 2024 at 11:43 AM beliefer wrote: > >> Congratulations! >> >> >> >> At 2024-02-28 17:43:25, "Jungtaek Lim" >> wrote: >> >> Hi everyone, >> >> We

Re: When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

Hi, If you are observing correctness issues, you may hit some old (and fixed) correctness issues. For example, from Apache Spark 3.2.1 to 3.2.4, we fixed 31 correctness issues.

When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why. We have scenario when spark job complains FetchFailedException as one of the data node got rebooted middle of job running . Now due to this we have few duplicate data and