Re: Apache Spark 2.4.8 (and EOL of 2.4)

Thank you for the update. +1 for your plan. Bests, Dongjoon. On Tue, Mar 9, 2021 at 12:46 PM Liang-Chi Hsieh wrote: > I just contacted Shane and seems there is ongoing github fetches timing out > issue on Jenkins. > > That being said, currently the QA test is unavailable. I guess it is unsafe

[no subject]

Unsubscribe

Re: Apache Spark 2.4.8 (and EOL of 2.4)

I just contacted Shane and seems there is ongoing github fetches timing out issue on Jenkins. That being said, currently the QA test is unavailable. I guess it is unsafe to make a release cut due to lack of reliable QA test result. I may defer the cut until QA test comes back if no objection.

Re: [build system] github fetches timing out

Unsubscribe On Tue, Mar 9, 2021 at 3:32 PM shane knapp ☠ wrote: > it looks like over the past few days the master/branch builds have been > timing out... this hasn't happened in a few years, and honestly the last > times this happened there was nothing that either i, or github could do > about

[build system] github fetches timing out

it looks like over the past few days the master/branch builds have been timing out... this hasn't happened in a few years, and honestly the last times this happened there was nothing that either i, or github could do about it. it cleared up after a number of weeks, and we were never able to

Re: Property spark.sql.streaming.minBatchesToRetain

That property decides how many log files (log file is created per batch per type - types are like offsets, commits, etc.) to retain on the checkpoint. Unless you're struggling with a small files problem on checkpoint, you wouldn't need to tune the value. I guess that's why the configuration is

Re: Property spark.sql.streaming.minBatchesToRetain

Hey Maxim, ok! I didn't see them. Is this property documented somewhere? Thanks! On Tue, 9 Mar 2021 at 13:57, Maxim Gekk wrote: > Hi German, > > It is used at least at: > 1. >

Re: [VOTE] SPIP: Add FunctionCatalog

+1 (non-binding). Thanks for the work! Erik Krogen wrote > +1 from me (non-binding) > > On Tue, Mar 9, 2021 at 9:27 AM huaxin gao > huaxin.gao11@ > wrote: > >> +1 (non-binding) -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Apache Spark 2.4.8 (and EOL of 2.4)

Hi devs, I'm going to cut the branch yesterday. I'd like to share current progress. I hit a problem during dry run of the release script. Fixed it in SPARK-34672. The latest dry run looks okay as build, docs, publish all success. But the last step (push the tag) has a fatal error, I'm not sure

Re: [VOTE] SPIP: Add FunctionCatalog

+1 from me (non-binding) On Tue, Mar 9, 2021 at 9:27 AM huaxin gao wrote: > +1 (non-binding) > > On Tue, Mar 9, 2021 at 1:12 AM Kent Yao wrote: > >> +1, looks great! >> >> *Kent Yao * >> @ Data Science Center, Hangzhou Research Institute, NetEase Corp. >> *a spark enthusiast* >> *kyuubi

Re: [VOTE] SPIP: Add FunctionCatalog

+1 (non-binding) On Tue, Mar 9, 2021 at 1:12 AM Kent Yao wrote: > +1, looks great! > > *Kent Yao * > @ Data Science Center, Hangzhou Research Institute, NetEase Corp. > *a spark enthusiast* > *kyuubi is a unified multi-tenant JDBC > interface for large-scale

Re: Property spark.sql.streaming.minBatchesToRetain

Hi German, It is used at least at: 1. https://github.com/apache/spark/blob/a093d6feefb0e086d19c86ae53bf92df12ccf2fa/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala#L56 2.

Property spark.sql.streaming.minBatchesToRetain

Hello all, I wanted to ask if this property is still active? I can't find it in the doc https://spark.apache.org/docs/latest/configuration.html or anywhere in the code(only in Tests). If so, should we remove it? val MIN_BATCHES_TO_RETAIN = buildConf("spark.sql.streaming.minBatchesToRetain")

Re: [VOTE] SPIP: Add FunctionCatalog

+1, looks great! Kent Yao @ Data Science Center, Hangzhou Research Institute, NetEase Corp.a spark enthusiastkyuubiis a unified 

Re: [VOTE] SPIP: Add FunctionCatalog

+1 (binding) On Tue, Mar 9, 2021 at 1:47 PM Russell Spitzer wrote: > +1 (for what it's worth) > > Thanks for making such a robust proposal, i'm excited to see the new work > coming from this > > On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun > wrote: > > +1 (binding) > > Thank you, Ryan. > >