Re: [DISCUSS] Un-deprecate Trigger.Once

2024-04-19 Thread Dongjoon Hyun
For that case, I believe it's enough for us to revise the deprecation message only by making sure that Apache Spark will keep it without removal for backward-compatibility purposes only. That's what the users asked, isn't that? > deprecation of Trigger.Once confuses users that the trigger won't

[DISCUSS] Un-deprecate Trigger.Once

2024-04-19 Thread Jungtaek Lim
Hi dev, I'd like to raise a discussion to un-deprecate Trigger.Once in future releases. I've proposed deprecation of Trigger.Once because it's semantically broken and we made a change, but we've realized that there are really users who strictly require the behavior of Trigger.Once (only run a

[DISCUSS] SPIP: Stored Procedures API for Catalogs

2024-04-19 Thread Anton Okolnychyi
Hi folks, I'd like to start a discussion on SPARK-44167 that aims to enable catalogs to expose custom routines as stored procedures. I believe this functionality will enhance Spark’s ability to interact with external connectors and allow users to perform more operations in plain SQL. SPIP [1]

Re: Which version of spark version supports parquet version 2 ?

2024-04-19 Thread Steve Loughran
Those are some quite good improvements -but committing to storing all your data in an unstable format, is, well, "bold". For temporary data as part of a workflow though, it could be appealing Now, assuming you are going to be working with s3, you might want to start with merging PARQUET-2117 into