I do not think the issue is with DROP MATERIALIZED VIEW only, but also with
CREATE MATERIALIZED VIEW, because neither is supported in Spark. I guess
you must have created the view from Hive and are trying to drop it from
Spark and that is why you are running to the issue with DROP first.
There is
An issue I encountered while working with Materialized Views in Spark SQL.
It appears that there is an inconsistency between the behavior of
Materialized Views in Spark SQL and Hive.
When attempting to execute a statement like DROP MATERIALIZED VIEW IF
EXISTS test.mv in Spark SQL, I encountered a
+1
发件人: Jungtaek Lim
日期: 2024年5月2日 星期四 10:21
收件人: Holden Karau
抄送: Chao Sun , Xiao Li , Tathagata
Das , Wenchen Fan , Cheng Pan
, Nicholas Chammas , Dongjoon
Hyun , Cheng Pan , Spark dev list
, Anish Shrigondekar
主题: Re: [DISCUSS] Spark 4.0.0 release
+1 love to see it!
On Thu, May 2,
- Integration with additional external data sources or systems, say Hive
- Enhancements to the Spark UI for improved monitoring and debugging
- Enhancements to machine learning (MLlib) algorithms and capabilities,
like TensorFlow or PyTorch,( if any in the pipeline)
HTH
Mich
There's a new parquet RC up this week which would be good to pull in.
On Thu, 2 May 2024 at 03:20, Jungtaek Lim
wrote:
> +1 love to see it!
>
> On Thu, May 2, 2024 at 10:08 AM Holden Karau
> wrote:
>
>> +1 :) yay previews
>>
>> On Wed, May 1, 2024 at 5:36 PM Chao Sun wrote:
>>
>>> +1
>>>
>>>
To add some user perspective, I wanted to share our experience from
automatically upgrading tens of thousands of jobs from Spark 2 to 3 at Palantir:
We didn't mind "loud" changes that threw exceptions. We have some infra to try
run jobs with Spark 3 and fallback to Spark 2 if there's an
Hi Erik and Wenchen,
I think that usually a good practice with public api and with internal api
that has big impact and a lot of usage is to ease in changes by providing
defaults to new parameters that will keep former behaviour in a method with
the previous signature with deprecation notice, and
