Re: Issue with Materialized Views in Spark SQL

I do not think the issue is with DROP MATERIALIZED VIEW only, but also with CREATE MATERIALIZED VIEW, because neither is supported in Spark. I guess you must have created the view from Hive and are trying to drop it from Spark and that is why you are running to the issue with DROP first. There is

Re: [VOTE] SPIP: Catalog API for view metadata

1: Accept the proposal as an official SPIP > [ ] +0 > [ ] -1: I don’t think this is a good idea because … > > Thanks! > > On Fri, Jun 4, 2021 at 1:46 PM Walaa Eldin Moustafa > wrote: > >> Considering the API aspect, the ViewCatalog API sounds like a good idea. >> A v

Re: [VOTE] SPIP: Catalog API for view metadata

Considering the API aspect, the ViewCatalog API sounds like a good idea. A view catalog will enable us to integrate Coral (our view SQL translation and management layer) very cleanly to Spark. Currently we can only do it by maintaining our special

Re: [DISCUSS] SPIP: FunctionCatalog

avoid boxing with this >>> API design. >>> >>> To put a detailed proposal: let's have ScalarFuncion0, ScalarFuncion1, >>> ..., ScalarFuncion9 and VarargsScalarFunction. At execution time, if >>> Spark sees ScalarFuncion0-9, pass the input columns to the UDF >>

Re: [DISCUSS] SPIP: FunctionCatalog

I agree with Ryan on the questions around the expressivity of the Invoke method. It is not clear to me how the Invoke method can be used to declare UDFs with type-parameterized parameters. For example: a UDF to get the Nth element of an array (regardless of the Array element type) or a UDF to

Re: [DISCUSS] SPIP: FunctionCatalog

Ye Xianjin, Transport is a UDF API that can be used to write UDFs once and have them translated to native versions of multiple engine UDFs, such as Spark, Presto, and Hive, or data source UDFs such as Avro (others could be added). To get a feel of the UDF

Re: SPIP: Catalog API for view metadata

ed schema > unless you do something like SELECT *. > > On Fri, Aug 14, 2020 at 8:39 AM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> +1 to making views as special forms of tables. Sometimes a table can be >> converted to a view to hide some of the im

Re: SPIP: Catalog API for view metadata

+1 to making views as special forms of tables. Sometimes a table can be converted to a view to hide some of the implementation details while not impacting readers (provided that the write path is controlled). Also, views can be defined on top of either other views or base tables, so the less

Re: [SQL] Is it worth it (and advisable) to implement native UDFs?

For a general-purpose code example, you may take a look at the class we defined in Transport UDFs to express all Expression UDFs [1]. This is an internal class though and not a user-facing API. User-facing UDF example is in [2]. It leverages [1] behind the scenes. [1]

Re: [SQL] Is it worth it (and advisable) to implement native UDFs?

Hi, At LinkedIn, we have some benchmarks that show that UDFs in the Expression API are more performant than Hive Generic UDFs (I am not sure which APIs you used to implement your baseline, but I expect Scala UDFs or Hive Generic UDFs). In fact, we have built a full fledged UDF API (scalar for