Re: Why hint does not traverse down subquery alias

A meaningful error message will be great! On Tue, Jun 11, 2019 at 6:15 PM Maryann Xue wrote: > BTW, I've actually just done some work on hint error handling, which might > be helpful to what you mentioned: > > https://github.com/apache/spark/pull/24653 > > On Tue, Jun 11, 2019 at 8:04 PM

Re: Why hint does not traverse down subquery alias

BTW, I've actually just done some work on hint error handling, which might be helpful to what you mentioned: https://github.com/apache/spark/pull/24653 On Tue, Jun 11, 2019 at 8:04 PM Maryann Xue wrote: > I believe in the SQL standard, the original name cannot be accessed once > it’s aliased.

Re: Why hint does not traverse down subquery alias

I believe in the SQL standard, the original name cannot be accessed once it’s aliased. On Tue, Jun 11, 2019 at 7:54 PM John Zhuge wrote: > Yeah, it is a touch scenario. > > I actually have much simpler cases: > > 1) select /*+ broadcast(t1) */ * from db.t1 join db.t2 on t1.id = t2.id; > 2)

FlatMapGroupsInPandasExec with multiple record batches

Hi, I see the following comment in FlatMapGroupsInPandasExec.scala : "It's possible to further split one group into multiple record batches to reduce the

Re: Why hint does not traverse down subquery alias

Yeah, it is a touch scenario. I actually have much simpler cases: 1) select /*+ broadcast(t1) */ * from db.t1 join db.t2 on t1.id = t2.id; 2) select /*+ broadcast(t1) */ * from db.t1 a1 join db.t2 a2 on a1.id = a2.id; 2) is the same as 1) but with aliases. Many users were surprised that 2)

Re: Why hint does not traverse down subquery alias

Yes, and for a good reason: the hint relation has exactly the same scope with other elements of queries/sub-queries. Suppose there's a query like: select /*+ broadcast(s) */ from (select a, b from s) t join (select a, b from t) s on t1.a = t2.b If we allowed the hint resolving to "cross" the

RE: Adding Custom finalize method to RDDs.

I want to delete some files which I created In my datasource api, as soon as the RDD is cleaned up. Thanks, Nasrulla From: Vinoo Ganesh Sent: Monday, June 10, 2019 1:32 PM To: Nasrulla Khan Haris ; dev@spark.apache.org Subject: Re: Adding Custom finalize method to RDDs. Generally overriding

Why hint does not traverse down subquery alias

Hi Reynold and Maryann, ResolveHints javadoc indicates the traversal does not go past subquery alias. Is there any specific reason? Thanks, John Zhuge

Re: [SS] Why EventTimeStatsAccum for event-time watermark not a named accumulator?

Hi, After some thinking about it, I may have found out the reason why not to expose EventTimeStatsAccum as a named accumulator. The reason is that it's an internal part of how event-time watermark works and should not be exposed via web UI as much as if it was part of a Spark app (the web UI is