[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620310834 Sorry for maybe out-of-topic, but I'm seeing that column metadata may have more issues due to not properly defined its usage and how it should be handled. I'm actually

[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-620272637 Just FYI, looks like the merged commit doesn't reflect the credit properly (not showing as 2 authors) - maybe because of the authorship of the first commit. author and commi

[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-27 Thread GitBox
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-619831846 > Each Attribute/Alias has its own metadata and can easily be hidden by the outer-most Alias. Yeah I see the concern - I'm not sure the column metadata was considered

[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-26 Thread GitBox
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-619751876 > Sometimes we have to, for example, a + b as c, what should be the metadata of c? The metadata of expression for `a + b` if there's no explicit set of metadata. That

[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost

2020-04-26 Thread GitBox
HeartSaVioR commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-619722391 (Please consider that I'm not an expert of SQL area.) I've read through the code around Alias and played with the reproducer (test), and feel #24457 is the fix address