Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17371
gentle ping @uncleGen, is this PR still active?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17371
Can we add an analysis rule that just pulls up missing metadata from
attributes in the child? It could run once after other rules.
---
If your project is set up for it, you can reply to this
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17371
yeah, I just tried it. `IncrementalExecution` will re-new the attribute for
each batch. Although we can replace the attribute...
---
If your project is set up for it, you can reply to this email
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17371
I don't think that will solve the problem though. You will just get a
different error message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17371
Unfortunately, yes, allowing resolved attributes in user API will have this
kind of trouble.
> However, I don't think that piecemeal switching to unresolved attributes
is a good idea.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17371
I really think the core problem here is that we allow you to use resolved
attributes at all in the user API. Unfortunately we are somewhat stuck with
that bad decision. Personally, I never use
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17371
IMHO, the output after `withWatermark` should be new attribute and have new
expression id. Maybe @zsxwing @marmbrus have more insights on this?
Btw, does this issue also happen in Scala
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/17371
@viirya Great, you give a more clear explanation.
> I am thinking, should we create new expression id for the watermarking
column with withWatermark? So we must write the query like:
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17371
For now, after `withWatermark`, we only update the metadata for the column
of event time. The expression id is the same. So once we use the column before
adding watermark `words.timestamp` as
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74967/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17371
**[Test build #74967 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74967/testReport)**
for PR 17371 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17371
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74959/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17371
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17371
**[Test build #74959 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74959/testReport)**
for PR 17371 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17371
**[Test build #74959 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74959/testReport)**
for PR 17371 at commit
16 matches
Mail list logo
