Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17268
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17268
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74660/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17268
**[Test build #74660 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74660/testReport)**
for PR 17268 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17268
**[Test build #74660 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74660/testReport)**
for PR 17268 at commit
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17268
Sorry, I wasn't suggestion we mandate this. There may be use cases where
users are okay deduping a short lived stream w/o a watermark. I'm only saying
the timestamp is mandatory for the
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/17268
Thank you @marmbrus for the detailed explanation!
> For that reason, I think its safest to require the user to explicitly
include the timestamp.
Yea, let me update this in this
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17268
Say the eventtime column chosen is the time of delivery into something like
Kafka. Due to retries we end up with two events with different timestamps.
Consider the following stream with a
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/17268
@marmbrus thanks for the comments.
> In the worst case... it is possible that the result actually ends up with
duplicates in it.
Ah, could you elaborate? I'm not sure why there
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/17268
I'm mixed if we want this to happen implicitly. Here's how I think about
the tradeoffs for this change: On the pro side, with this change we avoid the
case where the user forgets to include the
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/17268
@marmbrus @zsxwing would you take a look at this, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
10 matches
Mail list logo
