cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-695161242
> @cchighman > Thanks for understanding. I still think even in SS case we want to provide timestamp to effectively filter out files - the point is whether the timestamp is static or dynamic. Let's focus on the batch case here. This PR initially allowed support for both the batch and streaming use cases. Semantics for the option were to be shared and filtering implemented in the same place.. A rather long, in-depth, and thoughtful discussion ensued where @HeartSaVioR patiently outlined subtle differences between approaches. The result of this conversation was to restrict this PR to only focus on the batch use case. while creating a JIRA item (SPARK-32155) to track file-based semantics for the streaming scenario. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
