[GitHub] [spark] cchighman commented on a change in pull request #28841: [SPARK-31962][SQL] Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

[GitBox]

cchighman commented on a change in pull request #28841:
URL: https://github.com/apache/spark/pull/28841#discussion_r450576025



##########
File path: examples/src/main/python/sql/datasource.py
##########
@@ -69,6 +69,25 @@ def generic_file_source_options_example(spark):
     # +-------------+
     # $example off:load_with_path_glob_filter$
 
+    # $example on:load_with_modified_date_filter$
+    # Assume file in provided path were last modified today.
+    df = spark.read.load("examples/src/main/resources/dir1",
+                         format="parquet", 
modifiedDateFilter="2020-06-01T08:30:00")

Review comment:
       I wonder if we should be defaulting to UTC unless this option below or a 
spark conf setting for timezone is specified.  If one is specified, interpret 
the timestamp according to that offset from UTC?
   
   
![image](https://user-images.githubusercontent.com/659214/86691815-08c45f00-bfbe-11ea-91e1-57605333aa45.png)
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org