matthewmturner commented on issue #7925: URL: https://github.com/apache/arrow-datafusion/issues/7925#issuecomment-2018009049
@alamb I didnt, but here is the python script I was working on in case it helps anyone. ``` import datafusion import pyarrow ctx = datafusion.SessionContext() # Ref https://github.com/apache/arrow-datafusion/blob/main/datafusion-examples/examples/to_timestamp.rs ctx.sql( "COPY (SELECT * FROM (VALUES (to_timestamp('2023-07-01 00:00:00-04:00', '%Y-%m-%d %H:%M:%S%#z'), to_timestamp('2023-07-01 00:00:00'))) AS data (tz_timestamp, timestamp)) TO 'timestamp.parquet'" ).collect() ctx.register_parquet( "data", "timestamp.parquet", schema=pyarrow.schema( [ ("tz_timestamp", pyarrow.timestamp("ns", "America/New_York")), ("timestamp", pyarrow.timestamp("ns")), ] ), ) # Works without pushdown filters on timestamp without timezone ctx.sql("SELECT * FROM data WHERE timestamp > '2023-01-01 00:00:00'").show() ctx.sql("SELECT * FROM data WHERE timestamp > '2023-01-01 00:00:00-04:00'").show() # Works with pushdown filter on timestamp without timezone ctx.sql("SET datafusion.execution.parquet.pushdown_filters=true").collect() ctx.sql("SELECT * FROM data WHERE timestamp > '2023-01-01 00:00:00'").show() ctx.sql("SELECT * FROM data WHERE timestamp > '2023-01-01 00:00:00-04:00'").show() # Works without pushdown filter on timestamp with timezone ctx.sql("SET datafusion.execution.parquet.pushdown_filters=false").collect() ctx.sql("SELECT * FROM data WHERE tz_timestamp > '2023-01-01 00:00:00'").show() ctx.sql("SELECT * FROM data WHERE tz_timestamp > '2023-01-01 00:00:00-04:00'").show() # Does NOT work with pushdown filter on timestamp with timezone ctx.sql("SET datafusion.execution.parquet.pushdown_filters=true").collect() # ctx.sql("SELECT * FROM data WHERE tz_timestamp > '2023-01-01 00:00:00'").show() ctx.sql("SELECT * FROM data WHERE tz_timestamp > '2023-01-01 00:00:00-04:00'").show() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
