alamb opened a new issue, #18138: URL: https://github.com/apache/datafusion/issues/18138
- part of https://github.com/apache/datafusion/issues/17214 In https://github.com/apache/datafusion/pull/18103 Basically you can do something like ```sql \object_store_profiling trace CREATE EXTERNAL TABLE overture_partitioned STORED AS PARQUET LOCATION 's3://overturemaps-us-west-2/release/2025-09-24.0/theme=addresses/'; ``` But Object Store Profiling Instrumented Object Store: instrument_mode: Trace, inner: AmazonS3(overturemaps-us-west-2) 2025-10-17T17:05:27.922724180+00:00 operation=List duration=0.132154s path=release/2025-09-24.0/theme=addresses 2025-10-17T17:05:28.054894440+00:00 operation=List duration=0.049048s path=release/2025-09-24.0/theme=addresses/type=address 2025-10-17T17:05:28.104233937+00:00 operation=Get duration=0.053522s size=8 range: bytes=1070778162-1070778169 path=release/2025-09-24.0/theme=addresses/type=address/part-00000-52872134-68de-44a6-822d-15fa29a0f606-c000.zstd.parquet 2025-10-17T17:05:28.106862343+00:00 operation=Get duration=0.108103s size=8 range: bytes=1017940335-1017940342 path=release/2025-09-24.0/theme=addresses/type=address/part-00003-52872134-68de-44a6-822d-15fa29a0f606-c000.zstd.parquet ``` Yes, not being able to easily evaluate a meaningful duration from this is a pretty big bummer honestly. I think time to first response is probably the ideal measurement to take here. I briefly looked into what it would take to make that happen within this instrumented store and I think it ends up being quite complex. I'm pretty sure we'd have to write a custom future to wrap the elements within the stream since the duration is only meaningful once elements in the stream start reporting `Poll::Ready`. Hopefully there's an easier way, because that sounds pretty painful. _Originally posted by @BlakeOrth in https://github.com/apache/datafusion/pull/18103#discussion_r2437229437_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
