BlakeOrth commented on PR #17266:
URL: https://github.com/apache/datafusion/pull/17266#issuecomment-3234598245
@alamb I've made some changes that I think get the functional side of the
code where it needs to be and pushed those for review. This is still lacking
tests and docs, but I thought it would be good to more or less have the
architecture and functional code settled before wrapping everything up and
marking it as ready for review.
Here's a little demo of the output from the current code:
```console
$ ./datafusion-cli --object-store-profiling summary
```
```sql
DataFusion CLI v49.0.1
> CREATE EXTERNAL TABLE nyc_taxi_rides
STORED AS PARQUET LOCATION
's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/';
0 row(s) fetched.
Elapsed 2.573 seconds.
Object Store Profiling
List Summary:
count: 1
Get Summary:
count: 288
duration min: 0.059328s
duration max: 0.714468s
duration avg: 0.128311s
size min: 8 B
size max: 44247 B
size avg: 18870 B
size sum: 5434702 B
List Summary:
count: 1
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> select count(*) from
's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/';
+------------+
| count(*) |
+------------+
| 1310903963 |
+------------+
1 row(s) fetched.
Elapsed 0.577 seconds.
Object Store Profiling
2025-08-28T18:49:22.066425183+00:00 operation=List
path=nyc_taxi_rides/data/tripdata_parquet
2025-08-28T18:49:22.303706449+00:00 operation=List
path=nyc_taxi_rides/data/tripdata_parquet
2025-08-28T18:49:22.419878784+00:00 operation=List
path=nyc_taxi_rides/data/tripdata_parquet
2025-08-28T18:49:22.493157995+00:00 operation=List
path=nyc_taxi_rides/data/tripdata_parquet
List Summary:
count: 4
> \object_store_profiling disabled
ObjectStore Profile mode set to Disabled
> select count(*) from
's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/' where
vendor_id='CMT';
+-----------+
| count(*) |
+-----------+
| 505603754 |
+-----------+
1 row(s) fetched.
Elapsed 45.531 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
