BlakeOrth opened a new pull request, #18004:
URL: https://github.com/apache/datafusion/pull/18004
## Which issue does this PR close?
This does not fully close, but is an incremental building block component
for:
- https://github.com/apache/datafusion/issues/17207
The full context of how this code is likely to progress can be seen in the
POC for this effort:
- https://github.com/apache/datafusion/pull/17266
## Rationale for this change
This change allows a user of `datafusion-cli` to enable or disable object
store profiling, either from the command line or from an internal command while
in a cli session.
Note that since no actual instrumentation is implemented for the
`InstrumentedObjectStore`s the output for the user at this point is generally
not interesting (yet). This PR implements a tapas-sized functional unit of work
towards #17207
## What changes are included in this PR?
- Adds a CLI option and command to datafusion-cli to enable or disabled
object store profiling
- Integrates the command with the instrumented object stores to allow the
user input to change the mode of the instrumented stores
- Adds tests to exercise the expected behavior of the commands
- Adds user docs for the commands/CLI options
- Updates visibility of `InstrumentedObjectStore` now that it needs to be
interacted with outside of its module
## Are these changes tested?
Yes. New Unit tests have been added for all meaningful changes.
Example functional output from `datafusion-cli`
```console
$ ../../target/debug/datafusion-cli --object-store-profiling enabled
```
```sql
DataFusion CLI v50.1.0
> CREATE EXTERNAL TABLE hits
STORED AS PARQUET
LOCATION
'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
0 row(s) fetched.
Elapsed 0.170 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Enabled, inner: HttpStore
> \object_store_profiling disabled
ObjectStore Profile mode set to Disabled
> CREATE EXTERNAL TABLE hits2
STORED AS PARQUET
LOCATION
'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_2.parquet';
0 row(s) fetched.
Elapsed 4.615 seconds.
> \object_store_profiling enabled
ObjectStore Profile mode set to Enabled
> CREATE EXTERNAL TABLE hits3
STORED AS PARQUET
LOCATION
'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_3.parquet';
0 row(s) fetched.
Elapsed 3.349 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Enabled, inner: HttpStore
Instrumented Object Store: instrument_mode: Enabled, inner: HttpStore
Instrumented Object Store: instrument_mode: Enabled, inner: HttpStore
```
## Are there any user-facing changes?
Yes. Unfortunately this work seems to necessitate adding a new public item
to the `datafusion-cli::print_options::PrintOptions` type which is exposed as a
public type. Other obvious user-facing changes are the addition of the new
command and CLI option to `datafusion-cli`, which have been documented in the
CLI user guide accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
