eric-wang-1990 opened a new pull request, #3660:
URL: https://github.com/apache/arrow-adbc/pull/3660
## Summary
Adds comprehensive E2E benchmark for Databricks CloudFetch to measure
real-world performance with actual cluster and configurable queries.
## Changes
- **CloudFetchRealE2EBenchmark**: Real E2E benchmark against actual
Databricks cluster
- Configurable via JSON file (DATABRICKS_TEST_CONFIG_FILE environment
variable)
- Power BI consumption simulation with batch-size proportional delays (5ms
per 10K rows)
- Peak memory tracking using Process.WorkingSet64
- Custom peak memory column in results table with console output reference
- **CloudFetchBenchmarkRunner**: Standalone runner for CloudFetch benchmarks
- Simplified to only run real E2E benchmark
- Optimized iteration counts (1 warmup + 3 actual) for faster execution
- Hides confusing Error/StdDev columns from summary table
- **README.md**: Documentation for running and understanding the benchmarks
## Configuration
Benchmark requires `DATABRICKS_TEST_CONFIG_FILE` environment variable
pointing to JSON config:
```json
{
"uri":
"https://your-workspace.cloud.databricks.com/sql/1.0/warehouses/xxx";,
"token": "dapi...",
"query": "select * from main.tpcds_sf1_delta.catalog_sales"
}
```
## Run Command
```bash
export DATABRICKS_TEST_CONFIG_FILE=/path/to/config.json
cd csharp
dotnet run -c Release --project Benchmarks/Benchmarks.csproj --framework
net8.0 CloudFetchBenchmarkRunner -- --filter "*"
```
## Test Plan
- [x] Built successfully
- [x] Verified benchmark runs with real Databricks cluster
- [x] Confirmed peak memory tracking works
- [x] Validated Power BI simulation delays are proportional to batch size
- [x] Checked results table formatting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
