CurtHagenlocher commented on code in PR #2691:
URL: https://github.com/apache/arrow-adbc/pull/2691#discussion_r2047325192
##########
csharp/src/Drivers/Databricks/DatabricksConnection.cs:
##########
@@ -29,10 +29,50 @@ namespace Apache.Arrow.Adbc.Drivers.Databricks
{
internal class DatabricksConnection : SparkHttpConnection
{
+ // CloudFetch configuration
+ private const long DefaultMaxBytesPerFile = 20 * 1024 * 1024; // 20MB
+
+ private bool _useCloudFetch = true;
+ private bool _canDecompressLz4 = true;
+ private long _maxBytesPerFile = DefaultMaxBytesPerFile;
+
public DatabricksConnection(IReadOnlyDictionary<string, string>
properties) : base(properties)
{
+ // Parse CloudFetch options from connection properties
+ if (Properties.TryGetValue(DatabricksParameters.UseCloudFetch, out
string? useCloudFetchStr) &&
+ bool.TryParse(useCloudFetchStr, out bool useCloudFetchValue))
+ {
+ _useCloudFetch = useCloudFetchValue;
Review Comment:
It's unfortunate that we're essentially validating these in two places: both
here and in `DatabricksStatement`.
We've also somehow arrived in a weird place where these values are both
connection properties (that can only be set before the connection is opened)
and statement properties that can be set any time after the statement was
created. But that's a preexisting condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
