birschick-bq commented on PR #2666:
URL: https://github.com/apache/arrow-adbc/pull/2666#issuecomment-2773990668
> @eric-wang-1990 - I tried this for the Apache Spark reader and had to
revert. #2273
>
> I'll test your changes to see if there is and issue with interleaved use
of the buffer using the the Thrift library.
As I suspected, I am able to reproduce the problem ...
```
Apache.Arrow.Adbc.Tests.Drivers.Apache.Spark.DriverTests.CanExecuteQuery(batchSizeFactor:
0.10000000000000001)
No source available
Duration: 44.2 sec
Message:
System.InvalidOperationException : Unexpectedly reached the end of the
stream before a full buffer was read.
Stack Trace:
ArrowStreamReaderImplementation.EnsureFullRead(Memory`1 buffer, Int32
bytesRead)
ArrowStreamReaderImplementation.ReadMessageAsync(CancellationToken
cancellationToken)
ArrowStreamReaderImplementation.ReadRecordBatchAsync(CancellationToken
cancellationToken)
ArrowStreamReaderImplementation.ReadNextRecordBatchAsync(CancellationToken
cancellationToken)
SparkDatabricksReader.TryReadFromCurrentReader(CancellationToken
cancellationToken) line 94
SparkDatabricksReader.ReadNextRecordBatchAsync(CancellationToken
cancellationToken) line 66
DriverTests.CanExecuteQuery(QueryResult queryResult, Int64
expectedNumberOfResults, String environmentName) line 50
DriverTests`2.CanExecuteQuery(Nullable`1 batchSizeFactor) line 555
RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature
sig, Boolean isConstructor)
MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1
copyOfArgs, BindingFlags invokeAttr)
Standard Output:
BatchSize: 5474. ExpectedResultCount: 54748
SELECT * FROM `hive_metastore`.`default`.`nyctaxigreen`
```
From PR [#2273](https://github.com/apache/arrow-adbc/pull/2273)
> The attempt to use “look-ahead” buffering in the HiveServer2Reader is not
supported by the Thrift library. The issue is that the Thrift library uses a
shared buffer on the Client/Protocol/Transport object -so interleaving Fetch
and ExecuteStatement will fail because the buffer will get closed unexpectedly.
I created a ticket with Apache Thrift
https://issues.apache.org/jira/browse/THRIFT-5830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
