shfshihuafeng commented on code in PR #2889:
URL: https://github.com/apache/drill/pull/2889#discussion_r1529743261
##########
exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java:
##########
@@ -155,12 +157,18 @@ public void readFromStreamWithContainer(VectorContainer
myContainer, InputStream
for (SerializedField metaData : fieldList) {
final int dataLength = metaData.getBufferLength();
final MaterializedField field = MaterializedField.create(metaData);
- final DrillBuf buf = allocator.buffer(dataLength);
- final ValueVector vector;
+ DrillBuf buf = null;
+ ValueVector vector = null;
try {
+ buf = allocator.buffer(dataLength);
buf.writeBytes(input, dataLength);
vector = TypeHelper.getNewVector(field, allocator);
vector.load(metaData, buf);
+ } catch (OutOfMemoryException oom) {
+ for (ValueVector valueVector : vectorList) {
+ valueVector.clear();
+ }
+ throw UserException.memoryError(oom).message("Allocator memory
failed").build(logger);
Review Comment:
when we prepare to allocator memory using "allocator.buffer(dataLength)"
for hashjoinPop allocator, if actual memory > maxAllocation(The parameter is
calculated by call computeOperatorMemory) ,then it throw exception, like
following my test。
user can adjust directMemory parameters (DRILL_MAX_DIRECT_MEMORY) or
reduce concurrency based on actual conditions.
**throw exception code**
```
public DrillBuf buffer(final int initialRequestSize, BufferManager manager) {
assertOpen();
AllocationOutcome outcome = allocateBytes(actualRequestSize);
if (!outcome.isOk()) {
throw new OutOfMemoryException(createErrorMsg(this, actualRequestSize,
initialRequestSize));
}
```
**my test scenario**
```
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
allocate buffer of size 16384 (rounded from 14359) due to memory limit
(41943040). Current allocation: 22583616
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241)
at
org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216)
at
org.apache.drill.exec.cache.VectorAccessibleSerializable.readFromStreamWithContainer(VectorAccessibleSerializable.java:172)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
