This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6811-6de37efa301b4635e3f5189d8872b7b450dca982 in repository https://gitbox.apache.org/repos/asf/texera.git
commit 72522b0637ff496ae1041506b5142295589befb9 Author: roshiiiiz <[email protected]> AuthorDate: Thu Aug 13 06:11:26 2026 +0000 feat(operator): provide user-friendly error message when binary file scan hits memory limit (#6811) <!-- Thanks for sending a pull request (PR)! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: [Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) 2. Ensure you have added or run the appropriate tests for your PR 3. If the PR is work in progress, mark it a draft on GitHub. 4. Please write your PR title to summarize what this PR proposes, we are following Conventional Commits style for PR titles as well. 5. Be sure to keep the PR description updated to reflect all changes. --> ### What changes were proposed in this PR? <!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes. Here are some tips for you: 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. 3. If it is a refactoring, clarify what has been changed. 3. It would be helpful to include a before-and-after comparison using screenshots or GIFs. 4. Please consider writing useful notes for better and faster reviews. --> This PR improves the user experience for the File Scan operator's in-memory read path by gracefully catching natural Java memory limits and surfacing a helpful UI error message, rather than allowing the worker JVM to crash. **Why is it needed?** Previously, when users mistakenly attempted to read massive files using the `binary` attribute type (instead of the streaming `large binary` type), `ByteArrayOutputStream` would attempt to allocate the entire file into memory. If the file exceeded the JVM's available heap space or the maximum Java array size, it triggered an unhandled `OutOfMemoryError` or `IllegalArgumentException`, causing the `computing-unit-master` to lock up or crash entirely without reporting a user-friendly error to the frontend UI. **What was changed:** - Wrapped the stream reader in `FileScanUtils.safeToByteArray` with a `try-catch` block. - Intercepts natural `OutOfMemoryError` and `IllegalArgumentException` thrown by the JVM or `ByteArrayOutputStream`. - Throws a clean, user-friendly `RuntimeException` directly to the frontend directing the user to use the `large binary` attribute type instead for massive files. *(Note: Based on maintainer feedback, an initial hardcoded size threshold approach was dropped in favor of this cleaner architectural approach that relies on natural JVM limits).* ### Any related issues, documentation, discussions? <!-- Please use this section to link other resources if not mentioned already. 1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves #1234` or `Closes #1234`. If it is only related, simply mention the issue number. 2. If there is design documentation, please add the link. 3. If there is a discussion in the mailing list, please add the link. --> Closes #3271 ### How was this PR tested? <!-- If tests were added, say they were added here. Or simply mention that if the PR is tested with existing test cases. Make sure to include/update test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> **Manual Verification:** 1. Uploaded an 8.9 GB CSV test file to the workspace. 2. Created a workflow with the `File Scan` operator configured to use the `binary` attribute type (which intentionally attempts to load the entire file into memory). 3. Ran the workflow. **Result:** The workflow caught the JVM's `OutOfMemoryError` when attempting the massive allocation, safely aborted the thread, and successfully threw the custom user-friendly error message in the UI without crashing the server. **Automated Tests:** - Updated the mock in `FileScanUtilsSpec.scala` to simulate a natural `OutOfMemoryError` being thrown during stream reading to prove the `catch` block reliably intercepts it and translates it to the user-friendly exception. ### Was this PR authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this PR, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> Generated-by: Antigravity (DeepMind) --------- Co-authored-by: probe <probe@x> Co-authored-by: Yicong Huang <[email protected]> Co-authored-by: Xinyuan Lin <[email protected]> --- .../operator/source/scan/file/FileScanUtils.scala | 34 ++++++++++++++++++++-- .../source/scan/file/FileScanUtilsSpec.scala | 17 +++++++++++ 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtils.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtils.scala index 2c52fa9e8e..b6cbdc1b93 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtils.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtils.scala @@ -21,7 +21,7 @@ package org.apache.texera.amber.operator.source.scan.file import org.apache.commons.compress.archivers.ArchiveStreamFactory import org.apache.commons.compress.archivers.zip.ZipArchiveInputStream -import org.apache.commons.io.IOUtils.toByteArray + import org.apache.texera.amber.core.storage.DocumentFactory import org.apache.texera.amber.core.tuple.AttributeTypeUtils.parseField import org.apache.texera.amber.core.tuple.LargeBinary @@ -39,6 +39,34 @@ import scala.collection.mutable import scala.jdk.CollectionConverters.IteratorHasAsScala private[file] object FileScanUtils { + + private[file] def safeToByteArray( + entry: InputStream, + attributeType: FileAttributeType + ): Array[Byte] = { + try { + val out = new ByteArrayOutputStream() + val buffer = new Array[Byte](8192) + var bytesRead = entry.read(buffer) + + while (bytesRead != -1) { + out.write(buffer, 0, bytesRead) + bytesRead = entry.read(buffer) + } + out.toByteArray + } catch { + case _: OutOfMemoryError | _: IllegalArgumentException => + val largeBinaryHint = attributeType match { + case FileAttributeType.BINARY => "Please use 'large binary' attribute type instead." + case FileAttributeType.SINGLE_STRING => + "Please split the file or use a chunked reading method." + case _ => "File is too large to fit in memory." + } + throw new RuntimeException( + s"File exceeds maximum safe memory size for '${attributeType.getName}' type. $largeBinaryHint" + ) + } + } def createTuplesFromFile( fileName: String, displayFileName: String, @@ -90,7 +118,7 @@ private[file] object FileScanUtils { } fields += (attributeType match { case FileAttributeType.SINGLE_STRING => - new String(toByteArray(entry), fileEncoding.getCharset) + new String(safeToByteArray(entry, attributeType), fileEncoding.getCharset) case FileAttributeType.LARGE_BINARY => val largeBinary = new LargeBinary() val out = new LargeBinaryOutputStream(largeBinary) @@ -105,7 +133,7 @@ private[file] object FileScanUtils { out.close() } largeBinary - case _ => parseField(toByteArray(entry), attributeType.getType) + case _ => parseField(safeToByteArray(entry, attributeType), attributeType.getType) }) TupleLike(fields.toSeq: _*) } diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala index 6e170aa60c..2de0eacfac 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/scan/file/FileScanUtilsSpec.scala @@ -240,4 +240,21 @@ class FileScanUtilsSpec extends AnyFlatSpec with BeforeAndAfterAll { .toSeq assert(contents(tuples) == Seq("l1\nl2\nl3\nl4\nl5")) } + + it should "throw RuntimeException when binary file exceeds natural memory limit" in { + val mockLargeInputStream = new java.io.InputStream { + override def read(): Int = { + throw new OutOfMemoryError("Requested array size exceeds VM limit") + } + override def read(b: Array[Byte], off: Int, len: Int): Int = { + throw new OutOfMemoryError("Requested array size exceeds VM limit") + } + } + + val exception = intercept[RuntimeException] { + FileScanUtils.safeToByteArray(mockLargeInputStream, FileAttributeType.BINARY) + } + assert(exception.getMessage.contains("exceeds maximum safe memory size")) + assert(exception.getMessage.contains("Please use 'large binary'")) + } }
