peter-toth commented on code in PR #59324:
URL: https://github.com/apache/spark/pull/59324#discussion_r4283438528
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala:
##########
@@ -264,39 +217,26 @@ object ParquetStorageFilter {
}
/**
- * Rethrows an error the reader met while applying a storage filter, if it
must not fall back on
- * it by giving the filter up. `fromRead` says the error came from decoding
the key pages rather
- * than from evaluating the filter. The checks run in the executor's own
order:
- * - a fatal error goes out as it is, kill or no kill.
- * - a pending task kill goes out wrapped when the error is one the
executor's kill case takes,
- * see [[throwIfKilled]]. The error may be the kill's interrupt surfacing
through a UDF that
- * wrapped it. `PythonRunner` reads a kill off the task context in the
same way.
- * - a fatal error deeper in the cause chain, found by
`Executor.isFatalError` as deep as the
- * executor would look had a plain plan raised `e`, goes out as itself,
so the executor finds
- * it at the top whatever wraps it on the way.
+ * Rethrows an error the reader met while applying a storage filter, unless
it may give the filter
+ * up on it. `fromRead` says the error came from decoding the key pages. In
the executor's order:
+ * - a fatal error goes out as it is;
+ * - a pending task kill goes out wrapped when the executor would report
the error as a kill,
+ * see [[throwIfKilled]], since the error may be the kill's interrupt
surfacing through a UDF;
Review Comment:
Done in
[0c77f7ddd22](https://github.com/apache/spark/pull/59324/commits/0c77f7ddd22e4f70db4072d4db39d832ed53dbe7):
"..., and as it is otherwise, see [[throwIfKilled]], ...".
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetBlockAccounting.java:
##########
@@ -38,14 +38,8 @@
/**
* What a partial read of a Parquet file transfers, and whether one is
possible at all, answered
- * from the footer the {@link ParquetFileReader} already holds. Every method
here is metadata
- * arithmetic and causes no IO of its own, which is what lets a reader report
these numbers per row
- * group. For a strict subset of a block that holds only once the block's
column index store is
- * built, see {@link #compressedBytesForRowRanges}. A walk before that would
read the offset indexes
- * itself and cache a store over the reader's current columns.
- *
- * <p>One instance per reader, which reads one split of a file, because the
warning below is
- * reported once per instance.
+ * from the metadata the {@link ParquetFileReader} already holds, with no IO
of its own. One
Review Comment:
Done in
[0c77f7ddd22](https://github.com/apache/spark/pull/59324/commits/0c77f7ddd22e4f70db4072d4db39d832ed53dbe7):
"with no IO of its own once the block's store is built, see {@link
#compressedBytesForRowRanges}".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
