Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
codope merged PR #19822: URL: https://github.com/apache/hudi/pull/19822 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-agent commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4079915758
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLegacyParquetReadPath.scala:
##
@@ -301,6 +306,39 @@ class TestLegacyParquetReadPath extends
HoodieSparkClientTestBase with ScalaAsse
}
}
+ @Test
+ def testBroadcastJoinHonorsPlanTimeBatchingDecision(): Unit = {
Review Comment:
🤖 Does this test actually fail without the fix on the default (Spark 3.5)
profile? Spark35/34's `getReturningBatch` already routes through
`supportsColumnar`, which checks `conf.wholeStageEnabled`, so the pre-fix code
should also return false here — only Spark 3.3's override (`supportBatch`,
which in 3.3 is just `isBatchReadSupportedForSchema`) has the gap. If so, the
regression would only be guarded on the spark3.3 CI profile.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.
##
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark3LegacyHoodieParquetFileFormat.scala:
##
@@ -171,7 +171,11 @@ abstract class
Spark3LegacyHoodieParquetFileFormat(shouldAppendPartitionValues:
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
// Whole stage codegen (PhysicalRDD) is able to deal with batches directly.
-val returningBatch = getReturningBatch(sparkSession, resultSchema)
+// Respect the plan-time OPTION_RETURNING_BATCH decision when present,
instead of recomputing it here.
+val returningBatch = enableVectorizedReader &&
Review Comment:
🤖 The Spark 4.0/4.1/4.2 legacy formats are standalone copies that still
recompute `returningBatch` from `supportsColumnar(sparkSession, resultSchema)`
and ignore the option. They're less exposed (their `supportsColumnar` includes
the whole-stage check), but `isTooManyFields` there is evaluated over
`partitionSchema ++ requiredSchema` rather than the plan's output schema, so
they can still disagree with the plan. Worth applying the same fix there?
⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
sumitsingh-in commented on code in PR #19822: URL: https://github.com/apache/hudi/pull/19822#discussion_r4081680903 ## hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40LegacyHoodieParquetFileFormat.scala: ## @@ -152,10 +152,11 @@ class Spark40LegacyHoodieParquetFileFormat(private val shouldAppendPartitionValu val datetimeRebaseModeInRead = parquetOptions.datetimeRebaseModeInRead val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead val timeZoneId = Option(sqlConf.sessionLocalTimeZone) -// Should always be set by FileSourceScanExec creating this. -// Check conf before checking option, to allow working around an issue by changing conf. +// Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here. val returningBatch = sparkSession.sessionState.conf.parquetVectorizedReaderEnabled && Review Comment: we are not changing existing check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5791199006 ## CI report: * f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771) * 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250) * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5795354854 ## CI report: * 0733587bcfa350068eef96fc513efd07bed39382 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5794470196 ## CI report: * 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255) * 0733587bcfa350068eef96fc513efd07bed39382 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5790549195 ## CI report: * f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771) * 87b0d2b9f47b2d7c3400b49db324157fb1716e69 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5792885982 ## CI report: * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) * 3a2e0e8ce77375641286cf121687c7a1ab51853d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5793880102 ## CI report: * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) * 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255) * 0733587bcfa350068eef96fc513efd07bed39382 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17259) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5792965876 ## CI report: * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) * 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5793056919 ## CI report: * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) * 3a2e0e8ce77375641286cf121687c7a1ab51853d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17255) * 0733587bcfa350068eef96fc513efd07bed39382 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5792797407 ## CI report: * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-agent commented on code in PR #19822: URL: https://github.com/apache/hudi/pull/19822#discussion_r4080876428 ## hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40LegacyHoodieParquetFileFormat.scala: ## @@ -152,10 +152,11 @@ class Spark40LegacyHoodieParquetFileFormat(private val shouldAppendPartitionValu val datetimeRebaseModeInRead = parquetOptions.datetimeRebaseModeInRead val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead val timeZoneId = Option(sqlConf.sessionLocalTimeZone) -// Should always be set by FileSourceScanExec creating this. -// Check conf before checking option, to allow working around an issue by changing conf. +// Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here. val returningBatch = sparkSession.sessionState.conf.parquetVectorizedReaderEnabled && Review Comment: 🤖 The spark3-common version gates this on `enableVectorizedReader &&`, but here it's only `parquetVectorizedReaderEnabled &&`. Previously the `supportsColumnar(resultSchema)` fallback implied `supportBatch(resultSchema)` (== `enableVectorizedReader`), so the guard was implicit; with the option path it isn't. If the plan says `true` but `enableVectorizedReader` is false (resultSchema = partitionSchema ++ requiredSchema is a superset of the plan's output schema), the reader takes the row branch while the plan expects columnar. Could you mirror the `enableVectorizedReader &&` guard here (and in 4.1/4.2)? ⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5791739560 ## CI report: * 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250) * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5791421401 ## CI report: * f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771) * 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250) * f2b51c7b7bca72e087b0fbc3b8f375e0fe3b8dff Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17252) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
sumitsingh-in commented on code in PR #19822: URL: https://github.com/apache/hudi/pull/19822#discussion_r4080080929 ## hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark3LegacyHoodieParquetFileFormat.scala: ## @@ -171,7 +171,11 @@ abstract class Spark3LegacyHoodieParquetFileFormat(shouldAppendPartitionValues: val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead val timeZoneId = Option(sqlConf.sessionLocalTimeZone) // Whole stage codegen (PhysicalRDD) is able to deal with batches directly. -val returningBatch = getReturningBatch(sparkSession, resultSchema) +// Respect the plan-time OPTION_RETURNING_BATCH decision when present, instead of recomputing it here. +val returningBatch = enableVectorizedReader && Review Comment: added fix in these module as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
sumitsingh-in commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r4080079534
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLegacyParquetReadPath.scala:
##
@@ -301,6 +306,39 @@ class TestLegacyParquetReadPath extends
HoodieSparkClientTestBase with ScalaAsse
}
}
+ @Test
+ def testBroadcastJoinHonorsPlanTimeBatchingDecision(): Unit = {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5790617747 ## CI report: * f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771) * 87b0d2b9f47b2d7c3400b49db324157fb1716e69 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17250) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
codecov-commenter commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5586395359 ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19822?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report :white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 78.30%. Comparing base ([`a2788ee`](https://app.codecov.io/gh/apache/hudi/commit/a2788eef922e2375432ead3fe50af7fe5f55faf7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f8c81ac`](https://app.codecov.io/gh/apache/hudi/commit/f8c81acdae41941880e2a3c266eed2093776ff5a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)). :warning: Report is 25 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #19822 +/- ## - Coverage 78.31% 78.30% -0.01% - Complexity3388533887 +2 Files 2541 2541 Lines141728 141859 +131 Branches 1718217182 + Hits 110988 111078 +90 - Misses2303223071 +39 - Partials 7708 7710 +2 ``` | [Components](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.58% <ø> (-0.07%)` | :arrow_down: | | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.21% <ø> (ø)` | | | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.65% <ø> (ø)` | | | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.22% <ø> (+<0.01%)` | :arrow_up: | | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.55% <ø> (ø)` | | | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | | | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.72% <ø> (ø)` | | | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.54% <ø> (ø)` | | | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.02% <ø> (+0.04%)` | :arrow_up: | | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | | | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | | | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19822/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | | | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19822/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| |
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5509847957 ## CI report: * f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5507989018 ## CI report: * 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5508302520 ## CI report: * 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770) * f8c81acdae41941880e2a3c266eed2093776ff5a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5508712223 ## CI report: * 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770) * f8c81acdae41941880e2a3c266eed2093776ff5a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16771) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-agent commented on code in PR #19822:
URL: https://github.com/apache/hudi/pull/19822#discussion_r3913108689
##
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark3LegacyHoodieParquetFileFormat.scala:
##
@@ -171,7 +171,11 @@ abstract class
Spark3LegacyHoodieParquetFileFormat(shouldAppendPartitionValues:
val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead
val timeZoneId = Option(sqlConf.sessionLocalTimeZone)
// Whole stage codegen (PhysicalRDD) is able to deal with batches directly.
-val returningBatch = getReturningBatch(sparkSession, resultSchema)
+// Respect the plan-time OPTION_RETURNING_BATCH decision when present,
instead of recomputing it here.
+val returningBatch = enableVectorizedReader &&
+ options.get(FileFormat.OPTION_RETURNING_BATCH)
+.map(_.equals("true"))
Review Comment:
🤖 nit: `_.equals("true")` is Java-style — in Scala `_ == "true"` (or
`_.toBoolean`) is the idiomatic equivalent here.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5508052116 ## CI report: * 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16770) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
hudi-bot commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5507915340 ## CI report: * 0dbd1e9d2f8b0e4860a3633d2093a5cf9d07e79d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] fix(spark): honor plan-time OPTION_RETURNING_BATCH in legacy parquet … [hudi]
sumitsingh-in commented on PR #19822: URL: https://github.com/apache/hudi/pull/19822#issuecomment-5507861795 @codope PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
