Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
codope merged PR #11415: URL: https://github.com/apache/hudi/pull/11415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159831263 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * 0eab8af324efc3a31b898abfd8ed1c5f94557856 UNKNOWN * 58b0fbd74cb50adca712bea81cfec195e22f1a42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24349) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159774832 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * 1f0f1a8fa3f7476d4c0e206cd2fb0802eb160acd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24348) * 0eab8af324efc3a31b898abfd8ed1c5f94557856 UNKNOWN * 58b0fbd74cb50adca712bea81cfec195e22f1a42 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24349) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159727319 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * 8234c0866f0c386c9496e4d2b8499456dd01ed4e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24346) * 1f0f1a8fa3f7476d4c0e206cd2fb0802eb160acd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24348) * 0eab8af324efc3a31b898abfd8ed1c5f94557856 UNKNOWN * 58b0fbd74cb50adca712bea81cfec195e22f1a42 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159721644 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * 8234c0866f0c386c9496e4d2b8499456dd01ed4e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24346) * 1f0f1a8fa3f7476d4c0e206cd2fb0802eb160acd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24348) * 0eab8af324efc3a31b898abfd8ed1c5f94557856 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1634113767
##
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##
@@ -165,21 +166,24 @@ private static HoodieDataBlock getDataBlock(
HoodieLogBlock.HoodieLogBlockType dataBlockType,
List records,
Map header,
- StoragePath logFilePath
+ StoragePath logFilePath,
+ boolean writePositions
) {
return createDataBlock(
dataBlockType,
-records.stream().map(HoodieAvroIndexedRecord::new)
+records.stream().map(r -> new HoodieAvroIndexedRecord(r, new
HoodieRecordLocation("", "", Long.parseLong((String) r.get(1)) - 1)))
Review Comment:
ok, made it better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159685811 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * 8234c0866f0c386c9496e4d2b8499456dd01ed4e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24346) * 1f0f1a8fa3f7476d4c0e206cd2fb0802eb160acd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159662174 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * 8234c0866f0c386c9496e4d2b8499456dd01ed4e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24346) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159621967 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * aa8b5e13252f0c7b18b2e3a50b3545a357929492 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24338) * 8234c0866f0c386c9496e4d2b8499456dd01ed4e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24346) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159613468 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * aa8b5e13252f0c7b18b2e3a50b3545a357929492 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24338) * 8234c0866f0c386c9496e4d2b8499456dd01ed4e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159146212 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * aa8b5e13252f0c7b18b2e3a50b3545a357929492 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24338) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
codope commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633697715
##
hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithLatestMerger.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*
Review Comment:
remove redundant licenses here and below.
##
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##
@@ -163,4 +166,18 @@ default String[]
getMandatoryFieldsForMerging(HoodieTableConfig cfg) {
* The kind of merging strategy this recordMerger belongs to. An UUID
represents merging strategy.
*/
String getMergingStrategy();
+
+ /**
+ * The record merge mode that corresponds to this record merger
+ */
+ default RecordMergeMode getRecordMergeMode() {
+switch (getMergingStrategy()) {
+ case DEFAULT_MERGER_STRATEGY_UUID:
+return RecordMergeMode.EVENT_TIME_ORDERING;
+ case OVERWRITE_MERGER_STRATEGY_UUID:
+return RecordMergeMode.OVERWRITE_WITH_LATEST;
+ default:
+return RecordMergeMode.CUSTOM;
Review Comment:
Can we reuse `inferRecordMergeMode` in `HoodieTableMetaClient` in some way?
Even if we cannot, let's ensure that we keep the logic conistent, e.g. strategy
could be `DEFAULT_MERGER_STRATEGY_UUID` while payload class could be
`OverwriteWithLatestAvroPayload`, in that case merge mode should be
`OVERWRITE_WITH_LATEST`.
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieCommonConfig, HoodieReaderConfig,
HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieRecordMerger,
OverwriteWithLatestAvroPayload, OverwriteWithLatestMerger}
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
Review Comment:
yes please go ahead and remove it in that case
##
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##
@@ -219,21 +223,22 @@ public static HoodieDeleteBlock getDeleteBlock(
List records,
Map header,
Schema schema,
- Properties props
+ Properties props,
+ boolean writePositions
) {
List hoodieRecords = records.stream()
.map(r -> {
String rowKey = (String)
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159075347 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * fe02d41add679fd08c33af62d286625a31316a58 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24337) * aa8b5e13252f0c7b18b2e3a50b3545a357929492 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24338) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633693656
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieCommonConfig, HoodieReaderConfig,
HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieRecordMerger,
OverwriteWithLatestAvroPayload, OverwriteWithLatestMerger}
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
Review Comment:
So I am wondering if I should just delete this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633693375
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieCommonConfig, HoodieReaderConfig,
HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieRecordMerger,
OverwriteWithLatestAvroPayload, OverwriteWithLatestMerger}
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
Review Comment:
The tests above cover fallback logic and they don't require spark so they
are much faster for these small test datasets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2159058949 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * fe02d41add679fd08c33af62d286625a31316a58 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24337) * aa8b5e13252f0c7b18b2e3a50b3545a357929492 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
codope commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633687788
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieCommonConfig, HoodieReaderConfig,
HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieRecordMerger,
OverwriteWithLatestAvroPayload, OverwriteWithLatestMerger}
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
Review Comment:
Do all the above tests also cover fallback logic? Or do you mean to move the
tests from this class to the respective merger tests? I am ok with doing that
in a separate PR. It's not a blocker for this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633660129
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieCommonConfig, HoodieReaderConfig,
HoodieStorageConfig, RecordMergeMode}
+import org.apache.hudi.common.model.{HoodieRecordMerger,
OverwriteWithLatestAvroPayload, OverwriteWithLatestMerger}
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
Review Comment:
Do we want to remove this class? IDK if it's worth the extra CI time. The
testing should be covered by TestCustomMerger.java, TestEventTimeMerging.java,
and TestOverwriteWithLatestMerger.java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2158949694 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * fe02d41add679fd08c33af62d286625a31316a58 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24337) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
codope commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633621696
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig}
+import org.apache.hudi.common.model.HoodieRecordMerger
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
+ override def getSparkSessionExtensionsInjector:
util.Option[Consumer[SparkSessionExtensions]] =
+toJavaOption(
+ Some(
+JFunction.toJavaConsumer((receiver: SparkSessionExtensions) => new
HoodieSparkSessionExtension().apply(receiver)))
+)
+
+ @BeforeEach override def setUp(): Unit = {
+initPath()
+initSparkContexts()
+sparkSession.conf.set(SQLConf.PARQUET_RECORD_FILTER_ENABLED.key, "true")
+initTestDataGenerator()
+initHoodieStorage()
+ }
+
+ @AfterEach override def tearDown(): Unit = {
+cleanupSparkContexts()
+cleanupTestDataGenerator()
+cleanupFileSystem()
+FileSystem.closeAll()
+System.gc()
Review Comment:
yeah, i don't know why it was added. But, then let's do it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2158934957 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 35906d3ac7564746530e9a4866a35f99bbeb202e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24334) * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN * fe02d41add679fd08c33af62d286625a31316a58 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2158861404 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 35906d3ac7564746530e9a4866a35f99bbeb202e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24334) * 8191d21c8643c1e630abbc73fb9057fcf0b09795 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633551290
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig}
+import org.apache.hudi.common.model.HoodieRecordMerger
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
+ override def getSparkSessionExtensionsInjector:
util.Option[Consumer[SparkSessionExtensions]] =
+toJavaOption(
+ Some(
+JFunction.toJavaConsumer((receiver: SparkSessionExtensions) => new
HoodieSparkSessionExtension().apply(receiver)))
+)
+
+ @BeforeEach override def setUp(): Unit = {
+initPath()
+initSparkContexts()
+sparkSession.conf.set(SQLConf.PARQUET_RECORD_FILTER_ENABLED.key, "true")
+initTestDataGenerator()
+initHoodieStorage()
+ }
+
+ @AfterEach override def tearDown(): Unit = {
+cleanupSparkContexts()
+cleanupTestDataGenerator()
+cleanupFileSystem()
+FileSystem.closeAll()
+System.gc()
+ }
+
+ @ParameterizedTest
+ @MethodSource(Array("testArgs"))
+ def testPositionFallback(updateWithRecordPositions: String,
deleteWithRecordPositions: String, secondUpdateWithPositions: String): Unit = {
+val columns = Seq("ts", "key", "name", "_hoodie_is_deleted")
+val data = Seq(
+ (10, "1", "A", false),
+ (10, "2", "B", false),
+ (10, "3", "C", false),
+ (10, "4", "D", false),
+ (10, "5", "E", false))
+
+val inserts = sparkSession.createDataFrame(data).toDF(columns: _*)
+inserts.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option("hoodie.table.name", "test_table").
+ option(TABLE_TYPE.key(), "MERGE_ON_READ").
+ option(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), "parquet").
+ option(DataSourceWriteOptions.RECORD_MERGER_STRATEGY.key(),
HoodieRecordMerger.DEFAULT_MERGER_STRATEGY_UUID).
+ option(DataSourceWriteOptions.RECORD_MERGER_IMPLS.key(),
"org.apache.hudi.HoodieSparkRecordMerger").
+ option(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), "true").
+ option(HoodieWriteConfig.WRITE_RECORD_POSITIONS.key(), "true").
+ mode(Overwrite).
+ save(basePath)
+
+val updateData = Seq((11, "1", "A_1", false), (9, "2", "B_1", false))
+
+val updates = sparkSession.createDataFrame(updateData).toDF(columns: _*)
+
+updates.write.format("hudi").
+ option(RECORDKEY_FIELD.key(), "key").
+ option(PRECOMBINE_FIELD.key(), "ts").
+ option("hoodie.table.name", "test_table").
+ option(TABLE_TYPE.key(), "MERGE_ON_READ").
+ option(OPERATION.key(), "upsert").
+ option(HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key(), "parquet").
+ option(DataSourceWriteOptions.RECORD_MERGER_STRATEGY.key(),
HoodieRecordMerger.DEFAULT_MERGER_STRATEGY_UUID).
+ option(DataSourceWriteOptions.RECORD_MERGER_IMPLS.key(),
"org.apache.hudi.HoodieSparkRecordMerger").
+ option(HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key(), "true").
+
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633548011
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig}
+import org.apache.hudi.common.model.HoodieRecordMerger
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
+ override def getSparkSessionExtensionsInjector:
util.Option[Consumer[SparkSessionExtensions]] =
+toJavaOption(
+ Some(
+JFunction.toJavaConsumer((receiver: SparkSessionExtensions) => new
HoodieSparkSessionExtension().apply(receiver)))
+)
+
+ @BeforeEach override def setUp(): Unit = {
+initPath()
+initSparkContexts()
+sparkSession.conf.set(SQLConf.PARQUET_RECORD_FILTER_ENABLED.key, "true")
+initTestDataGenerator()
+initHoodieStorage()
+ }
+
+ @AfterEach override def tearDown(): Unit = {
+cleanupSparkContexts()
+cleanupTestDataGenerator()
+cleanupFileSystem()
+FileSystem.closeAll()
+System.gc()
Review Comment:
I copied this from
[TestCOWDataSource.scala](https://github.com/apache/hudi/blob/e63722d24b5f42d5411f02c0b0872310111887b3/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala#L101).
Should we change this there as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1633546119
##
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedFileGroupRecordBuffer.java:
##
@@ -174,20 +189,97 @@ public boolean containsLogRecord(String recordKey) {
}
@Override
- protected boolean doHasNext() throws IOException {
-ValidationUtils.checkState(baseFileIterator != null, "Base file iterator
has not been set yet");
-
-// Handle merging.
-while (baseFileIterator.hasNext()) {
- T baseRecord = baseFileIterator.next();
- nextRecordPosition = readerContext.extractRecordPosition(baseRecord,
readerSchema, ROW_INDEX_TEMPORARY_COLUMN_NAME, nextRecordPosition);
- Pair, Map> logRecordInfo =
records.remove(nextRecordPosition++);
- if (hasNextBaseRecord(baseRecord, logRecordInfo)) {
-return true;
+ protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
+if (!readerContext.getShouldMergeUseRecordPosition()) {
+ return doHasNextFallbackBaseRecord(baseRecord);
+}
+
+nextRecordPosition = readerContext.extractRecordPosition(baseRecord,
readerSchema,
+ROW_INDEX_COLUMN_NAME, nextRecordPosition);
+Pair, Map> logRecordInfo =
records.remove(nextRecordPosition++);
+
+Map metadata = readerContext.generateMetadataForRecord(
+baseRecord, readerSchema);
+
+Option resultRecord = logRecordInfo != null
+? merge(Option.of(baseRecord), metadata, logRecordInfo.getLeft(),
logRecordInfo.getRight())
+: merge(Option.empty(), Collections.emptyMap(), Option.of(baseRecord),
metadata);
+if (resultRecord.isPresent()) {
+ nextRecord = readerContext.seal(resultRecord.get());
+ return true;
+}
+return false;
+ }
+
+ private boolean doHasNextFallbackBaseRecord(T baseRecord) throws IOException
{
+if (needToDoHybridStrategy) {
+ //see if there is a delete block with record positions
+ nextRecordPosition = readerContext.extractRecordPosition(baseRecord,
readerSchema,
+ ROW_INDEX_TEMPORARY_COLUMN_NAME, nextRecordPosition);
+ Pair, Map> logRecordInfo =
records.remove(nextRecordPosition++);
Review Comment:
My understanding is that FG reader is single threaded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2158674880 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 35906d3ac7564746530e9a4866a35f99bbeb202e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24334) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2158578265 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 22d1bdc6320ddbd1232bb7d9edaf8162f33e2081 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24301) * 35906d3ac7564746530e9a4866a35f99bbeb202e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24334) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2158557264 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 22d1bdc6320ddbd1232bb7d9edaf8162f33e2081 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24301) * 35906d3ac7564746530e9a4866a35f99bbeb202e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2156173131 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 22d1bdc6320ddbd1232bb7d9edaf8162f33e2081 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24301) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2156153405 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 795b0473b4abca7626de895e81f6750863fa67d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24293) * 22d1bdc6320ddbd1232bb7d9edaf8162f33e2081 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24301) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2156151877 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 795b0473b4abca7626de895e81f6750863fa67d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24293) * 22d1bdc6320ddbd1232bb7d9edaf8162f33e2081 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1632094178
##
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedFileGroupRecordBuffer.java:
##
@@ -123,46 +142,42 @@ public void processDataBlock(HoodieDataBlock dataBlock,
Option keySpecO
}
}
- @Override
- public void processNextDataRecord(T record, Map metadata,
Serializable recordPosition) throws IOException {
-Pair, Map> existingRecordMetadataPair =
records.get(recordPosition);
-Option>> mergedRecordAndMetadata =
-doProcessNextDataRecord(record, metadata, existingRecordMetadataPair);
-if (mergedRecordAndMetadata.isPresent()) {
- records.put(recordPosition, Pair.of(
-
Option.ofNullable(readerContext.seal(mergedRecordAndMetadata.get().getLeft())),
- mergedRecordAndMetadata.get().getRight()));
+ private void fallbackToKeyBasedBuffer() {
+readerContext.setShouldMergeUseRecordPosition(false);
+//need to make a copy of the keys to avoid concurrent modification
exception
+ArrayList positions = new ArrayList<>(records.keySet());
Review Comment:
No, those are positions. The map is recordpositon->record. After we fallback
it becomes recordkey->record
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex commented on code in PR #11415: URL: https://github.com/apache/hudi/pull/11415#discussion_r1632093807 ## hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java: ## @@ -319,60 +311,6 @@ protected Option merge(Option older, Map olderInfoMap, return Option.empty(); } - /** Review Comment: Moved these out of the base record buffer. extractRecordPositions is specific to position based buffer and shouldskip is only used there as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2155855404 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 795b0473b4abca7626de895e81f6750863fa67d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24293) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2155853054 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * bfea0d3a2dd9e6ba2d96c1d7d20a07e085883da6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24278) * 795b0473b4abca7626de895e81f6750863fa67d3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
codope commented on code in PR #11415:
URL: https://github.com/apache/hudi/pull/11415#discussion_r1631913348
##
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestPositionBasedMergingFallback.scala:
##
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hadoop.fs.FileSystem
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions.{OPERATION, PRECOMBINE_FIELD,
RECORDKEY_FIELD, TABLE_TYPE}
+import org.apache.hudi.HoodieConversionUtils.toJavaOption
+import org.apache.hudi.common.config.{HoodieReaderConfig, HoodieStorageConfig}
+import org.apache.hudi.common.model.HoodieRecordMerger
+import org.apache.hudi.common.util
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.util.JFunction
+import org.apache.spark.sql.SaveMode.{Append, Overwrite}
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.internal.SQLConf
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.{AfterEach, BeforeEach}
+import org.junit.jupiter.params.ParameterizedTest
+import org.junit.jupiter.params.provider.{Arguments, MethodSource}
+
+import java.util.function.Consumer
+
+class TestPositionBasedMergingFallback extends HoodieSparkClientTestBase {
+ override def getSparkSessionExtensionsInjector:
util.Option[Consumer[SparkSessionExtensions]] =
+toJavaOption(
+ Some(
+JFunction.toJavaConsumer((receiver: SparkSessionExtensions) => new
HoodieSparkSessionExtension().apply(receiver)))
+)
+
+ @BeforeEach override def setUp(): Unit = {
+initPath()
+initSparkContexts()
+sparkSession.conf.set(SQLConf.PARQUET_RECORD_FILTER_ENABLED.key, "true")
+initTestDataGenerator()
+initHoodieStorage()
+ }
+
+ @AfterEach override def tearDown(): Unit = {
+cleanupSparkContexts()
+cleanupTestDataGenerator()
+cleanupFileSystem()
+FileSystem.closeAll()
+System.gc()
Review Comment:
let's avoid System.gc()
##
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedFileGroupRecordBuffer.java:
##
@@ -174,20 +189,97 @@ public boolean containsLogRecord(String recordKey) {
}
@Override
- protected boolean doHasNext() throws IOException {
-ValidationUtils.checkState(baseFileIterator != null, "Base file iterator
has not been set yet");
-
-// Handle merging.
-while (baseFileIterator.hasNext()) {
- T baseRecord = baseFileIterator.next();
- nextRecordPosition = readerContext.extractRecordPosition(baseRecord,
readerSchema, ROW_INDEX_TEMPORARY_COLUMN_NAME, nextRecordPosition);
- Pair, Map> logRecordInfo =
records.remove(nextRecordPosition++);
- if (hasNextBaseRecord(baseRecord, logRecordInfo)) {
-return true;
+ protected boolean hasNextBaseRecord(T baseRecord) throws IOException {
+if (!readerContext.getShouldMergeUseRecordPosition()) {
+ return doHasNextFallbackBaseRecord(baseRecord);
+}
+
+nextRecordPosition = readerContext.extractRecordPosition(baseRecord,
readerSchema,
+ROW_INDEX_COLUMN_NAME, nextRecordPosition);
+Pair, Map> logRecordInfo =
records.remove(nextRecordPosition++);
+
+Map metadata = readerContext.generateMetadataForRecord(
+baseRecord, readerSchema);
+
+Option resultRecord = logRecordInfo != null
+? merge(Option.of(baseRecord), metadata, logRecordInfo.getLeft(),
logRecordInfo.getRight())
+: merge(Option.empty(), Collections.emptyMap(), Option.of(baseRecord),
metadata);
+if (resultRecord.isPresent()) {
+ nextRecord = readerContext.seal(resultRecord.get());
+ return true;
+}
+return false;
+ }
+
+ private boolean doHasNextFallbackBaseRecord(T baseRecord) throws IOException
{
+if (needToDoHybridStrategy) {
Review Comment:
let's test this logic as well.
##
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedFileGroupRecordBuffer.java:
##
@@ -123,46
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2155410555 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * bfea0d3a2dd9e6ba2d96c1d7d20a07e085883da6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24278) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2155393640 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 40932069f637e82d80731fe8625331d293fdc1e0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24277) * bfea0d3a2dd9e6ba2d96c1d7d20a07e085883da6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2155337978 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN * 40932069f637e82d80731fe8625331d293fdc1e0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #11415: URL: https://github.com/apache/hudi/pull/11415#issuecomment-2155328233 ## CI report: * 644a1d216307d8660ff7654c5273f2356974bcb8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex opened a new pull request, #11415: URL: https://github.com/apache/hudi/pull/11415 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex closed pull request #10991: [HUDI-7269] Fallback to key based merge if positions are missing from log block URL: https://github.com/apache/hudi/pull/10991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2098943332 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 75d6213f80d3e37c25a7fd57adf12d82b421315a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23745) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2098868013 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 1ccd5cf1dcd5a65971cf44b953ec2e450ff51f85 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23554) * 75d6213f80d3e37c25a7fd57adf12d82b421315a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2083206521 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 1ccd5cf1dcd5a65971cf44b953ec2e450ff51f85 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23554) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2083124564 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * d5a567ec55c8a3acc7c79ffe07e9870876fdb0fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23551) * 5b2ce304951275b5c175916578665794871043da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23553) * 1ccd5cf1dcd5a65971cf44b953ec2e450ff51f85 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2083102113 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * d5a567ec55c8a3acc7c79ffe07e9870876fdb0fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23551) * 5b2ce304951275b5c175916578665794871043da UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2083086127 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * d5a567ec55c8a3acc7c79ffe07e9870876fdb0fe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23551) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2082989151 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7a333e271d99a5364627445c4ff97f74166f879c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23327) * d5a567ec55c8a3acc7c79ffe07e9870876fdb0fe Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23551) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2082970001 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7a333e271d99a5364627445c4ff97f74166f879c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23327) * d5a567ec55c8a3acc7c79ffe07e9870876fdb0fe UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2062619771 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7a333e271d99a5364627445c4ff97f74166f879c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23327) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2062546558 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7db55a55071a0c445e53b06f8a8b7ce01c369046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23326) * 7a333e271d99a5364627445c4ff97f74166f879c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23327) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2062523957 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7db55a55071a0c445e53b06f8a8b7ce01c369046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23326) * 7a333e271d99a5364627445c4ff97f74166f879c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2062503218 ## CI report: * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7db55a55071a0c445e53b06f8a8b7ce01c369046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23326) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2062365428 ## CI report: * 9b5a2a5f69fa40f9dbd6e10d0c1c3fe9457b71da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23269) * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN * 7db55a55071a0c445e53b06f8a8b7ce01c369046 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2062336382 ## CI report: * 9b5a2a5f69fa40f9dbd6e10d0c1c3fe9457b71da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23269) * e50bc34717247521d84a2f7f9a014f3322c37bdd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2057814505 ## CI report: * 9b5a2a5f69fa40f9dbd6e10d0c1c3fe9457b71da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23269) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2057613466 ## CI report: * 2af03c004aef66248dae6283e9c2f1e63e062e75 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23229) * 9b5a2a5f69fa40f9dbd6e10d0c1c3fe9457b71da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23269) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2057600936 ## CI report: * 2af03c004aef66248dae6283e9c2f1e63e062e75 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23229) * 9b5a2a5f69fa40f9dbd6e10d0c1c3fe9457b71da UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2052596559 ## CI report: * 2af03c004aef66248dae6283e9c2f1e63e062e75 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23229) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2052437226 ## CI report: * 7dfe5ef7fa89cebfca107cd54ca9f417eff2ba3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23194) * 2af03c004aef66248dae6283e9c2f1e63e062e75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23229) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2052428509 ## CI report: * 7dfe5ef7fa89cebfca107cd54ca9f417eff2ba3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23194) * 2af03c004aef66248dae6283e9c2f1e63e062e75 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2050317203 ## CI report: * 7dfe5ef7fa89cebfca107cd54ca9f417eff2ba3c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23194) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2050201427 ## CI report: * 71cccd9b017eabe36316f2d793945aabab7a407f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23185) * 7dfe5ef7fa89cebfca107cd54ca9f417eff2ba3c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23194) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2050189132 ## CI report: * 71cccd9b017eabe36316f2d793945aabab7a407f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23185) * 7dfe5ef7fa89cebfca107cd54ca9f417eff2ba3c UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2048958866 ## CI report: * 71cccd9b017eabe36316f2d793945aabab7a407f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23185) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2048840423 ## CI report: * 12d7d0bf249cbc532e8f251ad8b846d334ed44e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23180) * 71cccd9b017eabe36316f2d793945aabab7a407f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23185) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2048835589 ## CI report: * 12d7d0bf249cbc532e8f251ad8b846d334ed44e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23180) * 71cccd9b017eabe36316f2d793945aabab7a407f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2048775604 ## CI report: * 12d7d0bf249cbc532e8f251ad8b846d334ed44e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23180) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2048671880 ## CI report: * 12d7d0bf249cbc532e8f251ad8b846d334ed44e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23180) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
hudi-bot commented on PR #10991: URL: https://github.com/apache/hudi/pull/10991#issuecomment-2048655156 ## CI report: * 12d7d0bf249cbc532e8f251ad8b846d334ed44e3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7269] Fallback to key based merge if positions are missing from log block [hudi]
jonvex opened a new pull request, #10991: URL: https://github.com/apache/hudi/pull/10991 ### Change Logs Fallback to key based merge if positions are missing from log block STILL NEED TO ADD TESTS ### Impact positional queries won't fail if a log file is missing positions ### Risk level (write none, low medium or high below) medium ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
