Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
danny0405 commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1591712468
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieWriteConflictException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommitMetadata;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createInflightCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createPendingCompaction;
+
+public class
TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable extends
HoodieCommonTestHarness {
+ @Override
+ protected HoodieTableType getTableType() {
+return HoodieTableType.MERGE_ON_READ;
+ }
+
+ @BeforeEach
+ public void init() throws IOException {
+initMetaClient();
+ }
+
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// Consider commits before this are all successful.
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+
+// Writer 1 starts.
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+
+// Compaction 1 gets scheduled and becomes inflight.
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+// Writer 1 tries to commit.
+Option currentInstant = Option.of(
+new HoodieInstant(HoodieInstant.State.INFLIGHT,
HoodieTimeline.DELTA_COMMIT_ACTION, currentWriterInstant));
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+
+// Do conflict resolution.
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy =
+new SimpleConcurrentFileWritesConflictResolutionStrategy();
+List candidateInstants = strategy.getCandidateInstants(
+metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(Collectors.toList());
+Assertions.assertEquals(1, candidateInstants.size());
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+Assertions.assertThrows(
+HoodieWriteConflictException.class,
Review Comment:
I have left some comments in Lin's last fix, I can not figure a case where
the requested compaction and writer conflict with each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
jonvex merged PR #11151: URL: https://github.com/apache/hudi/pull/11151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2096646236 ## CI report: * 9b3d8e2d83d44f9c970057d05bc8f1219cc2a203 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23704) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2096634404 ## CI report: * 9b3d8e2d83d44f9c970057d05bc8f1219cc2a203 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
codope commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1591377388
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieWriteConflictException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommitMetadata;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createInflightCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createPendingCompaction;
+
+public class
TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable extends
HoodieCommonTestHarness {
+ @Override
+ protected HoodieTableType getTableType() {
+return HoodieTableType.MERGE_ON_READ;
+ }
+
+ @BeforeEach
+ public void init() throws IOException {
+initMetaClient();
+ }
+
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// Consider commits before this are all successful.
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+
+// Writer 1 starts.
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+
+// Compaction 1 gets scheduled and becomes inflight.
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+// Writer 1 tries to commit.
+Option currentInstant = Option.of(
+new HoodieInstant(HoodieInstant.State.INFLIGHT,
HoodieTimeline.DELTA_COMMIT_ACTION, currentWriterInstant));
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+
+// Do conflict resolution.
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy =
+new SimpleConcurrentFileWritesConflictResolutionStrategy();
+List candidateInstants = strategy.getCandidateInstants(
+metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(Collectors.toList());
+Assertions.assertEquals(1, candidateInstants.size());
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+Assertions.assertThrows(
+HoodieWriteConflictException.class,
Review Comment:
Ok, let;s track it separately. Not related to this PR. @danny0405 Is there a
new conflict resolution strategy with the file slicing based on completion time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1591296650
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieWriteConflictException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommitMetadata;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createInflightCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createPendingCompaction;
+
+public class
TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable extends
HoodieCommonTestHarness {
+ @Override
+ protected HoodieTableType getTableType() {
+return HoodieTableType.MERGE_ON_READ;
+ }
+
+ @BeforeEach
+ public void init() throws IOException {
+initMetaClient();
+ }
+
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// Consider commits before this are all successful.
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+
+// Writer 1 starts.
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+
+// Compaction 1 gets scheduled and becomes inflight.
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+// Writer 1 tries to commit.
+Option currentInstant = Option.of(
+new HoodieInstant(HoodieInstant.State.INFLIGHT,
HoodieTimeline.DELTA_COMMIT_ACTION, currentWriterInstant));
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+
+// Do conflict resolution.
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy =
+new SimpleConcurrentFileWritesConflictResolutionStrategy();
+List candidateInstants = strategy.getCandidateInstants(
+metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(Collectors.toList());
+Assertions.assertEquals(1, candidateInstants.size());
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+Assertions.assertThrows(
+HoodieWriteConflictException.class,
Review Comment:
Just confirmed: without the fix, the unit test would throw NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1591251073
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieWriteConflictException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommitMetadata;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createInflightCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createPendingCompaction;
+
+public class
TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable extends
HoodieCommonTestHarness {
+ @Override
+ protected HoodieTableType getTableType() {
+return HoodieTableType.MERGE_ON_READ;
+ }
+
+ @BeforeEach
+ public void init() throws IOException {
+initMetaClient();
+ }
+
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// Consider commits before this are all successful.
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+
+// Writer 1 starts.
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+
+// Compaction 1 gets scheduled and becomes inflight.
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+// Writer 1 tries to commit.
+Option currentInstant = Option.of(
+new HoodieInstant(HoodieInstant.State.INFLIGHT,
HoodieTimeline.DELTA_COMMIT_ACTION, currentWriterInstant));
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+
+// Do conflict resolution.
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy =
+new SimpleConcurrentFileWritesConflictResolutionStrategy();
+List candidateInstants = strategy.getCandidateInstants(
+metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(Collectors.toList());
+Assertions.assertEquals(1, candidateInstants.size());
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+Assertions.assertThrows(
+HoodieWriteConflictException.class,
Review Comment:
Probably not since if I don't add the fix, and run the new unit test, it
will throw NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151: URL: https://github.com/apache/hudi/pull/11151#discussion_r1591241978 ## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/SimpleConcurrentFileWritesConflictResolutionStrategy.java: ## @@ -69,7 +68,6 @@ public Stream getCandidateInstants(HoodieTableMetaClient metaClie .getTimelineOfActions(CollectionUtils.createSet(REPLACE_COMMIT_ACTION, COMPACTION_ACTION)) .findInstantsAfter(currentInstant.getTimestamp()) .filterInflightsAndRequested() -.filter(i -> (!i.getAction().equals(COMPACTION_ACTION)) || i.getState().equals(REQUESTED)) Review Comment: This is my last fix, which is wrong; so I remove it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
codope commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1591194888
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.transaction;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieWriteConflictException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createCommitMetadata;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createInflightCommit;
+import static
org.apache.hudi.client.transaction.TestConflictResolutionStrategyUtil.createPendingCompaction;
+
+public class
TestSimpleConcurrentFileWritesConflictResolutionStrategyWithMORTable extends
HoodieCommonTestHarness {
+ @Override
+ protected HoodieTableType getTableType() {
+return HoodieTableType.MERGE_ON_READ;
+ }
+
+ @BeforeEach
+ public void init() throws IOException {
+initMetaClient();
+ }
+
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// Consider commits before this are all successful.
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+
+// Writer 1 starts.
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+
+// Compaction 1 gets scheduled and becomes inflight.
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+// Writer 1 tries to commit.
+Option currentInstant = Option.of(
+new HoodieInstant(HoodieInstant.State.INFLIGHT,
HoodieTimeline.DELTA_COMMIT_ACTION, currentWriterInstant));
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+
+// Do conflict resolution.
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy =
+new SimpleConcurrentFileWritesConflictResolutionStrategy();
+List candidateInstants = strategy.getCandidateInstants(
+metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(Collectors.toList());
+Assertions.assertEquals(1, candidateInstants.size());
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+Assertions.assertThrows(
+HoodieWriteConflictException.class,
Review Comment:
I think with the current master code, we should not be seeing conflict
exception between writer and compactor. We already resolved it with the new
file slicing algorithm. @danny0405 please correct me if i'm wrong.
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/SimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -69,7 +68,6 @@ public Stream
getCandidateInstants(HoodieTableMetaClient metaClie
.getTimelineOfActions(CollectionUtils.c
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2096181246 ## CI report: * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * 9b3d8e2d83d44f9c970057d05bc8f1219cc2a203 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23704) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2096163605 ## CI report: * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * eb1c4d47c1464b870719109da1944cec21cd4327 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23681) * 9b3d8e2d83d44f9c970057d05bc8f1219cc2a203 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2096079305 @danny0405 , updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1591047353
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +198,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION,
currentWriterInstant));
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy = new
SimpleConcurrentFileWritesConflictResolutionStrategy();
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+List candidateInstants =
strategy.getCandidateInstants(metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(
+Collectors.toList());
+// writer 1 conflicts with compaction 1
+Assertions.assertTrue(candidateInstants.size() == 1);
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+try {
+ strategy.resolveConflict(null, thisCommitOperation, thatCommitOperation);
+ Assertions.fail("Cannot reach here, should have thrown a conflict");
Review Comment:
Done.
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +198,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION,
currentWriterInstant));
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy = new
SimpleConcurrentFileWritesConflictResolutionStrategy();
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+List candidateInstants =
strategy.getCandidateInstants(metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(
+Collectors.toList());
+// writer 1 conflicts with compaction 1
+Assertions.assertTrue(candidateInstants.size() == 1);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2095279790 ## CI report: * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * eb1c4d47c1464b870719109da1944cec21cd4327 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23681) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2095223554 ## CI report: * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * bd09a1b36becfbcdc75195427148eb948e384ac5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23664) * eb1c4d47c1464b870719109da1944cec21cd4327 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23681) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2095217286 ## CI report: * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * bd09a1b36becfbcdc75195427148eb948e384ac5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23664) * eb1c4d47c1464b870719109da1944cec21cd4327 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
danny0405 commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590544744
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +198,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION,
currentWriterInstant));
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy = new
SimpleConcurrentFileWritesConflictResolutionStrategy();
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+List candidateInstants =
strategy.getCandidateInstants(metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(
+Collectors.toList());
+// writer 1 conflicts with compaction 1
+Assertions.assertTrue(candidateInstants.size() == 1);
+ConcurrentOperation thatCommitOperation = new
ConcurrentOperation(candidateInstants.get(0), metaClient);
+ConcurrentOperation thisCommitOperation = new
ConcurrentOperation(currentInstant.get(), currentMetadata);
+Assertions.assertTrue(strategy.hasConflict(thisCommitOperation,
thatCommitOperation));
+try {
+ strategy.resolveConflict(null, thisCommitOperation, thatCommitOperation);
+ Assertions.fail("Cannot reach here, should have thrown a conflict");
Review Comment:
Use `assertThrows` instead, and let's also supplement the msgs for these
assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
danny0405 commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590543447
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +198,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION,
currentWriterInstant));
+SimpleConcurrentFileWritesConflictResolutionStrategy strategy = new
SimpleConcurrentFileWritesConflictResolutionStrategy();
+HoodieCommitMetadata currentMetadata =
createCommitMetadata(currentWriterInstant);
+metaClient.reloadActiveTimeline();
+List candidateInstants =
strategy.getCandidateInstants(metaClient, currentInstant.get(),
lastSuccessfulInstant).collect(
+Collectors.toList());
+// writer 1 conflicts with compaction 1
+Assertions.assertTrue(candidateInstants.size() == 1);
Review Comment:
Using `assertEquals` should be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590529084
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +193,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION,
currentWriterInstant));
Review Comment:
Second thoughts. Let me use MOR instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590513268
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +193,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION,
currentWriterInstant));
Review Comment:
Yeah, I know we should use MOR table for compaction. But this entire test
class uses COW to test concurrency with compaction. It should be OK since we
only test the conflict resolution logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590513268
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +193,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION,
currentWriterInstant));
Review Comment:
Yeah, I know we should use MOR table for compaction. But this entire test
class uses COW to test concurrency with compaction. I guess it should be OK
since we only test the conflict resolution logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590513268
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +193,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION,
currentWriterInstant));
Review Comment:
Yeah, I know we should use MOR table for compaction. But this entire test
uses COW to test concurrency with compaction. I guess it should be OK since we
only test the conflict resolution logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
danny0405 commented on code in PR #11151:
URL: https://github.com/apache/hudi/pull/11151#discussion_r1590453222
##
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/TestSimpleConcurrentFileWritesConflictResolutionStrategy.java:
##
@@ -193,6 +193,38 @@ public void
testConcurrentWritesWithInterleavingScheduledCompaction() throws Exc
}
}
+ @Test
+ public void testConcurrentWritesWithInterleavingInflightCompaction() throws
Exception {
+createCommit(metaClient.createNewInstantTime(), metaClient);
+HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+// consider commits before this are all successful
+Option lastSuccessfulInstant =
timeline.getCommitsTimeline().filterCompletedInstants().lastInstant();
+// writer 1 starts
+String currentWriterInstant = metaClient.createNewInstantTime();
+createInflightCommit(currentWriterInstant, metaClient);
+// compaction 1 gets scheduled and becomes inflight
+String newInstantTime = metaClient.createNewInstantTime();
+createPendingCompaction(newInstantTime, metaClient);
+
+Option currentInstant = Option.of(new
HoodieInstant(State.INFLIGHT, HoodieTimeline.COMMIT_ACTION,
currentWriterInstant));
Review Comment:
So this is a COW table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
linliu-code commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2094949761 @yihua @nsivabalan Please review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2094661509 ## CI report: * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * bd09a1b36becfbcdc75195427148eb948e384ac5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23664) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2094649256 ## CI report: * c68b630ed8b878dffc4df1f1074d6fa3987899d9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23663) * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * bd09a1b36becfbcdc75195427148eb948e384ac5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23664) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2094647826 ## CI report: * c68b630ed8b878dffc4df1f1074d6fa3987899d9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23663) * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN * bd09a1b36becfbcdc75195427148eb948e384ac5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2094646391 ## CI report: * c68b630ed8b878dffc4df1f1074d6fa3987899d9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23663) * 2985ea2ec2f8a0b62086a6ac9933654051a65738 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7710] Use compaction.requested during conflict resolution [hudi]
hudi-bot commented on PR #11151: URL: https://github.com/apache/hudi/pull/11151#issuecomment-2094637684 ## CI report: * c68b630ed8b878dffc4df1f1074d6fa3987899d9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
