[flink] branch master updated (d32ca07 -> d98d731)
This is an automated email from the ASF dual-hosted git repository. godfrey pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from d32ca07 [FLINK-22310][table-planner-blink] Fix the incorrect deserialization result of LogicalWindowJsonDeserializer add c6147f2 [FLINK-21627][table-planner-blink] The digest of TableScan should consider table hints add d98d731 [FLINK-21627][table-planner-blink] The digest of Sink should consider table hints No new revisions were added by this update. Summary of changes: .../calcite/rel/logical/LogicalTableScan.java | 137 + .../table/planner/connectors/DynamicSinkUtils.java | 12 +- .../PushFilterInCalcIntoTableSourceScanRule.java | 3 +- .../logical/PushLimitIntoTableSourceScanRule.java | 3 +- .../PushWatermarkIntoTableSourceScanRuleBase.java | 3 +- .../planner/calcite/FlinkLogicalRelFactories.scala | 2 + .../calcite/RelTimeIndicatorConverter.scala| 2 + .../table/planner/delegation/PlannerBase.scala | 7 ++ .../planner/plan/nodes/calcite/LegacySink.scala| 6 + .../plan/nodes/calcite/LogicalLegacySink.scala | 21 +++- .../planner/plan/nodes/calcite/LogicalSink.scala | 7 +- .../table/planner/plan/nodes/calcite/Sink.scala| 30 +++-- .../logical/FlinkLogicalDataStreamTableScan.scala | 25 ++-- .../nodes/logical/FlinkLogicalLegacySink.scala | 12 +- .../FlinkLogicalLegacyTableSourceScan.scala| 20 +-- .../plan/nodes/logical/FlinkLogicalSink.scala | 8 +- .../logical/FlinkLogicalTableSourceScan.scala | 19 +-- .../batch/BatchPhysicalBoundedStreamScan.scala | 10 +- .../physical/batch/BatchPhysicalLegacySink.scala | 8 +- .../batch/BatchPhysicalLegacyTableSourceScan.scala | 6 +- .../nodes/physical/batch/BatchPhysicalSink.scala | 13 +- .../batch/BatchPhysicalTableSourceScan.scala | 6 +- .../CommonPhysicalLegacyTableSourceScan.scala | 8 +- .../common/CommonPhysicalTableSourceScan.scala | 8 +- .../stream/StreamPhysicalDataStreamScan.scala | 10 +- .../physical/stream/StreamPhysicalLegacySink.scala | 8 +- .../StreamPhysicalLegacyTableSourceScan.scala | 8 +- .../nodes/physical/stream/StreamPhysicalSink.scala | 15 ++- .../stream/StreamPhysicalTableSourceScan.scala | 6 +- .../batch/BatchPhysicalBoundedStreamScanRule.scala | 7 +- .../batch/BatchPhysicalLegacySinkRule.scala| 25 ++-- .../BatchPhysicalLegacyTableSourceScanRule.scala | 1 + .../physical/batch/BatchPhysicalSinkRule.scala | 35 +++--- .../batch/BatchPhysicalTableSourceScanRule.scala | 1 + .../stream/StreamPhysicalDataStreamScanRule.scala | 1 + .../stream/StreamPhysicalLegacySinkRule.scala | 25 ++-- .../StreamPhysicalLegacyTableSourceScanRule.scala | 1 + .../physical/stream/StreamPhysicalSinkRule.scala | 35 +++--- .../stream/StreamPhysicalTableSourceScanRule.scala | 1 + .../table/planner/plan/utils/RelExplainUtil.scala | 27 +++- .../planner/factories/TestValuesTableFactory.java | 42 ++- .../planner/plan/batch/sql/LegacySinkTest.xml | 24 .../plan/batch/sql/LegacyTableSourceTest.xml | 38 -- .../table/planner/plan/batch/sql/TableSinkTest.xml | 24 .../planner/plan/batch/sql/TableSourceTest.xml | 73 +++ .../planner/plan/batch/sql/join/LookupJoinTest.xml | 130 ++- .../table/planner/plan/hint/OptionsHintTest.xml| 48 .../rules/logical/CalcRankTransposeRuleTest.xml| 12 +- .../logical/RankNumberColumnRemoveRuleTest.xml | 8 +- .../planner/plan/batch/sql/LegacySinkTest.scala| 37 ++ .../plan/batch/sql/LegacyTableSourceTest.scala | 46 +++ .../planner/plan/batch/sql/TableSinkTest.scala | 45 ++- .../planner/plan/batch/sql/TableSourceTest.scala | 113 + .../plan/metadata/FlinkRelMdHandlerTestBase.scala | 11 +- .../plan/metadata/FlinkRelMdUniqueKeysTest.scala | 5 + .../batch/sql/LegacyTableSourceITCase.scala| 54 +++- .../runtime/batch/sql/TableSinkITCase.scala| 79 .../runtime/batch/sql/TableSourceITCase.scala | 81 +++- .../flink/table/planner/utils/TableTestBase.scala | 13 +- .../table/planner/utils/testTableSourceSinks.scala | 26 +++- .../calcite/rel/logical/LogicalTableScan.java | 137 + 61 files changed, 1341 insertions(+), 287 deletions(-) create mode 100644 flink-table/flink-table-planner-blink/src/main/java/org/apache/calcite/rel/logical/LogicalTableScan.java create mode 100644 flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/batch/sql/TableSinkITCase.scala create mode 100644 flink-table/flink-table-planner/src/main/java/org/apache/calcite/rel/logical/LogicalTableScan.java
[flink] branch master updated (d293577 -> d32ca07)
This is an automated email from the ASF dual-hosted git repository. godfrey pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from d293577 [FLINK-22307][network] Increase the default value of data writing cache size (not configurable) for sort-merge blocking shuffle add d32ca07 [FLINK-22310][table-planner-blink] Fix the incorrect deserialization result of LogicalWindowJsonDeserializer No new revisions were added by this update. Summary of changes: .../expressions/PlannerWindowReference.java| 13 +++ .../exec/serde/LogicalWindowJsonDeserializer.java | 2 +- .../table/planner/plan/logical/groupWindows.scala | 71 +- .../nodes/exec/serde/LogicalWindowSerdeTest.java | 104 ++--- 4 files changed, 129 insertions(+), 61 deletions(-)
[flink] branch master updated: [FLINK-22307][network] Increase the default value of data writing cache size (not configurable) for sort-merge blocking shuffle
This is an automated email from the ASF dual-hosted git repository.
guoweima pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new d293577 [FLINK-22307][network] Increase the default value of data
writing cache size (not configurable) for sort-merge blocking shuffle
d293577 is described below
commit d29357791315ace0218308f9f96bf65c7c8079d7
Author: kevin.cyj
AuthorDate: Thu Apr 15 23:36:07 2021 +0800
[FLINK-22307][network] Increase the default value of data writing cache
size (not configurable) for sort-merge blocking shuffle
Currently, the data writing cache of sort-merge blocking shuffle is 8M,
which can be not enough if data compression is enabled. This patch increases
the cache size to 16M which can improve the performance for high compression
ratio scenarios.
This closes #15651.
---
.../flink/runtime/io/network/partition/SortMergeResultPartition.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
index db15bca..4442ca1 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
@@ -66,9 +66,9 @@ public class SortMergeResultPartition extends ResultPartition
{
/**
* Number of expected buffer size to allocate for data writing. Currently,
it is an empirical
- * value (8M) which can not be configured.
+ * value (16M) which can not be configured.
*/
-private static final int NUM_WRITE_BUFFER_BYTES = 8 * 1024 * 1024;
+private static final int NUM_WRITE_BUFFER_BYTES = 16 * 1024 * 1024;
private final Object lock = new Object();
[flink] branch master updated: [FLINK-22305][network] Improve log messages of sort-merge blocking shuffle
This is an automated email from the ASF dual-hosted git repository.
guoweima pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 77e0478 [FLINK-22305][network] Improve log messages of sort-merge
blocking shuffle
77e0478 is described below
commit 77e0478a7879bd41f9f52872d84c467d158b8098
Author: kevin.cyj
AuthorDate: Sat Apr 17 15:02:44 2021 +0800
[FLINK-22305][network] Improve log messages of sort-merge blocking shuffle
This closes #15652.
---
.../io/disk/BatchShuffleReadBufferPool.java| 10 -
.../io/network/partition/PartitionedFile.java | 25 --
.../network/partition/PartitionedFileWriter.java | 15 -
.../partition/SortMergeResultPartition.java| 20 -
4 files changed, 52 insertions(+), 18 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
index 01da315..1000a44 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
@@ -135,11 +135,6 @@ public class BatchShuffleReadBufferPool {
/** Initializes this buffer pool which allocates all the buffers. */
public void initialize() {
-LOG.info(
-"Initializing batch shuffle IO buffer pool: numBuffers={},
bufferSize={}.",
-numTotalBuffers,
-bufferSize);
-
synchronized (buffers) {
checkState(!destroyed, "Buffer pool is already destroyed.");
@@ -175,6 +170,11 @@ public class BatchShuffleReadBufferPool {
TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
}
}
+
+LOG.info(
+"Batch shuffle IO buffer pool initialized: numBuffers={},
bufferSize={}.",
+numTotalBuffers,
+bufferSize);
}
/**
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFile.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFile.java
index 145a775..dd34fe0 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFile.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFile.java
@@ -63,6 +63,15 @@ public class PartitionedFile {
/** Path of the index file which stores all index entries in this {@link
PartitionedFile}. */
private final Path indexFilePath;
+/** Size of the data file. */
+private final long dataFileSize;
+
+/** Size of the index file. */
+private final long indexFileSize;
+
+/** Total number of buffers in the data file. */
+private final long numBuffers;
+
/** Used to accelerate index data access. */
@Nullable private final ByteBuffer indexEntryCache;
@@ -71,6 +80,9 @@ public class PartitionedFile {
int numSubpartitions,
Path dataFilePath,
Path indexFilePath,
+long dataFileSize,
+long indexFileSize,
+long numBuffers,
@Nullable ByteBuffer indexEntryCache) {
checkArgument(numRegions >= 0, "Illegal number of data regions.");
checkArgument(numSubpartitions > 0, "Illegal number of
subpartitions.");
@@ -79,6 +91,9 @@ public class PartitionedFile {
this.numSubpartitions = numSubpartitions;
this.dataFilePath = checkNotNull(dataFilePath);
this.indexFilePath = checkNotNull(indexFilePath);
+this.dataFileSize = dataFileSize;
+this.indexFileSize = indexFileSize;
+this.numBuffers = numBuffers;
this.indexEntryCache = indexEntryCache;
}
@@ -144,8 +159,14 @@ public class PartitionedFile {
+ dataFilePath
+ ", indexFilePath="
+ indexFilePath
-+ ", indexDataCache="
-+ indexEntryCache
++ ", dataFileSize="
++ dataFileSize
++ ", indexFileSize="
++ indexFileSize
++ ", numBuffers="
++ numBuffers
++ ", indexDataCached="
++ (indexEntryCache != null)
+ '}';
}
}
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFileWriter.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFileWriter.java
index 5090d87..e9952c4 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionedFileWriter.java
+++
[flink] branch release-1.12 updated (902a0f5 -> d39b3fb)
This is an automated email from the ASF dual-hosted git repository. sewen pushed a change to branch release-1.12 in repository https://gitbox.apache.org/repos/asf/flink.git. from 902a0f5 [FLINK-20761][hive] Escape the location path when creating input splits (#15625) add 09613ef [refactor][runtime] Update FutureUtils to support delayed completing of a CompletableFuture add 75e203a [refactor][runtime] Extend AkkaRpcServiceUtils to support instantiating custom AkkaRpcServices. add 5a678af [FLINK-21996][refactor] Make IteratorSourceReader work with multiple split requests. add 45bb748 [FLINK-21996][refactor] Make NumberSequenceSource extensible to allow specifying the number of desired sequence splits. add da48ac8 [FLINK-21996][tests] Add ITCase to test for delayed and failed operator event sending. add 872c422 [FLINK-21996][refactor] Unify exception handling for Operator Coordinator Events sent to not-running tasks add 33ff04f [FLINK-21996][refactor] Pull in-line serialization call into separate statement to improve Exception handling accuracy. add 33bb107 [FLINK-18071][coordination] (part 1) All event sending and checkpoint actions for OperatorCoordinator happen in Scheduler Thread add befcbfa [hotfix][coordination] Add Main-Thread check to OperatorEvent sending on Execution. add bafdebd [FLINK-18071][coordination] (part 2) OperatorCoordinatorHolder does not implement OperatorCoordinator interface any more add 52e52a6 [FLINK-18071][coordination] (part 3) Adjust OperatorEventValve to accept self-contained "send actions". add 207e255 [FLINK-18071][coordination] (part 4) Add to Execution a future for states INITIALIZING/RUNNING add d2f5df3 [FLINK-18071][coordination] (part 5) Communication from Coordinators to Tasks happens through gateways that are scoped to a single execution attempt. add 4b590f2 [hotfix][coordination] Make failed event valve shutting smoother. add 29c274a [hotfix][coordination] Reduce lambda nesting for action on CompletableFuture add 5705646 [hotfix][coordination] Remove unnecessary null check add c0b4a20 [hotfix][tests] Minor debuggability improvements to CoordinatedSourceRescaleITCase add 7ec9054 [hotfix][tests] Simplify and harden CoordinatorEventsExactlyOnceITCase add 19b5a1b [FLINK-21996][coordination] Ensure exactly-once guarantees for OperatorEvent RPCs add d39b3fb [hotfix][coordination] Add safety guard against uncaught exceptions for Future dependent lambdas No new revisions were added by this update. Summary of changes: .../reader/CoordinatedSourceRescaleITCase.java | 11 +- .../connector/source/lib/NumberSequenceSource.java | 45 ++- .../source/lib/util/IteratorSourceReader.java | 80 ++-- .../source/lib/NumberSequenceSourceTest.java | 6 +- .../flink/runtime/concurrent/FutureUtils.java | 15 + .../flink/runtime/executiongraph/Execution.java| 29 +- ...skNotRunningException.java => EventSender.java} | 20 +- .../coordination/ExecutionSubtaskAccess.java | 127 ++ .../coordination/OperatorCoordinator.java | 83 +++- .../coordination/OperatorCoordinatorHolder.java| 311 +- .../operators/coordination/OperatorEventValve.java | 261 +--- .../RecreateOnResetOperatorCoordinator.java| 17 +- .../operators/coordination/SubtaskAccess.java | 98 + .../operators/coordination/SubtaskGatewayImpl.java | 104 + .../coordination/TaskNotRunningException.java | 5 + .../util/IncompleteFuturesTracker.java | 111 + .../runtime/rpc/akka/AkkaRpcServiceUtils.java | 9 +- .../flink/runtime/scheduler/DefaultScheduler.java | 5 +- .../flink/runtime/scheduler/SchedulerBase.java | 2 +- .../source/coordinator/SourceCoordinator.java | 11 + .../coordinator/SourceCoordinatorContext.java | 83 ++-- .../TaskExecutorGatewayDecoratorBase.java | 222 ++ .../CoordinatorEventsExactlyOnceITCase.java| 244 --- .../coordination/EventReceivingTasks.java | 233 +++ .../coordination/MockOperatorCoordinator.java | 5 + .../MockOperatorCoordinatorContext.java| 54 +-- .../OperatorCoordinatorHolderTest.java | 207 +- .../OperatorCoordinatorSchedulerTest.java | 42 +- .../coordination/OperatorEventValveTest.java | 140 +++ .../RecreateOnResetOperatorCoordinatorTest.java| 38 +- .../operators/coordination/TestEventSender.java| 109 - .../coordination/TestingOperatorCoordinator.java | 21 +- .../util/IncompleteFuturesTrackerTest.java | 117 ++ .../coordinator/SourceCoordinatorContextTest.java | 97 +++-- .../source/coordinator/SourceCoordinatorTest.java | 293 +- .../coordinator/SourceCoordinatorTestBase.java | 117 -- .../source/coordinator/TestingSplitEnumerator.java |
[flink] branch master updated (fd8e34c -> 577113f)
This is an automated email from the ASF dual-hosted git repository. roman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from fd8e34c [FLINK-20855][table-runtime-blink] Fix calculating numBuckets overflow (#14566) add 577113f [FLINK-22117] Reduce the logs if not all tasks are RUNNING when checkpointing No new revisions were added by this update. Summary of changes: .../org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java | 5 - 1 file changed, 4 insertions(+), 1 deletion(-)
[flink-benchmarks] branch master updated: [FLINK-22290][checkpointing] Use duration for alignment timeout #2.
This is an automated email from the ASF dual-hosted git repository. arvid pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink-benchmarks.git The following commit(s) were added to refs/heads/master by this push: new 6900fb9 [FLINK-22290][checkpointing] Use duration for alignment timeout #2. 6900fb9 is described below commit 6900fb956903fef0315a276785c6ebf189c8fa73 Author: Arvid Heise AuthorDate: Sat Apr 17 09:16:32 2021 +0200 [FLINK-22290][checkpointing] Use duration for alignment timeout #2. --- .../org/apache/flink/benchmark/UnalignedCheckpointTimeBenchmark.java | 1 + 1 file changed, 1 insertion(+) diff --git a/src/main/java/org/apache/flink/benchmark/UnalignedCheckpointTimeBenchmark.java b/src/main/java/org/apache/flink/benchmark/UnalignedCheckpointTimeBenchmark.java index c40a3e3..df736b8 100644 --- a/src/main/java/org/apache/flink/benchmark/UnalignedCheckpointTimeBenchmark.java +++ b/src/main/java/org/apache/flink/benchmark/UnalignedCheckpointTimeBenchmark.java @@ -40,6 +40,7 @@ import org.openjdk.jmh.runner.options.OptionsBuilder; import org.openjdk.jmh.runner.options.VerboseMode; import java.io.IOException; +import java.time.Duration; import static java.util.concurrent.TimeUnit.MINUTES;
