[jira] [Comment Edited] (FLINK-28334) PushProjectIntoTableSourceScanRule should cover the case when table source SupportsReadingMetadata and not SupportsProjectionPushDown
[
https://issues.apache.org/jira/browse/FLINK-28334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563593#comment-17563593
]
godfrey he edited comment on FLINK-28334 at 7/7/22 6:55 AM:
Fix in
master: fb9843af5ffeb6d7561876704d463dea1fcdc153
1.15.2: 1dc4c5ba71e2c920ee94bf4274a850db6bc870d9
was (Author: godfreyhe):
Fix in
master: fb9843af5ffeb6d7561876704d463dea1fcdc153
> PushProjectIntoTableSourceScanRule should cover the case when table source
> SupportsReadingMetadata and not SupportsProjectionPushDown
> -
>
> Key: FLINK-28334
> URL: https://issues.apache.org/jira/browse/FLINK-28334
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> "SELECT id, ts FROM src" query on such a table:
> {code}
> CREATE TABLE src (
> id int,
> name varchar,
> tags varchar METADATA VIRTUAL,
> ts timestamp(3) METADATA VIRTUAL
> ) WITH (
> 'connector' = 'values',
> 'readable-metadata' = 'tags:varchar,ts:timestamp(3)',
> 'enable-projection-push-down' = 'false'
> )
> {code}
>
> error occurs
> {code}
> java.lang.AssertionError: Sql optimization: Assertion error: RexInputRef
> index 3 out of range 0..2
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:84)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:62)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
> at scala.collection.Iterator.foreach(Iterator.scala:937)
> at scala.collection.Iterator.foreach$(Iterator.scala:937)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
> at scala.collection.IterableLike.foreach(IterableLike.scala:70)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
> at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:58)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.optimizeTree(StreamCommonSubGraphBasedOptimizerV2.scala:209)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.optimizeBlock(StreamCommonSubGraphBasedOptimizerV2.scala:156)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.$anonfun$doOptimize$1(StreamCommonSubGraphBasedOptimizerV2.scala:79)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.$anonfun$doOptimize$1$adapted(StreamCommonSubGraphBasedOptimizerV2.scala:78)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.doOptimize(StreamCommonSubGraphBasedOptimizerV2.scala:78)
> at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:94)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:389)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1199)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan2(TableTestBase.scala:1109)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:1066)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:687)
> at
> org.apache.flink.table.planner.plan.nodes.exec.stream.TableSourceJsonPlanTest.testReuseSourceWithoutProjectionPushDown(TableSourceJsonPlanTest.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
[GitHub] [flink-table-store] LadyForest commented on a diff in pull request #182: [FLINK-27708] Add background compaction task for append-only table when ingesting
LadyForest commented on code in PR #182:
URL: https://github.com/apache/flink-table-store/pull/182#discussion_r915516473
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/operation/AppendOnlyFileStoreWrite.java:
##
@@ -82,9 +100,48 @@ public Callable createCompactWriter(
}
private RecordWriter createWriter(
-BinaryRowData partition, int bucket, long maxSeqNum) {
+BinaryRowData partition,
+int bucket,
+List restoredFiles,
+ExecutorService compactExecutor) {
+// let writer and compact manager hold the same reference
+// and make restore files mutable to update
+LinkedList toCompact = new LinkedList<>(restoredFiles);
Review Comment:
> Maybe we don't need to restore files.
Could you explain the reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] LadyForest commented on a diff in pull request #182: [FLINK-27708] Add background compaction task for append-only table when ingesting
LadyForest commented on code in PR #182:
URL: https://github.com/apache/flink-table-store/pull/182#discussion_r915513633
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/data/AppendOnlyCompactManager.java:
##
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.file.data;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.table.store.file.compact.CompactManager;
+import org.apache.flink.table.store.file.compact.CompactResult;
+import org.apache.flink.table.store.file.compact.CompactTask;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ExecutorService;
+
+/** Compact manager for {@link
org.apache.flink.table.store.file.AppendOnlyFileStore}. */
+public class AppendOnlyCompactManager extends CompactManager {
+
+private final int minFileNum;
+private final int maxFileNum;
+private final long targetFileSize;
+private final CompactRewriter rewriter;
+private final LinkedList toCompact;
+
+public AppendOnlyCompactManager(
+ExecutorService executor,
+LinkedList toCompact,
+int minFileNum,
+int maxFileNum,
+long targetFileSize,
+CompactRewriter rewriter) {
+super(executor);
+this.toCompact = toCompact;
+this.maxFileNum = maxFileNum;
+this.minFileNum = minFileNum;
+this.targetFileSize = targetFileSize;
+this.rewriter = rewriter;
+}
+
+@Override
+public void submitCompaction() {
+if (taskFuture != null) {
+throw new IllegalStateException(
+"Please finish the previous compaction before submitting
new one.");
+}
+pickCompactBefore()
+.ifPresent(
+(compactBefore) ->
+taskFuture =
+executor.submit(
+new CompactTask() {
+@Override
+protected CompactResult
compact()
+throws Exception {
+
collectBeforeStats(compactBefore);
+List
compactAfter =
+
rewriter.rewrite(compactBefore);
+
collectAfterStats(compactAfter);
+return
result(compactBefore, compactAfter);
+}
+}));
+}
+
+@VisibleForTesting
+Optional> pickCompactBefore() {
+long totalFileSize = 0L;
+int fileNum = 0;
+int releaseCtr = 0;
+for (int i = 0; i < toCompact.size(); i++) {
+DataFileMeta file = toCompact.get(i);
+totalFileSize += file.fileSize();
+fileNum++;
+int pos = i - fileNum + 1;
+if ((totalFileSize >= targetFileSize && fileNum >= minFileNum)
+|| fileNum >= maxFileNum) {
+// trigger compaction for [pos, i]
+List compactBefore = new
ArrayList<>(toCompact.subList(pos, i + 1));
+// files in [0, pos - 1] can be released immediately
+// [pos, i] should be released after compaction finished
+for (int j = 0; j <= pos - 1; j++) {
+toCompact.pollFirst();
+}
+return Optional.of(compactBefore);
+} else if (totalFileSize >= targetFileSize) {
+// this is equivalent to shift one pos to right
+fileNum--;
Review Comment:
> Why not poll first directly?
Because we still need to iterate on `toCompact`. I think we can poll
`toCompact` only when we find a qualified sub-list to be compa
[jira] [Created] (FLINK-28436) test_multi_sessionjob.sh is failing intermittently
Matyas Orhidi created FLINK-28436: - Summary: test_multi_sessionjob.sh is failing intermittently Key: FLINK-28436 URL: https://issues.apache.org/jira/browse/FLINK-28436 Project: Flink Issue Type: Bug Components: Kubernetes Operator Affects Versions: kubernetes-operator-1.1.0 Reporter: Matyas Orhidi Fix For: kubernetes-operator-1.1.0 https://github.com/apache/flink-kubernetes-operator/runs/7222745771?check_suite_focus=true -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] swuferhong commented on a diff in pull request #20009: [FLINK-27991][table-planner] ORC format supports reporting statistics
swuferhong commented on code in PR #20009:
URL: https://github.com/apache/flink/pull/20009#discussion_r915507045
##
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileFormatFactory.java:
##
@@ -167,5 +188,179 @@ public ChangelogMode getChangelogMode() {
public void applyFilters(List filters) {
this.filters = filters;
}
+
+@Override
+public TableStats reportStatistics(List files, DataType
producedDataType) {
+try {
+Properties properties = getOrcProperties(formatOptions);
+Configuration hadoopConfig = new Configuration();
+properties.forEach((k, v) -> hadoopConfig.set(k.toString(),
v.toString()));
+
+long rowCount = 0;
+Map columnStatisticsMap = new
HashMap<>();
+RowType producedRowType = (RowType)
producedDataType.getLogicalType();
+for (Path file : files) {
+rowCount +=
+updateStatistics(
+hadoopConfig, file, columnStatisticsMap,
producedRowType);
+}
+
+Map columnStatsMap =
+convertToColumnStats(rowCount, columnStatisticsMap,
producedRowType);
+
+return new TableStats(rowCount, columnStatsMap);
+} catch (Exception e) {
+return TableStats.UNKNOWN;
+}
+}
+
+private long updateStatistics(
+Configuration hadoopConf,
+Path file,
+Map columnStatisticsMap,
+RowType producedRowType)
+throws IOException {
+org.apache.hadoop.fs.Path path = new
org.apache.hadoop.fs.Path(file.toUri());
+Reader reader =
+OrcFile.createReader(
+path,
+OrcFile.readerOptions(hadoopConf)
+
.maxLength(OrcConf.MAX_FILE_LENGTH.getLong(hadoopConf)));
+ColumnStatistics[] statistics = reader.getStatistics();
+TypeDescription schema = reader.getSchema();
+List fieldNames = schema.getFieldNames();
+List columnTypes = schema.getChildren();
+for (String column : producedRowType.getFieldNames()) {
+int fieldIdx = fieldNames.indexOf(column);
+if (fieldIdx >= 0) {
+int colId = columnTypes.get(fieldIdx).getId();
+ColumnStatistics statistic = statistics[colId];
+updateStatistics(statistic, column, columnStatisticsMap);
+}
+}
+
+return reader.getNumberOfRows();
+}
+
+private void updateStatistics(
+ColumnStatistics statistic,
+String column,
+Map columnStatisticsMap) {
+ColumnStatistics previousStatistics =
columnStatisticsMap.get(column);
+if (previousStatistics == null) {
+columnStatisticsMap.put(column, statistic);
+} else {
+if (previousStatistics instanceof ColumnStatisticsImpl) {
+((ColumnStatisticsImpl) previousStatistics)
+.merge((ColumnStatisticsImpl) statistic);
+}
+}
+}
+
+private Map convertToColumnStats(
+long totalRowCount,
+Map columnStatisticsMap,
+RowType logicalType) {
+Map columnStatsMap = new HashMap<>();
+for (String column : logicalType.getFieldNames()) {
+ColumnStatistics columnStatistics =
columnStatisticsMap.get(column);
+if (columnStatistics == null) {
+continue;
+}
+ColumnStats columnStats =
+convertToColumnStats(
+totalRowCount,
+
logicalType.getTypeAt(logicalType.getFieldIndex(column)),
+columnStatistics);
+columnStatsMap.put(column, columnStats);
+}
+
+return columnStatsMap;
+}
+
+private ColumnStats convertToColumnStats(
+long totalRowCount, LogicalType logicalType, ColumnStatistics
columnStatistics) {
+ColumnStats.Builder builder =
+new
ColumnStats.Builder().setNdv(null).setAvgLen(null).setMaxLen(null);
+if (!columnStatistics.hasNull()) {
+builder.setNullCount(0L);
+} else {
+builder.setNullCount(totalRowCount -
columnStatistics.getNumberOfValues());
+}
+
+switch (logicalType.getTypeRoot()) {
+case BOOLEAN:
+builder.setMax(null).setMin(null);
+break;
+case TIN
[jira] [Closed] (FLINK-28334) PushProjectIntoTableSourceScanRule should cover the case when table source SupportsReadingMetadata and not SupportsProjectionPushDown
[
https://issues.apache.org/jira/browse/FLINK-28334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he closed FLINK-28334.
--
Resolution: Fixed
Fix in
master: fb9843af5ffeb6d7561876704d463dea1fcdc153
> PushProjectIntoTableSourceScanRule should cover the case when table source
> SupportsReadingMetadata and not SupportsProjectionPushDown
> -
>
> Key: FLINK-28334
> URL: https://issues.apache.org/jira/browse/FLINK-28334
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> "SELECT id, ts FROM src" query on such a table:
> {code}
> CREATE TABLE src (
> id int,
> name varchar,
> tags varchar METADATA VIRTUAL,
> ts timestamp(3) METADATA VIRTUAL
> ) WITH (
> 'connector' = 'values',
> 'readable-metadata' = 'tags:varchar,ts:timestamp(3)',
> 'enable-projection-push-down' = 'false'
> )
> {code}
>
> error occurs
> {code}
> java.lang.AssertionError: Sql optimization: Assertion error: RexInputRef
> index 3 out of range 0..2
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:84)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:62)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
> at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
> at scala.collection.Iterator.foreach(Iterator.scala:937)
> at scala.collection.Iterator.foreach$(Iterator.scala:937)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
> at scala.collection.IterableLike.foreach(IterableLike.scala:70)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
> at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:58)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.optimizeTree(StreamCommonSubGraphBasedOptimizerV2.scala:209)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.optimizeBlock(StreamCommonSubGraphBasedOptimizerV2.scala:156)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.$anonfun$doOptimize$1(StreamCommonSubGraphBasedOptimizerV2.scala:79)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.$anonfun$doOptimize$1$adapted(StreamCommonSubGraphBasedOptimizerV2.scala:78)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizerV2.doOptimize(StreamCommonSubGraphBasedOptimizerV2.scala:78)
> at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:94)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:389)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.assertPlanEquals(TableTestBase.scala:1199)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan2(TableTestBase.scala:1109)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.doVerifyPlan(TableTestBase.scala:1066)
> at
> org.apache.flink.table.planner.utils.TableTestUtilBase.verifyExecPlan(TableTestBase.scala:687)
> at
> org.apache.flink.table.planner.plan.nodes.exec.stream.TableSourceJsonPlanTest.testReuseSourceWithoutProjectionPushDown(TableSourceJsonPlanTest.java:308)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable
[GitHub] [flink] godfreyhe closed pull request #20118: [FLINK-28334][table-planner] Fix PushProjectIntoTableSourceScanRule: covers the case when table source SupportsReadingMetadata and not SupportsPr
godfreyhe closed pull request #20118: [FLINK-28334][table-planner] Fix PushProjectIntoTableSourceScanRule: covers the case when table source SupportsReadingMetadata and not SupportsProjectionPushDown URL: https://github.com/apache/flink/pull/20118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #202: [FLINK-28110] Support projection pushdown for Hive readers
JingsongLi commented on code in PR #202:
URL: https://github.com/apache/flink-table-store/pull/202#discussion_r915498253
##
flink-table-store-hive/flink-table-store-hive-connector/src/main/java/org/apache/flink/table/store/mapred/TableStoreRecordReader.java:
##
@@ -36,24 +41,44 @@ public class TableStoreRecordReader implements
RecordReader iterator;
private final long splitLength;
+@Nullable private final ProjectedRowData reusedProjectedRow;
+
private float progress;
public TableStoreRecordReader(
-org.apache.flink.table.store.file.utils.RecordReader
wrapped,
-long splitLength) {
-this.iterator = new RecordReaderIterator<>(wrapped);
-this.splitLength = splitLength;
+TableRead read,
+TableStoreInputSplit split,
+List columnNames,
+List selectedColumns)
Review Comment:
Add documentation here: Still output the original schema of the Row, the
role of `selectedColumns` is to read out the required fields, other fields to
fill Nulls.
##
flink-table-store-hive/flink-table-store-hive-connector/src/main/java/org/apache/flink/table/store/mapred/TableStoreRecordReader.java:
##
@@ -36,24 +41,44 @@ public class TableStoreRecordReader implements
RecordReader iterator;
private final long splitLength;
+@Nullable private final ProjectedRowData reusedProjectedRow;
+
private float progress;
public TableStoreRecordReader(
-org.apache.flink.table.store.file.utils.RecordReader
wrapped,
-long splitLength) {
-this.iterator = new RecordReaderIterator<>(wrapped);
-this.splitLength = splitLength;
+TableRead read,
+TableStoreInputSplit split,
+List columnNames,
+List selectedColumns)
+throws IOException {
+if (columnNames.equals(selectedColumns)) {
+reusedProjectedRow = null;
+} else {
+
read.withProjection(selectedColumns.stream().mapToInt(columnNames::indexOf).toArray());
+reusedProjectedRow =
+ProjectedRowData.from(
+
columnNames.stream().mapToInt(selectedColumns::indexOf).toArray());
+}
+
+this.iterator = new
RecordReaderIterator<>(read.createReader(split.split()));
+this.splitLength = split.getLength();
this.progress = 0;
}
@Override
public boolean next(Void key, RowDataContainer value) throws IOException {
RowData rowData = iterator.next();
+
if (rowData == null) {
progress = 1;
return false;
} else {
-value.set(rowData);
+if (reusedProjectedRow != null) {
+reusedProjectedRow.replaceRow(rowData);
+value.set(reusedProjectedRow);
Review Comment:
`value.set(reusedProjectedRow.replaceRow(rowData));`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (FLINK-28176) test_es_sink_dynamic failed in jdk11
[
https://issues.apache.org/jira/browse/FLINK-28176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo resolved FLINK-28176.
--
Resolution: Fixed
Merged into master via fa67c3b7072fb8d80d05e10b1703cff5700fcb39
> test_es_sink_dynamic failed in jdk11
>
>
> Key: FLINK-28176
> URL: https://issues.apache.org/jira/browse/FLINK-28176
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-06-21T03:01:35.4707985Z Jun 21 03:01:35 _
> FlinkElasticsearch7Test.test_es_sink_dynamic _
> 2022-06-21T03:01:35.4709206Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4710708Z Jun 21 03:01:35 self =
> testMethod=test_es_sink_dynamic>
> 2022-06-21T03:01:35.4711754Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4712481Z Jun 21 03:01:35 def
> test_es_sink_dynamic(self):
> 2022-06-21T03:01:35.4715653Z Jun 21 03:01:35 ds =
> self.env.from_collection(
> 2022-06-21T03:01:35.4718082Z Jun 21 03:01:35 [{'name': 'ada',
> 'id': '1'}, {'name': 'luna', 'id': '2'}],
> 2022-06-21T03:01:35.4719972Z Jun 21 03:01:35
> type_info=Types.MAP(Types.STRING(), Types.STRING()))
> 2022-06-21T03:01:35.4721209Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4722120Z Jun 21 03:01:35 > es_dynamic_index_sink =
> Elasticsearch7SinkBuilder() \
> 2022-06-21T03:01:35.4723876Z Jun 21 03:01:35
> .set_emitter(ElasticsearchEmitter.dynamic_index('name', 'id')) \
> 2022-06-21T03:01:35.4725448Z Jun 21 03:01:35
> .set_hosts(['localhost:9200']) \
> 2022-06-21T03:01:35.4726419Z Jun 21 03:01:35 .build()
> 2022-06-21T03:01:35.4727430Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4877335Z Jun 21 03:01:35
> pyflink/datastream/tests/test_connectors.py:132:
> 2022-06-21T03:01:35.4882723Z Jun 21 03:01:35 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2022-06-21T03:01:35.4884972Z Jun 21 03:01:35
> pyflink/datastream/connectors/elasticsearch.py:130: in set_hosts
> 2022-06-21T03:01:35.4886124Z Jun 21 03:01:35 j_http_hosts_array =
> to_jarray(JHttpHost, j_http_hosts_list)
> 2022-06-21T03:01:35.4887527Z Jun 21 03:01:35 pyflink/util/java_utils.py:37:
> in to_jarray
> 2022-06-21T03:01:35.4888600Z Jun 21 03:01:35 j_arr[i] = arr[i]
> 2022-06-21T03:01:35.4890812Z Jun 21 03:01:35
> .tox/py39-cython/lib/python3.9/site-packages/py4j/java_collections.py:238: in
> __setitem__
> 2022-06-21T03:01:35.4892201Z Jun 21 03:01:35 return self.__set_item(key,
> value)
> 2022-06-21T03:01:35.4893842Z Jun 21 03:01:35
> .tox/py39-cython/lib/python3.9/site-packages/py4j/java_collections.py:221: in
> __set_item
> 2022-06-21T03:01:35.4895153Z Jun 21 03:01:35 return
> get_return_value(answer, self._gateway_client)
> 2022-06-21T03:01:35.4896282Z Jun 21 03:01:35 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2022-06-21T03:01:35.4897191Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4900656Z Jun 21 03:01:35 answer =
> 'zsorg.apache.flink.api.python.shaded.py4j.Py4JException: Cannot convert
> org.apache.flink.elasticsearch7.shaded.org.ap...haded.py4j.GatewayConnection.run(GatewayConnection.java:238)\\n\tat
> java.base/java.lang.Thread.run(Thread.java:829)\\n'
> 2022-06-21T03:01:35.4903369Z Jun 21 03:01:35 gateway_client =
>
> 2022-06-21T03:01:35.4904543Z Jun 21 03:01:35 target_id = None, name = None
> 2022-06-21T03:01:35.4905404Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4906381Z Jun 21 03:01:35 def get_return_value(answer,
> gateway_client, target_id=None, name=None):
> 2022-06-21T03:01:35.4908583Z Jun 21 03:01:35 """Converts an answer
> received from the Java gateway into a Python object.
> 2022-06-21T03:01:35.4909687Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4910838Z Jun 21 03:01:35 For example, string
> representation of integers are converted to Python
> 2022-06-21T03:01:35.4912061Z Jun 21 03:01:35 integer, string
> representation of objects are converted to JavaObject
> 2022-06-21T03:01:35.4913137Z Jun 21 03:01:35 instances, etc.
> 2022-06-21T03:01:35.4913921Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4914859Z Jun 21 03:01:35 :param answer: the
> string returned by the Java gateway
> 2022-06-21T03:01:35.4916648Z Jun 21 03:01:35 :param gateway_client:
> the gateway client used to communicate with the Java
> 2022-06-21T03:01:35.4918294Z Jun 21 03:01:35 Gateway. Only
> necessary if the answer is a reference (e.g., object,
> 2022-06-21T03:01:35.4919591Z Jun 21 03:01:35 list, map)
> 2022-06-21T03:01:35.4920758Z Ju
[jira] [Assigned] (FLINK-28416) Add the new LookupFunction interface for lookup table source
[
https://issues.apache.org/jira/browse/FLINK-28416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren reassigned FLINK-28416:
-
Assignee: Qingsheng Ren
> Add the new LookupFunction interface for lookup table source
>
>
> Key: FLINK-28416
> URL: https://issues.apache.org/jira/browse/FLINK-28416
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Qingsheng Ren
>Assignee: Qingsheng Ren
>Priority: Major
> Labels: pull-request-available
>
> Add the new {{{}LookupFunction{}}}, {{AsyncLookupFunction}} and their related
> providers in place of the top-level {{TableFunction}} as the API for lookup
> table
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-28417) Add interface and default implementation for cache in lookup table
[
https://issues.apache.org/jira/browse/FLINK-28417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren reassigned FLINK-28417:
-
Assignee: Qingsheng Ren
> Add interface and default implementation for cache in lookup table
> --
>
> Key: FLINK-28417
> URL: https://issues.apache.org/jira/browse/FLINK-28417
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Qingsheng Ren
>Assignee: Qingsheng Ren
>Priority: Major
>
> Add interfaces for cache in lookup table, including {{{}LookupCache{}}}, a
> default implementation {{DefaultLookupCache}}, and cache related metric group
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] HuangXingBo closed pull request #20191: [FLINK-28176][python] Refactor the test in test_connectors.py
HuangXingBo closed pull request #20191: [FLINK-28176][python] Refactor the test in test_connectors.py URL: https://github.com/apache/flink/pull/20191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #182: [FLINK-27708] Add background compaction task for append-only table when ingesting
JingsongLi commented on code in PR #182:
URL: https://github.com/apache/flink-table-store/pull/182#discussion_r915444076
##
docs/layouts/shortcodes/generated/core_configuration.html:
##
@@ -36,13 +36,25 @@
compaction.max-size-amplification-percent
200
Integer
-The size amplification is defined as the amount (in
percentage) of additional storage needed to store a single byte of data in the
merge tree.
+The size amplification is defined as the amount (in
percentage) of additional storage needed to store a single byte of data in the
merge tree for changelog mode table.
+
+
+compaction.max.file-num
Review Comment:
maybe change a word?
If 100 small files are given directly to compaction, they will also be done
together, so that it is not understood as a typical max.
Maybe `compaction.early-max.file-num`?
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/CoreOptions.java:
##
@@ -214,17 +214,36 @@ public class CoreOptions implements Serializable {
.defaultValue(200)
.withDescription(
"The size amplification is defined as the amount
(in percentage) of additional storage "
-+ "needed to store a single byte of data
in the merge tree.");
++ "needed to store a single byte of data
in the merge tree for changelog mode table.");
public static final ConfigOption COMPACTION_SIZE_RATIO =
ConfigOptions.key("compaction.size-ratio")
.intType()
.defaultValue(1)
.withDescription(
-"Percentage flexibility while comparing sorted run
size. If the candidate sorted run(s) "
+"Percentage flexibility while comparing sorted run
size for changelog mode table. If the candidate sorted run(s) "
+ "size is 1% smaller than the next sorted
run's size, then include next sorted run "
+ "into this candidate set.");
+public static final ConfigOption COMPACTION_MIN_FILE_NUM =
+ConfigOptions.key("compaction.min.file-num")
+.intType()
+.defaultValue(4)
Review Comment:
5?
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/compact/CompactTask.java:
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.file.compact;
+
+import org.apache.flink.table.store.file.data.DataFileMeta;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.concurrent.Callable;
+
+/** Compact task. */
+public abstract class CompactTask implements Callable {
+
+private static final Logger LOG =
LoggerFactory.getLogger(CompactTask.class);
+
+// metrics
+protected long rewriteInputSize;
+protected long rewriteOutputSize;
+protected int rewriteFilesNum;
+
+public CompactTask() {
+this.rewriteInputSize = 0;
Review Comment:
We don't need this constructor.
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/compact/CompactTask.java:
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limit
[jira] [Comment Edited] (FLINK-28435) Flink version 1.15.0 consumption Kafka has backpressure.
[
https://issues.apache.org/jira/browse/FLINK-28435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563586#comment-17563586
]
HunterHunter edited comment on FLINK-28435 at 7/7/22 6:06 AM:
--
[~jark]
Can you help me find someone to have a look
was (Author: hunterhunter):
[~jark]
> Flink version 1.15.0 consumption Kafka has backpressure.
> -
>
> Key: FLINK-28435
> URL: https://issues.apache.org/jira/browse/FLINK-28435
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: HunterHunter
>Priority: Major
>
> A simple SQL program
> {code:java}
> CREATE temporary TABLE `print_sink` (
> `id` VARCHAR(2147483647)
> )WITH (
> 'connector' = 'print'
> );
> insert into print_sink
> select id
> from kafka-source
> /*+ OPTIONS('properties.bootstrap.servers'='brokers',
> 'topic' = 'topicname',
> 'properties.group.id'='test',
> 'scan.startup.mode'='latest-offset')
> */ {code}
> Back pressure occurs when I use `flink 1.15.0`,
> When I use `flink 1.13/ flink 1.14`, it works normally.
> I think this is caused by the version of Kafka.
> I have tried to consume versions `Kafka 1.1` and `Kafka 2.7.1`, and `Flink
> 1.13/1.14` is normal （`Kafka 2.4.1` is used in `Flink 1.13/1.14`）;
> But `Flink 1.15` has back pressure. (I had try to change Kafka version to
> `Kafka 2.7.1`, which is still back pressure).
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28435) Flink version 1.15.0 consumption Kafka has backpressure.
[
https://issues.apache.org/jira/browse/FLINK-28435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563586#comment-17563586
]
HunterHunter commented on FLINK-28435:
--
[~jark]
> Flink version 1.15.0 consumption Kafka has backpressure.
> -
>
> Key: FLINK-28435
> URL: https://issues.apache.org/jira/browse/FLINK-28435
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: HunterHunter
>Priority: Major
>
> A simple SQL program
> {code:java}
> CREATE temporary TABLE `print_sink` (
> `id` VARCHAR(2147483647)
> )WITH (
> 'connector' = 'print'
> );
> insert into print_sink
> select id
> from kafka-source
> /*+ OPTIONS('properties.bootstrap.servers'='brokers',
> 'topic' = 'topicname',
> 'properties.group.id'='test',
> 'scan.startup.mode'='latest-offset')
> */ {code}
> Back pressure occurs when I use `flink 1.15.0`,
> When I use `flink 1.13/ flink 1.14`, it works normally.
> I think this is caused by the version of Kafka.
> I have tried to consume versions `Kafka 1.1` and `Kafka 2.7.1`, and `Flink
> 1.13/1.14` is normal （`Kafka 2.4.1` is used in `Flink 1.13/1.14`）;
> But `Flink 1.15` has back pressure. (I had try to change Kafka version to
> `Kafka 2.7.1`, which is still back pressure).
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-28435) Flink version 1.15.0 consumption Kafka has backpressure.
HunterHunter created FLINK-28435:
Summary: Flink version 1.15.0 consumption Kafka has backpressure.
Key: FLINK-28435
URL: https://issues.apache.org/jira/browse/FLINK-28435
Project: Flink
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: HunterHunter
A simple SQL program
{code:java}
CREATE temporary TABLE `print_sink` (
`id` VARCHAR(2147483647)
)WITH (
'connector' = 'print'
);
insert into print_sink
select id
from kafka-source
/*+ OPTIONS('properties.bootstrap.servers'='brokers',
'topic' = 'topicname',
'properties.group.id'='test',
'scan.startup.mode'='latest-offset')
*/ {code}
Back pressure occurs when I use `flink 1.15.0`,
When I use `flink 1.13/ flink 1.14`, it works normally.
I think this is caused by the version of Kafka.
I have tried to consume versions `Kafka 1.1` and `Kafka 2.7.1`, and `Flink
1.13/1.14` is normal （`Kafka 2.4.1` is used in `Flink 1.13/1.14`）;
But `Flink 1.15` has back pressure. (I had try to change Kafka version to
`Kafka 2.7.1`, which is still back pressure).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] luoyuxia commented on pull request #19069: [FLINK-24862][Connectors / Hive][backport]Fix user-defined hive udaf/udtf cannot be used normally in hive dialect for flink1.14
luoyuxia commented on PR #19069: URL: https://github.com/apache/flink/pull/19069#issuecomment-1177119760 @hehuiyuan I think @beyond1920 can help merge when she is free. Just be patient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lpn666 closed pull request #20192: [sql-jdbc]
lpn666 closed pull request #20192: [sql-jdbc] URL: https://github.com/apache/flink/pull/20192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lpn666 commented on pull request #20192: [sql-jdbc]
lpn666 commented on PR #20192: URL: https://github.com/apache/flink/pull/20192#issuecomment-1177110023 > ## What is the purpose of the change > When I using the sql-jdbc to transform a big table from mysql to other database, the flink program load the entire table into memory. The source table is too big (16GB), and the taskmanager crashed. So What can I do, or what about add a new option to limit the speed of reading data (or batch the data ) > > ## Brief change log > ## Verifying this change > ## Does this pull request potentially affect one of the following parts: > ## Documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28431) CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager is restarted it fails to recover the job due to "checkpoint FileNotFound exception"
[ https://issues.apache.org/jira/browse/FLINK-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563584#comment-17563584 ] Yun Tang commented on FLINK-28431: -- [~aresyhzhang] Could you check the original JM logs to see whether checkpoint-1132 is subsumed by a newer completed checkpoint. I think that might be why checkpoint-1132 is deleted on HDFS. > CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager > is restarted it fails to recover the job due to "checkpoint FileNotFound > exception" > --- > > Key: FLINK-28431 > URL: https://issues.apache.org/jira/browse/FLINK-28431 > Project: Flink > Issue Type: Bug >Affects Versions: 1.13.2 > Environment: flink:1.13.2 > java:1.8 >Reporter: aresyhzhang >Priority: Major > Labels: checkpoint, ha, native-kubernetes > Attachments: error.log > > > We have built a lot of flink clusters in native Kubernetes session mode, > flink version 1.13.2, some clusters can run normally for 180 days and some > can run for 30 days. > The following takes an abnormal flink cluster > flink-k8s-session-opd-public-1132 as an example. > Problem Description: > Appears when jobmanager restarts > File does not exist: > /home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > The result of this is that the entire flink cluster cannot be started. > Because other tasks in session mode are also affected by the inability to > start, the impact is very serious. > Some auxiliary information: > 1. flink cluster id: flink-k8s-session-opd-public-1132 > 2. High-availability.storageDir of cluster configuration: > hdfs://neophdfsv2flink/home/flink/recovery/ > 3.error job id: 18193cde2c359f492f76c8ce4cd20271 > 4. There was a similar issue before: FLINK-8770, but I saw that it was closed > without being resolved. > 5. The complete jommanager log I have uploaded to the attachment > My investigation ideas: > 1. View the node information on the zookeeper corresponding to the jobid > 18193cde2c359f492f76c8ce4cd20271: > [zk: localhost:2181(CONNECTED) 17] ls > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271 > [0025852, 0025851] > [zk: localhost:2181(CONNECTED) 14] get > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271/0025852 > ??sr;org.apache.flink.runtime.state.RetrievableStreamStateHandle?U?+LwrappedStreamStateHandlet2Lorg/apache/flink/runtime/state/StreamStateHandle;xpsr9org.apache.flink.runtime.state.filesystem.FileStateHandle?u?b?J > > stateSizefilePathtLorg/apache/flink/core/fs/Path;xp??srorg.apache.flink.core.fs.PathLuritLjava/net/URI;xpsr > java.net.URI?x.C?I?LstringtLjava/lang/String;xptrhdfs://neophdfsv2flink/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4ax > cZxid = 0x1070932e2 > ctime = Wed Jul 06 02:28:51 UTC 2022 > mZxid = 0x1070932e2 > mtime = Wed Jul 06 02:28:51 UTC 2022 > pZxid = 0x30001c957 > cversion=222 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 545 > numChildren = 0. > I am sure that my zk node is normal, because there are 10+ flink clusters > using the same zk node, but only this cluster has problems, other clusters > are normal > 2. View the hdfs edits modification log of the directory corresponding to > hdfs: > ./hdfs-audit.log.1:2022-07-06 10:28:51,752 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= create > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=flinkuser:flinkuser:rw-r--r-- proto=rpc > ./hdfs-audit.log.1:2022-07-06 10:29:26,588 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= delete > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=null proto=rpc > I don't know why flink created the directory and then deleted it, and did not > update the metadata information to zookeeper, which caused the jobmanager to > restart without getting the correct directory and keep restarting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] lpn666 opened a new pull request, #20192: [sql-jdbc]
lpn666 opened a new pull request, #20192: URL: https://github.com/apache/flink/pull/20192 ## What is the purpose of the change When I using the sql-jdbc to transform a big table from mysql to other database, the flink program load the entire table into memory. The source table is too big (16GB), and the taskmanager crashed. So What can I do, or what about add a new option to limit the speed of reading data (or batch the data ) ## Brief change log ## Verifying this change ## Does this pull request potentially affect one of the following parts: ## Documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 commented on pull request #20061: [hotfix][flink-rpc] Fix the RemoteRpcInvocation class typo.
liuzhuang2017 commented on PR #20061: URL: https://github.com/apache/flink/pull/20061#issuecomment-1177101540 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28312) Introduce REST APIs for log URL retrieval
[ https://issues.apache.org/jira/browse/FLINK-28312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo closed FLINK-28312. -- Resolution: Fixed > Introduce REST APIs for log URL retrieval > - > > Key: FLINK-28312 > URL: https://issues.apache.org/jira/browse/FLINK-28312 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28312) Introduce REST APIs for log URL retrieval
[ https://issues.apache.org/jira/browse/FLINK-28312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563582#comment-17563582 ] Yangze Guo commented on FLINK-28312: master a99fc3ffbf95910dd8fad6833852ca5eed3f1896 > Introduce REST APIs for log URL retrieval > - > > Key: FLINK-28312 > URL: https://issues.apache.org/jira/browse/FLINK-28312 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] KarmaGYZ closed pull request #20179: [FLINK-28312][rest] Introduce REST APIs for log URL retrieval
KarmaGYZ closed pull request #20179: [FLINK-28312][rest] Introduce REST APIs for log URL retrieval URL: https://github.com/apache/flink/pull/20179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on pull request #20137: Just for CI
1996fanrui commented on PR #20137: URL: https://github.com/apache/flink/pull/20137#issuecomment-1177084314 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on pull request #19781: Just for test : output buffer aligned to uc
1996fanrui commented on PR #19781: URL: https://github.com/apache/flink/pull/19781#issuecomment-1177082931 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28297) Incorrect metric group order for namespaced operator metrics
[
https://issues.apache.org/jira/browse/FLINK-28297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563577#comment-17563577
]
Gyula Fora commented on FLINK-28297:
On a second thought we might simply want to get the resourcens tagname and keep
the ns always front. Let's discuss offiline first
> Incorrect metric group order for namespaced operator metrics
>
>
> Key: FLINK-28297
> URL: https://issues.apache.org/jira/browse/FLINK-28297
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Gyula Fora
>Assignee: Matyas Orhidi
>Priority: Major
> Fix For: kubernetes-operator-1.1.0
>
>
> The metric group for the resource namespace should follow the main metric
> group for the metric itself.
> So instead of
> {noformat}
> flink-kubernetes-operator-64d8cc77c4-w49nj.k8soperator.default.flink-kubernetes-operator.resourcens.default.FlinkDeployment.READY.Count:
> 1
> {noformat}
> we should have
> {noformat}
> flink-kubernetes-operator-64d8cc77c4-w49nj.k8soperator.default.flink-kubernetes-operator.FlinkDeployment.READY.resourcens.default.Count:
> 1{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28110) Table Store Hive Reader supports projection pushdown
[ https://issues.apache.org/jira/browse/FLINK-28110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28110: --- Labels: pull-request-available (was: ) > Table Store Hive Reader supports projection pushdown > > > Key: FLINK-28110 > URL: https://issues.apache.org/jira/browse/FLINK-28110 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > When the user declares fields in the DDL, we may not report an error when the > declared fields are incomplete, at this time we can assume that the user only > wants to read these fields, in fact, it is projection pushdown -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper opened a new pull request, #202: [FLINK-28110] Support projection pushdown for Hive readers
tsreaper opened a new pull request, #202: URL: https://github.com/apache/flink-table-store/pull/202 As Hive 2 only provides selected column names and requires the source not to change the schema, we implement projection pushdown by only reading selected columns from files and set the other columns to null. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sunshineJK commented on a diff in pull request #20127: [FLINK-26270] Flink SQL write data to kafka by CSV format , whether d…
sunshineJK commented on code in PR #20127:
URL: https://github.com/apache/flink/pull/20127#discussion_r915458758
##
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatOptions.java:
##
@@ -87,5 +87,11 @@ public class CsvFormatOptions {
"Optional null literal string that is interpreted
as a\n"
+ "null value (disabled by default)");
+public static final ConfigOption WRITE_BIGDECIMAL_AS_PLAIN =
+ConfigOptions.key("write-bigdecimal-as-plain")
+.booleanType()
+.defaultValue(true)
Review Comment:
The current default value is equal to "standard" Jackson
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] sunshineJK commented on a diff in pull request #20127: [FLINK-26270] Flink SQL write data to kafka by CSV format , whether d…
sunshineJK commented on code in PR #20127:
URL: https://github.com/apache/flink/pull/20127#discussion_r915458318
##
flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvFormatOptions.java:
##
@@ -87,5 +87,11 @@ public class CsvFormatOptions {
"Optional null literal string that is interpreted
as a\n"
+ "null value (disabled by default)");
+public static final ConfigOption WRITE_BIGDECIMAL_AS_PLAIN =
+ConfigOptions.key("write-bigdecimal-as-plain")
+.booleanType()
+.defaultValue(true)
+.withDescription("Optional whether write bigdecimal using
scientific notation");
Review Comment:
Thank you for your advice,I've made some updates
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-28393) Support AvroInputFormat in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-28393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-28393. --- Assignee: Juntao Hu Resolution: Fixed Merged to master via bbd5a813e9c6a17b0c3bb31ee27d92d00448c956 > Support AvroInputFormat in PyFlink > -- > > Key: FLINK-28393 > URL: https://issues.apache.org/jira/browse/FLINK-28393 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: shaded-7.0 >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28393) Support AvroInputFormat in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-28393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-28393: Affects Version/s: (was: shaded-7.0) > Support AvroInputFormat in PyFlink > -- > > Key: FLINK-28393 > URL: https://issues.apache.org/jira/browse/FLINK-28393 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] dianfu closed pull request #20163: [FLINK-28393][python][format] Support AvroInputFormat
dianfu closed pull request #20163: [FLINK-28393][python][format] Support AvroInputFormat URL: https://github.com/apache/flink/pull/20163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on a diff in pull request #20118: [FLINK-28334][table-planner] Fix PushProjectIntoTableSourceScanRule: covers the case when table source SupportsReadingMetadata an
godfreyhe commented on code in PR #20118:
URL: https://github.com/apache/flink/pull/20118#discussion_r915444008
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.java:
##
@@ -337,11 +339,71 @@ private RowType performPushDown(
private List rewriteProjections(
RelOptRuleCall call, TableSourceTable source, NestedSchema
projectedSchema) {
final LogicalProject project = call.rel(0);
+List newProjects = project.getProjects();
+
if (supportsProjectionPushDown(source.tableSource())) {
-return NestedProjectionUtil.rewrite(
-project.getProjects(), projectedSchema,
call.builder().getRexBuilder());
-} else {
-return project.getProjects();
+// if support project push down, then all input ref will be
rewritten includes metadata
+// columns.
+newProjects =
+NestedProjectionUtil.rewrite(
+newProjects, projectedSchema,
call.builder().getRexBuilder());
+} else if (supportsMetadata(source.tableSource())) {
+// supportsMetadataProjection only.
+// Note: why not reuse the NestedProjectionUtil to rewrite
metadata projection? because
+// it only works for sources which support projection push down.
+List metadataColumns =
+DynamicSourceUtils.extractMetadataColumns(
+source.contextResolvedTable().getResolvedSchema());
+if (metadataColumns.size() > 0) {
+Set metaCols =
+metadataColumns.stream().map(m ->
m.getName()).collect(Collectors.toSet());
+
+MetadataOnlyProjectionRewriter rewriter =
+new MetadataOnlyProjectionRewriter(
+project.getInput().getRowType(),
source.getRowType(), metaCols);
+
+newProjects =
+newProjects.stream()
+.map(p -> p.accept(rewriter))
+.collect(Collectors.toList());
+}
+}
+
+return newProjects;
+}
+
+private class MetadataOnlyProjectionRewriter extends RexShuttle {
Review Comment:
nit: This class can be marked as `static`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-27759) Rethink how to get the git commit id for docker image in Flink Kubernetes operator
[ https://issues.apache.org/jira/browse/FLINK-27759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563565#comment-17563565 ] Yang Wang commented on FLINK-27759: --- This is still not fixed. But it is not a high-priority ticket. We could revisit it later if necessary. > Rethink how to get the git commit id for docker image in Flink Kubernetes > operator > -- > > Key: FLINK-27759 > URL: https://issues.apache.org/jira/browse/FLINK-27759 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Yang Wang >Assignee: Nicholas Jiang >Priority: Minor > > Follow the discussion in the PR[1][2], we need to rethink how the get the git > commit id properly. Currently, we rely on the .git directory. And it is a > problem when building image from source release. > > [1]. [https://github.com/apache/flink-kubernetes-operator/pull/243] > [2]. https://github.com/apache/flink-kubernetes-operator/pull/241 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28110) Table Store Hive Reader supports projection pushdown
[ https://issues.apache.org/jira/browse/FLINK-28110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-28110: Assignee: Caizhi Weng > Table Store Hive Reader supports projection pushdown > > > Key: FLINK-28110 > URL: https://issues.apache.org/jira/browse/FLINK-28110 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Assignee: Caizhi Weng >Priority: Major > Fix For: table-store-0.2.0 > > > When the user declares fields in the DDL, we may not report an error when the > declared fields are incomplete, at this time we can assume that the user only > wants to read these fields, in fact, it is projection pushdown -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28289) Introduce Spark2 Reader for table store
[ https://issues.apache.org/jira/browse/FLINK-28289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-28289: Assignee: Nicholas Jiang > Introduce Spark2 Reader for table store > --- > > Key: FLINK-28289 > URL: https://issues.apache.org/jira/browse/FLINK-28289 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Assignee: Nicholas Jiang >Priority: Major > Fix For: table-store-0.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28040) Introduce Trino reader for table store
[ https://issues.apache.org/jira/browse/FLINK-28040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28040. Resolution: Won't Fix > Introduce Trino reader for table store > -- > > Key: FLINK-28040 > URL: https://issues.apache.org/jira/browse/FLINK-28040 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.2.0 > > > Can refer to FLINK-27947 to write a Trino reader. > See https://trino.io/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa commented on a diff in pull request #19960: [FLINK-27907][runtime] implement disk read and write logic for hybrid shuffle
reswqa commented on code in PR #19960:
URL: https://github.com/apache/flink/pull/19960#discussion_r915441265
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionFileReader.java:
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferRecycler;
+import org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.util.Deque;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingDeque;
+
+import static
org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil.positionToNextBuffer;
+import static
org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil.readFromByteChannel;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This component is responsible for reading data from disk for a specific
subpartition.
+ *
+ * In order to access the disk as sequentially as possible {@link
HsSubpartitionFileReader} need
+ * to be able to compare priorities.
+ *
+ * Note: This class is not thread safe.
+ */
+public class HsSubpartitionFileReader implements SubpartitionFileReader {
+
+private final ByteBuffer headerBuf =
BufferReaderWriterUtil.allocatedHeaderBuffer();
+
+private final int subpartitionId;
+
+private final FileChannel dataFileChannel;
+
+private final HsSubpartitionViewInternalOperations operations;
+
+private final CachedRegionManager cachedRegionManager;
+
+private final BufferIndexManager bufferIndexManager;
+
+private final Deque loadedBuffers = new
LinkedBlockingDeque<>();
+
+private volatile boolean isFailed;
+
+public HsSubpartitionFileReader(
+int subpartitionId,
+FileChannel dataFileChannel,
+HsSubpartitionViewInternalOperations operations,
+HsFileDataIndex dataIndex,
+int maxBufferReadAhead) {
+this.subpartitionId = subpartitionId;
+this.dataFileChannel = dataFileChannel;
+this.operations = operations;
+this.bufferIndexManager = new BufferIndexManager(maxBufferReadAhead);
+this.cachedRegionManager = new CachedRegionManager(subpartitionId,
dataIndex);
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+HsSubpartitionFileReader that = (HsSubpartitionFileReader) o;
+return subpartitionId == that.subpartitionId;
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(subpartitionId);
+}
+
+/**
+ * Read subpartition data into buffers.
+ *
+ * This transfers the ownership of used buffers to this class. It's
this class'
+ * responsibility to release the buffers using the recycler when no longer
needed.
+ *
+ * Calling this method does not always use up all the provided buffers.
It's this class'
+ * decision when to stop reading. Currently, it stops reading when: 1)
buffers are used up, or
+ * 2) reaches the end of the subpartition data within the region, or 3)
enough data have been
+ * read ahead the downstream consuming offset.
+ */
+@Override
+public synchronized void readBuffers(Queue buffers,
BufferRecycler recycler)
+throws IOException {
+if (isFailed) {
+throw new IOException("subpartition reader has already failed.");
+}
+int firstBufferToLoad = bufferIndexManager.getNextToLoad();
+if (firstBufferToLoad < 0) {
+ret
[GitHub] [flink] HuangZhenQiu commented on a diff in pull request #20001: [FLINK-27659][table] Planner support to use jar which is registered by 'CREATE FUNTION USING JAR' syntax
HuangZhenQiu commented on code in PR #20001:
URL: https://github.com/apache/flink/pull/20001#discussion_r915435030
##
flink-table/flink-sql-client/src/main/java/org/apache/flink/util/ClientMutableURLClassLoader.java:
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.util;
+
+import org.apache.flink.configuration.Configuration;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class loader extands {@link MutableURLClassLoader}, upon the {@code
addURL} method, it also
+ * exposes a {@code removeURL} method which used to remove unnecessary jar
from current classloader
+ * path. This class loader wraps a {@link MutableURLClassLoader} and an old
classloader list, the
+ * class load is delegated to the inner {@link MutableURLClassLoader}.
+ *
+ * This is only used to SqlClient for supporting {@code REMOVE JAR} clause
currently. When remove
+ * a jar, get the registered jar url list from current {@link
MutableURLClassLoader} firstly, then
+ * create a new instance of {@link MutableURLClassLoader} which urls doesn't
include the removed
+ * jar, and the currentClassLoader point to new instance object, the old
object is added to list to
+ * be closed when close {@link ClientMutableURLClassLoader}.
+ *
+ * Note: This classloader is not guaranteed to actually remove class or
resource, any classes or
+ * resources in the removed jar that are already loaded, are still accessible.
+ */
+public class ClientMutableURLClassLoader extends MutableURLClassLoader {
+
+private static final Logger LOG =
LoggerFactory.getLogger(ClientMutableURLClassLoader.class);
+
+static {
+ClassLoader.registerAsParallelCapable();
+}
+
+private final Configuration configuration;
Review Comment:
+1
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java:
##
@@ -822,6 +843,19 @@ private TableResultInternal
executeQueryOperation(QueryOperation operation) {
List> transformations =
translate(Collections.singletonList(sinkOperation));
final String defaultJobName = "collect";
+
+// Merge user jars to table configuration
+if (!resourceManager.getJarResourceURLs().isEmpty()) {
Review Comment:
move it to a private function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #20176: [FLINK-27660][table] add table api for registering function with resource
HuangZhenQiu commented on PR #20176: URL: https://github.com/apache/flink/pull/20176#issuecomment-1177020994 @lsyldliu Sure. I will rebase later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
[
https://issues.apache.org/jira/browse/FLINK-28291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiulong.zhu updated FLINK-28291:
Labels: (was: patch)
> Add kerberos delegation token renewer feature instead of logged from keytab
> individually
>
>
> Key: FLINK-28291
> URL: https://issues.apache.org/jira/browse/FLINK-28291
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
>Affects Versions: 1.13.5
>Reporter: jiulong.zhu
>Priority: Minor
> Fix For: 1.13.5
>
> Attachments: FLINK-28291.0001.patch
>
>
> h2. 1. Design
> LifeCycle of delegation token in RM:
> # Container starts with DT given by client.
> # Enable delegation token renewer by:
> ## set {{security.kerberos.token.renew.enabled}} true, default false. And
> ## specify {{security.kerberos.login.keytab}} and
> {{security.kerberos.login.principal}}
> # When enabled delegation token renewer, the renewer thread will re-obtain
> tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider
> now). Then the renewer thread will broadcast new tokens to RM locally, all
> JMs and all TMs by RPCGateway.
> # RM process adds new tokens in context by UserGroupInformation.
> LifeCycle of delegation token in JM / TM:
> # TaskManager starts with keytab stored in remote hdfs.
> # When registered successfully, JM / TM get the current tokens of RM boxed
> by {{JobMasterRegistrationSuccess}} / {{{}TaskExecutorRegistrationSuccess{}}}.
> # JM / TM process add new tokens in context by UserGroupInformation.
> It’s too heavy and unnecessary to retrieval leader of ResourceManager by
> HAService, so DelegationTokenManager is instanced by ResourceManager. So
> DelegationToken can hold the reference of ResourceManager, instead of RM
> RPCGateway or self gateway.
> h2. 2. Test
> # No local junit test. It’s too heavy to build junit environments including
> KDC and local hadoop.
> # Cluster test
> step 1: Specify krb5.conf with short token lifetime(ticket_lifetime,
> renew_lifetime) when submitting flink application.
> ```
> {{flink run -yD security.kerberos.token.renew.enabled=true -yD
> security.kerberos.krb5-conf.path= /home/work/krb5.conf -yD
> security.kerberos.login.use-ticket-cache=false ...}}
> ```
> step 2: Watch token identifier changelog and synchronizer between rm and
> worker.
> >>
> In RM / JM log,
> 2022-06-28 15:13:03,509 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> New token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy)
> created in KerberosDelegationToken, and next schedule delay is 64799880 ms.
> 2022-06-28 15:13:03,529 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user. 2022-06-28 15:13:04,729 INFO
> org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token
> (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) from RM.
> …
> 2022-06-29 09:13:03,732 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> New token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy)
> created in KerberosDelegationToken, and next schedule delay is 64800045 ms.
> 2022-06-29 09:13:03,805 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
> 2022-06-29 09:13:03,806 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on
> ha-hdfs:newfyyy) from RM.
> >>
> In TM log,
> 2022-06-28 15:13:17,983 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> TaskManager receives new token (HDFS_DELEGATION_TOKEN token 52101 for work on
> ha-hdfs:newfyyy) from RM.
> 2022-06-28 15:13:18,016 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
> …
> 2022-06-29 09:13:03,809 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> TaskManager receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on
> ha-hdfs:newfyyy) from RM.
> 2022-06-29 09:13:03,836 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
[
https://issues.apache.org/jira/browse/FLINK-28291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiulong.zhu updated FLINK-28291:
Labels: PatchAvailable patch-available (was: )
> Add kerberos delegation token renewer feature instead of logged from keytab
> individually
>
>
> Key: FLINK-28291
> URL: https://issues.apache.org/jira/browse/FLINK-28291
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
>Affects Versions: 1.13.5
>Reporter: jiulong.zhu
>Priority: Minor
> Labels: PatchAvailable, patch-available
> Fix For: 1.13.5
>
> Attachments: FLINK-28291.0001.patch
>
>
> h2. 1. Design
> LifeCycle of delegation token in RM:
> # Container starts with DT given by client.
> # Enable delegation token renewer by:
> ## set {{security.kerberos.token.renew.enabled}} true, default false. And
> ## specify {{security.kerberos.login.keytab}} and
> {{security.kerberos.login.principal}}
> # When enabled delegation token renewer, the renewer thread will re-obtain
> tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider
> now). Then the renewer thread will broadcast new tokens to RM locally, all
> JMs and all TMs by RPCGateway.
> # RM process adds new tokens in context by UserGroupInformation.
> LifeCycle of delegation token in JM / TM:
> # TaskManager starts with keytab stored in remote hdfs.
> # When registered successfully, JM / TM get the current tokens of RM boxed
> by {{JobMasterRegistrationSuccess}} / {{{}TaskExecutorRegistrationSuccess{}}}.
> # JM / TM process add new tokens in context by UserGroupInformation.
> It’s too heavy and unnecessary to retrieval leader of ResourceManager by
> HAService, so DelegationTokenManager is instanced by ResourceManager. So
> DelegationToken can hold the reference of ResourceManager, instead of RM
> RPCGateway or self gateway.
> h2. 2. Test
> # No local junit test. It’s too heavy to build junit environments including
> KDC and local hadoop.
> # Cluster test
> step 1: Specify krb5.conf with short token lifetime(ticket_lifetime,
> renew_lifetime) when submitting flink application.
> ```
> {{flink run -yD security.kerberos.token.renew.enabled=true -yD
> security.kerberos.krb5-conf.path= /home/work/krb5.conf -yD
> security.kerberos.login.use-ticket-cache=false ...}}
> ```
> step 2: Watch token identifier changelog and synchronizer between rm and
> worker.
> >>
> In RM / JM log,
> 2022-06-28 15:13:03,509 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> New token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy)
> created in KerberosDelegationToken, and next schedule delay is 64799880 ms.
> 2022-06-28 15:13:03,529 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user. 2022-06-28 15:13:04,729 INFO
> org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token
> (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) from RM.
> …
> 2022-06-29 09:13:03,732 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> New token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy)
> created in KerberosDelegationToken, and next schedule delay is 64800045 ms.
> 2022-06-29 09:13:03,805 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
> 2022-06-29 09:13:03,806 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on
> ha-hdfs:newfyyy) from RM.
> >>
> In TM log,
> 2022-06-28 15:13:17,983 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> TaskManager receives new token (HDFS_DELEGATION_TOKEN token 52101 for work on
> ha-hdfs:newfyyy) from RM.
> 2022-06-28 15:13:18,016 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
> …
> 2022-06-29 09:13:03,809 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> TaskManager receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on
> ha-hdfs:newfyyy) from RM.
> 2022-06-29 09:13:03,836 INFO org.apache.flink.runtime.util.HadoopUtils [] -
> Updating delegation tokens for current user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28433) Allow connection to mysql through mariadb driver
[ https://issues.apache.org/jira/browse/FLINK-28433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563557#comment-17563557 ] bo zhao commented on FLINK-28433: - Hi guys, I'm working on this now. Thanks > Allow connection to mysql through mariadb driver > > > Key: FLINK-28433 > URL: https://issues.apache.org/jira/browse/FLINK-28433 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: PengLei >Priority: Minor > Attachments: image-2022-07-07-09-29-06-834.png > > > Flink connector support connection to mysql. But the url must be started with > "jdbc:mysql". > Some user need to use mariadb dirver to connect to mysql. It can be achieved > by setting the driver parameter in jdbcOptions. Unfortunately， the url > verification fails. > > as follows: > !image-2022-07-07-09-29-06-834.png! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] lsyldliu commented on pull request #20176: [FLINK-27660][table] add table api for registering function with resource
lsyldliu commented on PR #20176: URL: https://github.com/apache/flink/pull/20176#issuecomment-1177006660 @HuangZhenQiu After https://github.com/apache/flink/pull/20001 merge, you can rebase master, and I will review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hehuiyuan commented on pull request #19069: [FLINK-24862][Connectors / Hive][backport]Fix user-defined hive udaf/udtf cannot be used normally in hive dialect for flink1.14
hehuiyuan commented on PR #19069: URL: https://github.com/apache/flink/pull/19069#issuecomment-1177000574 Hi @luoyuxia , who can merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hehuiyuan commented on pull request #19069: [FLINK-24862][Connectors / Hive][backport]Fix user-defined hive udaf/udtf cannot be used normally in hive dialect for flink1.14
hehuiyuan commented on PR #19069: URL: https://github.com/apache/flink/pull/19069#issuecomment-1176999775 Hi @beyond1920 , who will merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JesseAtSZ commented on pull request #20091: [FLINK-27570][runtime] Fix initialize base locations for checkpoint
JesseAtSZ commented on PR #20091: URL: https://github.com/apache/flink/pull/20091#issuecomment-1176999771 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26848) JDBC don't flush data when disable flush-max-rows and flush-interval
[
https://issues.apache.org/jira/browse/FLINK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563552#comment-17563552
]
hehuiyuan commented on FLINK-26848:
---
Hi [~jark] , have time to look at this issues?
> JDBC don't flush data when disable flush-max-rows and flush-interval
>
>
> Key: FLINK-26848
> URL: https://issues.apache.org/jira/browse/FLINK-26848
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Reporter: hehuiyuan
>Assignee: hehuiyuan
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Attachments: image-2022-03-24-22-05-23-717.png
>
>
> ||h5. sink.buffer-flush.interval||The flush interval mills, over this time,
> asynchronous threads will flush data. Can be set to {{'0'}} to disable it.
> Note, {{'sink.buffer-flush.max-rows'}} can be set to {{'0'}} with the flush
> interval set allowing for complete async processing of buffered actions.||
> |h5. sink.buffer-flush.max-rows|The max size of buffered records before
> flush. Can be set to zero to disable it.|
>
> h5. `Asynchronous thread flush` is disabled when set
> `sink.buffer-flush.interval` = 0, then disable `sink.buffer-flush.max-rows`,
> which causes the data is not written out.
> The data are not written when disable `flush.interval` and `max-rows`. I
> think it should write out when the data arrive.
> !image-2022-03-24-22-05-23-717.png|width=740,height=360!
>
> h5.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] xintongsong commented on a diff in pull request #19960: [FLINK-27907][runtime] implement disk read and write logic for hybrid shuffle
xintongsong commented on code in PR #19960:
URL: https://github.com/apache/flink/pull/19960#discussion_r915383242
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/SubpartitionFileReader.java:
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.buffer.BufferRecycler;
+
+import java.io.IOException;
+import java.nio.channels.FileChannel;
+import java.util.Queue;
+
+public interface SubpartitionFileReader extends
Comparable {
Review Comment:
We should name this interface `HsSubpartitionFileReader`, and the
implementation class `HsSubpartitionFileReaderImpl`.
##
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionFileReader.java:
##
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferRecycler;
+import org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.util.Deque;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingDeque;
+
+import static
org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil.positionToNextBuffer;
+import static
org.apache.flink.runtime.io.network.partition.BufferReaderWriterUtil.readFromByteChannel;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This component is responsible for reading data from disk for a specific
subpartition.
+ *
+ * In order to access the disk as sequentially as possible {@link
HsSubpartitionFileReader} need
+ * to be able to compare priorities.
+ *
+ * Note: This class is not thread safe.
+ */
+public class HsSubpartitionFileReader implements SubpartitionFileReader {
+
+private final ByteBuffer headerBuf =
BufferReaderWriterUtil.allocatedHeaderBuffer();
+
+private final int subpartitionId;
+
+private final FileChannel dataFileChannel;
+
+private final HsSubpartitionViewInternalOperations operations;
+
+private final CachedRegionManager cachedRegionManager;
+
+private final BufferIndexManager bufferIndexManager;
+
+private final Deque loadedBuffers = new
LinkedBlockingDeque<>();
+
+private volatile boolean isFailed;
+
+public HsSubpartitionFileReader(
+int subpartitionId,
+FileChannel dataFileChannel,
+HsSubpartitionViewInternalOperations operations,
+HsFileDataIndex dataIndex,
+int maxBufferReadAhead) {
+this.subpartitionId = subpartitionId;
+this.dataFileChannel = dataFileChannel;
+this.operations = operations;
+this.bufferIndexManager = new BufferIndexManager(maxBufferReadAhead);
+this.cachedRegionManager = new CachedReg
[GitHub] [flink] flinkbot commented on pull request #20191: [FLINK-28176][python] Refactor the test in test_connectors.py
flinkbot commented on PR #20191: URL: https://github.com/apache/flink/pull/20191#issuecomment-1176985873 ## CI report: * 08c0ada2f1fe1af11deb8c54dac06e80579b5c1d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.
[
https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563546#comment-17563546
]
Yun Tang commented on FLINK-28010:
--
[~Ming Li] Since we do not have a special separator between the primary key and
user key, we need to use the next byte array to represent the prefix bytes. If
so, I think we can make this.
> Use deleteRange to optimize the clear operation of RocksDBMapState.
> ---
>
> Key: FLINK-28010
> URL: https://issues.apache.org/jira/browse/FLINK-28010
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: ming li
>Priority: Major
>
> [FLINK-21321|https://issues.apache.org/jira/browse/FLINK-21321] has
> introduced {{deleteRange}} for fast clipping of Incremental checkpoint, so
> can the {{clear}} method in {{RocksDBMapState}} be replaced with
> {{{}deleteRange{}}}?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28176) test_es_sink_dynamic failed in jdk11
[
https://issues.apache.org/jira/browse/FLINK-28176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-28176:
---
Labels: pull-request-available test-stability (was: test-stability)
> test_es_sink_dynamic failed in jdk11
>
>
> Key: FLINK-28176
> URL: https://issues.apache.org/jira/browse/FLINK-28176
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-06-21T03:01:35.4707985Z Jun 21 03:01:35 _
> FlinkElasticsearch7Test.test_es_sink_dynamic _
> 2022-06-21T03:01:35.4709206Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4710708Z Jun 21 03:01:35 self =
> testMethod=test_es_sink_dynamic>
> 2022-06-21T03:01:35.4711754Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4712481Z Jun 21 03:01:35 def
> test_es_sink_dynamic(self):
> 2022-06-21T03:01:35.4715653Z Jun 21 03:01:35 ds =
> self.env.from_collection(
> 2022-06-21T03:01:35.4718082Z Jun 21 03:01:35 [{'name': 'ada',
> 'id': '1'}, {'name': 'luna', 'id': '2'}],
> 2022-06-21T03:01:35.4719972Z Jun 21 03:01:35
> type_info=Types.MAP(Types.STRING(), Types.STRING()))
> 2022-06-21T03:01:35.4721209Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4722120Z Jun 21 03:01:35 > es_dynamic_index_sink =
> Elasticsearch7SinkBuilder() \
> 2022-06-21T03:01:35.4723876Z Jun 21 03:01:35
> .set_emitter(ElasticsearchEmitter.dynamic_index('name', 'id')) \
> 2022-06-21T03:01:35.4725448Z Jun 21 03:01:35
> .set_hosts(['localhost:9200']) \
> 2022-06-21T03:01:35.4726419Z Jun 21 03:01:35 .build()
> 2022-06-21T03:01:35.4727430Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4877335Z Jun 21 03:01:35
> pyflink/datastream/tests/test_connectors.py:132:
> 2022-06-21T03:01:35.4882723Z Jun 21 03:01:35 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2022-06-21T03:01:35.4884972Z Jun 21 03:01:35
> pyflink/datastream/connectors/elasticsearch.py:130: in set_hosts
> 2022-06-21T03:01:35.4886124Z Jun 21 03:01:35 j_http_hosts_array =
> to_jarray(JHttpHost, j_http_hosts_list)
> 2022-06-21T03:01:35.4887527Z Jun 21 03:01:35 pyflink/util/java_utils.py:37:
> in to_jarray
> 2022-06-21T03:01:35.4888600Z Jun 21 03:01:35 j_arr[i] = arr[i]
> 2022-06-21T03:01:35.4890812Z Jun 21 03:01:35
> .tox/py39-cython/lib/python3.9/site-packages/py4j/java_collections.py:238: in
> __setitem__
> 2022-06-21T03:01:35.4892201Z Jun 21 03:01:35 return self.__set_item(key,
> value)
> 2022-06-21T03:01:35.4893842Z Jun 21 03:01:35
> .tox/py39-cython/lib/python3.9/site-packages/py4j/java_collections.py:221: in
> __set_item
> 2022-06-21T03:01:35.4895153Z Jun 21 03:01:35 return
> get_return_value(answer, self._gateway_client)
> 2022-06-21T03:01:35.4896282Z Jun 21 03:01:35 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> 2022-06-21T03:01:35.4897191Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4900656Z Jun 21 03:01:35 answer =
> 'zsorg.apache.flink.api.python.shaded.py4j.Py4JException: Cannot convert
> org.apache.flink.elasticsearch7.shaded.org.ap...haded.py4j.GatewayConnection.run(GatewayConnection.java:238)\\n\tat
> java.base/java.lang.Thread.run(Thread.java:829)\\n'
> 2022-06-21T03:01:35.4903369Z Jun 21 03:01:35 gateway_client =
>
> 2022-06-21T03:01:35.4904543Z Jun 21 03:01:35 target_id = None, name = None
> 2022-06-21T03:01:35.4905404Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4906381Z Jun 21 03:01:35 def get_return_value(answer,
> gateway_client, target_id=None, name=None):
> 2022-06-21T03:01:35.4908583Z Jun 21 03:01:35 """Converts an answer
> received from the Java gateway into a Python object.
> 2022-06-21T03:01:35.4909687Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4910838Z Jun 21 03:01:35 For example, string
> representation of integers are converted to Python
> 2022-06-21T03:01:35.4912061Z Jun 21 03:01:35 integer, string
> representation of objects are converted to JavaObject
> 2022-06-21T03:01:35.4913137Z Jun 21 03:01:35 instances, etc.
> 2022-06-21T03:01:35.4913921Z Jun 21 03:01:35
> 2022-06-21T03:01:35.4914859Z Jun 21 03:01:35 :param answer: the
> string returned by the Java gateway
> 2022-06-21T03:01:35.4916648Z Jun 21 03:01:35 :param gateway_client:
> the gateway client used to communicate with the Java
> 2022-06-21T03:01:35.4918294Z Jun 21 03:01:35 Gateway. Only
> necessary if the answer is a reference (e.g., object,
> 2022-06-21T03:01:35.4919591Z Jun 21 03:01:35 list, map)
> 2022-06-21T03:01:35.4920758Z Jun 21 03:01:3
[GitHub] [flink] HuangXingBo opened a new pull request, #20191: [FLINK-28176][python] Refactor the test in test_connectors.py
HuangXingBo opened a new pull request, #20191: URL: https://github.com/apache/flink/pull/20191 ## What is the purpose of the change *This pull request will refactor the test in `test_connectors.py`* ## Brief change log - *Refactor the test in test_connectors.py* ## Verifying this change This change added tests and can be verified as follows: - *Original tests* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28431) CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager is restarted it fails to recover the job due to "checkpoint FileNotFound exception"
[ https://issues.apache.org/jira/browse/FLINK-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aresyhzhang updated FLINK-28431: Attachment: error.log > CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager > is restarted it fails to recover the job due to "checkpoint FileNotFound > exception" > --- > > Key: FLINK-28431 > URL: https://issues.apache.org/jira/browse/FLINK-28431 > Project: Flink > Issue Type: Bug >Affects Versions: 1.13.2 > Environment: flink:1.13.2 > java:1.8 >Reporter: aresyhzhang >Priority: Major > Labels: checkpoint, ha, native-kubernetes > Attachments: error.log > > > We have built a lot of flink clusters in native Kubernetes session mode, > flink version 1.13.2, some clusters can run normally for 180 days and some > can run for 30 days. > The following takes an abnormal flink cluster > flink-k8s-session-opd-public-1132 as an example. > Problem Description: > Appears when jobmanager restarts > File does not exist: > /home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > The result of this is that the entire flink cluster cannot be started. > Because other tasks in session mode are also affected by the inability to > start, the impact is very serious. > Some auxiliary information: > 1. flink cluster id: flink-k8s-session-opd-public-1132 > 2. High-availability.storageDir of cluster configuration: > hdfs://neophdfsv2flink/home/flink/recovery/ > 3.error job id: 18193cde2c359f492f76c8ce4cd20271 > 4. There was a similar issue before: FLINK-8770, but I saw that it was closed > without being resolved. > 5. The complete jommanager log I have uploaded to the attachment > My investigation ideas: > 1. View the node information on the zookeeper corresponding to the jobid > 18193cde2c359f492f76c8ce4cd20271: > [zk: localhost:2181(CONNECTED) 17] ls > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271 > [0025852, 0025851] > [zk: localhost:2181(CONNECTED) 14] get > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271/0025852 > ??sr;org.apache.flink.runtime.state.RetrievableStreamStateHandle?U?+LwrappedStreamStateHandlet2Lorg/apache/flink/runtime/state/StreamStateHandle;xpsr9org.apache.flink.runtime.state.filesystem.FileStateHandle?u?b?J > > stateSizefilePathtLorg/apache/flink/core/fs/Path;xp??srorg.apache.flink.core.fs.PathLuritLjava/net/URI;xpsr > java.net.URI?x.C?I?LstringtLjava/lang/String;xptrhdfs://neophdfsv2flink/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4ax > cZxid = 0x1070932e2 > ctime = Wed Jul 06 02:28:51 UTC 2022 > mZxid = 0x1070932e2 > mtime = Wed Jul 06 02:28:51 UTC 2022 > pZxid = 0x30001c957 > cversion=222 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 545 > numChildren = 0. > I am sure that my zk node is normal, because there are 10+ flink clusters > using the same zk node, but only this cluster has problems, other clusters > are normal > 2. View the hdfs edits modification log of the directory corresponding to > hdfs: > ./hdfs-audit.log.1:2022-07-06 10:28:51,752 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= create > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=flinkuser:flinkuser:rw-r--r-- proto=rpc > ./hdfs-audit.log.1:2022-07-06 10:29:26,588 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= delete > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=null proto=rpc > I don't know why flink created the directory and then deleted it, and did not > update the metadata information to zookeeper, which caused the jobmanager to > restart without getting the correct directory and keep restarting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] sunshineJK commented on a diff in pull request #20127: [FLINK-26270] Flink SQL write data to kafka by CSV format , whether d…
sunshineJK commented on code in PR #20127: URL: https://github.com/apache/flink/pull/20127#discussion_r915399942 ## docs/content.zh/docs/connectors/table/formats/csv.md: ## @@ -3,7 +3,7 @@ title: CSV weight: 2 type: docs aliases: - - /zh/dev/table/connectors/formats/csv.html Review Comment: sorry, I will revert -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28431) CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager is restarted it fails to recover the job due to "checkpoint FileNotFound exception"
[ https://issues.apache.org/jira/browse/FLINK-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aresyhzhang updated FLINK-28431: Attachment: (was: error.log) > CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager > is restarted it fails to recover the job due to "checkpoint FileNotFound > exception" > --- > > Key: FLINK-28431 > URL: https://issues.apache.org/jira/browse/FLINK-28431 > Project: Flink > Issue Type: Bug >Affects Versions: 1.13.2 > Environment: flink:1.13.2 > java:1.8 >Reporter: aresyhzhang >Priority: Major > Labels: checkpoint, ha, native-kubernetes > > We have built a lot of flink clusters in native Kubernetes session mode, > flink version 1.13.2, some clusters can run normally for 180 days and some > can run for 30 days. > The following takes an abnormal flink cluster > flink-k8s-session-opd-public-1132 as an example. > Problem Description: > Appears when jobmanager restarts > File does not exist: > /home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > The result of this is that the entire flink cluster cannot be started. > Because other tasks in session mode are also affected by the inability to > start, the impact is very serious. > Some auxiliary information: > 1. flink cluster id: flink-k8s-session-opd-public-1132 > 2. High-availability.storageDir of cluster configuration: > hdfs://neophdfsv2flink/home/flink/recovery/ > 3.error job id: 18193cde2c359f492f76c8ce4cd20271 > 4. There was a similar issue before: FLINK-8770, but I saw that it was closed > without being resolved. > 5. The complete jommanager log I have uploaded to the attachment > My investigation ideas: > 1. View the node information on the zookeeper corresponding to the jobid > 18193cde2c359f492f76c8ce4cd20271: > [zk: localhost:2181(CONNECTED) 17] ls > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271 > [0025852, 0025851] > [zk: localhost:2181(CONNECTED) 14] get > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271/0025852 > ??sr;org.apache.flink.runtime.state.RetrievableStreamStateHandle?U?+LwrappedStreamStateHandlet2Lorg/apache/flink/runtime/state/StreamStateHandle;xpsr9org.apache.flink.runtime.state.filesystem.FileStateHandle?u?b?J > > stateSizefilePathtLorg/apache/flink/core/fs/Path;xp??srorg.apache.flink.core.fs.PathLuritLjava/net/URI;xpsr > java.net.URI?x.C?I?LstringtLjava/lang/String;xptrhdfs://neophdfsv2flink/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4ax > cZxid = 0x1070932e2 > ctime = Wed Jul 06 02:28:51 UTC 2022 > mZxid = 0x1070932e2 > mtime = Wed Jul 06 02:28:51 UTC 2022 > pZxid = 0x30001c957 > cversion=222 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 545 > numChildren = 0. > I am sure that my zk node is normal, because there are 10+ flink clusters > using the same zk node, but only this cluster has problems, other clusters > are normal > 2. View the hdfs edits modification log of the directory corresponding to > hdfs: > ./hdfs-audit.log.1:2022-07-06 10:28:51,752 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= create > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=flinkuser:flinkuser:rw-r--r-- proto=rpc > ./hdfs-audit.log.1:2022-07-06 10:29:26,588 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= delete > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=null proto=rpc > I don't know why flink created the directory and then deleted it, and did not > update the metadata information to zookeeper, which caused the jobmanager to > restart without getting the correct directory and keep restarting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28431) CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager is restarted it fails to recover the job due to "checkpoint FileNotFound exception"
[ https://issues.apache.org/jira/browse/FLINK-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563540#comment-17563540 ] aresyhzhang commented on FLINK-28431: - Not tried with latest flink 1.15, but flink 1.14.2 will also appear > CompletedCheckPoints stored on ZooKeeper is not up-to-date, when JobManager > is restarted it fails to recover the job due to "checkpoint FileNotFound > exception" > --- > > Key: FLINK-28431 > URL: https://issues.apache.org/jira/browse/FLINK-28431 > Project: Flink > Issue Type: Bug >Affects Versions: 1.13.2 > Environment: flink:1.13.2 > java:1.8 >Reporter: aresyhzhang >Priority: Major > Labels: checkpoint, ha, native-kubernetes > Attachments: error.log > > > We have built a lot of flink clusters in native Kubernetes session mode, > flink version 1.13.2, some clusters can run normally for 180 days and some > can run for 30 days. > The following takes an abnormal flink cluster > flink-k8s-session-opd-public-1132 as an example. > Problem Description: > Appears when jobmanager restarts > File does not exist: > /home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > The result of this is that the entire flink cluster cannot be started. > Because other tasks in session mode are also affected by the inability to > start, the impact is very serious. > Some auxiliary information: > 1. flink cluster id: flink-k8s-session-opd-public-1132 > 2. High-availability.storageDir of cluster configuration: > hdfs://neophdfsv2flink/home/flink/recovery/ > 3.error job id: 18193cde2c359f492f76c8ce4cd20271 > 4. There was a similar issue before: FLINK-8770, but I saw that it was closed > without being resolved. > 5. The complete jommanager log I have uploaded to the attachment > My investigation ideas: > 1. View the node information on the zookeeper corresponding to the jobid > 18193cde2c359f492f76c8ce4cd20271: > [zk: localhost:2181(CONNECTED) 17] ls > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271 > [0025852, 0025851] > [zk: localhost:2181(CONNECTED) 14] get > /flink/flink/flink-k8s-session-opd-public-1132/checkpoints/18193cde2c359f492f76c8ce4cd20271/0025852 > ??sr;org.apache.flink.runtime.state.RetrievableStreamStateHandle?U?+LwrappedStreamStateHandlet2Lorg/apache/flink/runtime/state/StreamStateHandle;xpsr9org.apache.flink.runtime.state.filesystem.FileStateHandle?u?b?J > > stateSizefilePathtLorg/apache/flink/core/fs/Path;xp??srorg.apache.flink.core.fs.PathLuritLjava/net/URI;xpsr > java.net.URI?x.C?I?LstringtLjava/lang/String;xptrhdfs://neophdfsv2flink/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4ax > cZxid = 0x1070932e2 > ctime = Wed Jul 06 02:28:51 UTC 2022 > mZxid = 0x1070932e2 > mtime = Wed Jul 06 02:28:51 UTC 2022 > pZxid = 0x30001c957 > cversion=222 > dataVersion = 0 > aclVersion = 0 > ephemeralOwner = 0x0 > dataLength = 545 > numChildren = 0. > I am sure that my zk node is normal, because there are 10+ flink clusters > using the same zk node, but only this cluster has problems, other clusters > are normal > 2. View the hdfs edits modification log of the directory corresponding to > hdfs: > ./hdfs-audit.log.1:2022-07-06 10:28:51,752 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= create > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=flinkuser:flinkuser:rw-r--r-- proto=rpc > ./hdfs-audit.log.1:2022-07-06 10:29:26,588 INFO FSNamesystem.audit: > allowed=true ugi=flinku...@hadoop.163.gz (auth:KERBEROS) ip=/10.91.136.213 > cmd= delete > src=/home/flink/recovery/flink/flink-k8s-session-opd-public-1132/completedCheckpoint86fce98d7e4a > dst=null perm=null proto=rpc > I don't know why flink created the directory and then deleted it, and did not > update the metadata information to zookeeper, which caused the jobmanager to > restart without getting the correct directory and keep restarting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zoltar9264 commented on pull request #20093: [FLINK-28172][changelog] Scatter dstl files into separate directories…
zoltar9264 commented on PR #20093: URL: https://github.com/apache/flink/pull/20093#issuecomment-1176966940 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-28429) Reduce pyflink tests time
[ https://issues.apache.org/jira/browse/FLINK-28429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo reassigned FLINK-28429: Assignee: Huang Xingbo > Reduce pyflink tests time > - > > Key: FLINK-28429 > URL: https://issues.apache.org/jira/browse/FLINK-28429 > Project: Flink > Issue Type: Improvement > Components: API / Python >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: test-stability > Fix For: 1.16.0 > > > Currently, it costs about 1 hour 30mins in pyflink tests. We need to optimize > it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28426) PyFlink provides M1 wheel package
[ https://issues.apache.org/jira/browse/FLINK-28426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo reassigned FLINK-28426: Assignee: Huang Xingbo > PyFlink provides M1 wheel package > - > > Key: FLINK-28426 > URL: https://issues.apache.org/jira/browse/FLINK-28426 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > In [FLINK-25188|https://issues.apache.org/jira/browse/FLINK-25188], pyflink > has provided the support of M1 on MacOS. We also need to provide M1 wheel > package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28434) Extend "NestedProjectionUtil" to unify support for projection pushdown for physical fields or metadata only
[ https://issues.apache.org/jira/browse/FLINK-28434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563534#comment-17563534 ] lincoln lee commented on FLINK-28434: - cc [~fsk119] would you like to look at this issue when you have time later? > Extend "NestedProjectionUtil" to unify support for projection pushdown for > physical fields or metadata only > --- > > Key: FLINK-28434 > URL: https://issues.apache.org/jira/browse/FLINK-28434 > Project: Flink > Issue Type: Improvement >Reporter: lincoln lee >Priority: Minor > > Extend `NestedProjectionUtil` to unify support for projection pushdown for > physical fields or metadata only. > Currently, the `NestedProjectionUtil` relies on rewritten of the input row > type, this not works on the case that a table source only > [SupportsReadingMetadata|https://github.com/apache/flink/pull/20118#] and not > [SupportsProjectionPushDown (which was fixed by > |https://github.com/apache/flink/pull/20118#] > [FLINK-28334|https://github.com/apache/flink/pull/20118#] > [)|https://github.com/apache/flink/pull/20118#] > Some refactor work is needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] lincoln-lil commented on a diff in pull request #20118: [FLINK-28334][table-planner] Fix PushProjectIntoTableSourceScanRule: covers the case when table source SupportsReadingMetadata
lincoln-lil commented on code in PR #20118:
URL: https://github.com/apache/flink/pull/20118#discussion_r915383719
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushProjectIntoTableSourceScanRule.java:
##
@@ -337,11 +339,69 @@ private RowType performPushDown(
private List rewriteProjections(
RelOptRuleCall call, TableSourceTable source, NestedSchema
projectedSchema) {
final LogicalProject project = call.rel(0);
+List newProjects = project.getProjects();
+
if (supportsProjectionPushDown(source.tableSource())) {
-return NestedProjectionUtil.rewrite(
-project.getProjects(), projectedSchema,
call.builder().getRexBuilder());
-} else {
-return project.getProjects();
+// if support project push down, then all input ref will be
rewritten includes metadata
Review Comment:
I've created a ticket to track this:
https://issues.apache.org/jira/browse/FLINK-28434
cc @fsk119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-28434) Extend "NestedProjectionUtil" to unify support for projection pushdown for physical fields or metadata only
lincoln lee created FLINK-28434: --- Summary: Extend "NestedProjectionUtil" to unify support for projection pushdown for physical fields or metadata only Key: FLINK-28434 URL: https://issues.apache.org/jira/browse/FLINK-28434 Project: Flink Issue Type: Improvement Reporter: lincoln lee Extend `NestedProjectionUtil` to unify support for projection pushdown for physical fields or metadata only. Currently, the `NestedProjectionUtil` relies on rewritten of the input row type, this not works on the case that a table source only [SupportsReadingMetadata|https://github.com/apache/flink/pull/20118#] and not [SupportsProjectionPushDown (which was fixed by |https://github.com/apache/flink/pull/20118#] [FLINK-28334|https://github.com/apache/flink/pull/20118#] [)|https://github.com/apache/flink/pull/20118#] Some refactor work is needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28430) HiveFunctionWrapper should use UserClassLoader to load class
[ https://issues.apache.org/jira/browse/FLINK-28430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoyuxia updated FLINK-28430: - Description: Now, HiveFunctionWrapper will use current thread's ClassLoader to load the Hive UDFs class. But the class loader may differ with the class loader that loads the Hive UDFs, which will cause class not found issuse. To solve it, we need to use UserClassLoader to load the Hive UDFs class. was: Now, HiveFunctionWrapper will use current thread's ClassLoader to load the Hive UDFs class. But the class loader may differ with the class loader that loads the Hive UDFs, which will cause class not found issuse. > HiveFunctionWrapper should use UserClassLoader to load class > > > Key: FLINK-28430 > URL: https://issues.apache.org/jira/browse/FLINK-28430 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.16.0 >Reporter: luoyuxia >Priority: Major > > Now, HiveFunctionWrapper will use current thread's ClassLoader to load the > Hive UDFs class. But the class loader may differ with the class loader that > loads the Hive UDFs, which will cause class not found issuse. > To solve it, we need to use UserClassLoader to load the Hive UDFs class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28430) HiveFunctionWrapper should use UserClassLoader to load class
[ https://issues.apache.org/jira/browse/FLINK-28430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoyuxia updated FLINK-28430: - Description: Now, HiveFunctionWrapper will use current thread's ClassLoader to load the Hive UDFs class. But the class loader may differ with the class loader that loads the Hive UDFs, which will cause class not found issuse. was: Now, HiveFunctionWrapper will use current thread's ClassLoader to load the Hive UDFs class. But the class loader may differ with the class loader that loads the Hive UDFs, which will cause class not found issuse. To solve it, we need to use UserClassLoader to load the Hive UDFs class. > HiveFunctionWrapper should use UserClassLoader to load class > > > Key: FLINK-28430 > URL: https://issues.apache.org/jira/browse/FLINK-28430 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.16.0 >Reporter: luoyuxia >Priority: Major > > Now, HiveFunctionWrapper will use current thread's ClassLoader to load the > Hive UDFs class. But the class loader may differ with the class loader that > loads the Hive UDFs, which will cause class not found issuse. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27767) Introduce Endpoint API and utils
[ https://issues.apache.org/jira/browse/FLINK-27767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563531#comment-17563531 ] Shengkai Fang commented on FLINK-27767: --- Closed in master: eb395309e046418128363a5401693be3987a3bca > Introduce Endpoint API and utils > > > Key: FLINK-27767 > URL: https://issues.apache.org/jira/browse/FLINK-27767 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Gateway >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-27767) Introduce Endpoint API and utils
[ https://issues.apache.org/jira/browse/FLINK-27767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang closed FLINK-27767. - Resolution: Implemented > Introduce Endpoint API and utils > > > Key: FLINK-27767 > URL: https://issues.apache.org/jira/browse/FLINK-27767 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Gateway >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20190: [FLINK-28426][python] Supports m1 wheel package in PyFlink
flinkbot commented on PR #20190: URL: https://github.com/apache/flink/pull/20190#issuecomment-1176955030 ## CI report: * 4d0a083fb3bde009a06ffd5623f038ebd1985f9b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on pull request #20190: [FLINK-28426][python] Supports m1 wheel package in PyFlink
HuangXingBo commented on PR #20190: URL: https://github.com/apache/flink/pull/20190#issuecomment-1176952374 https://dev.azure.com/hxbks2ks/FLINK-TEST/_build/results?buildId=1895&view=results Manual Test in my private Azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 merged pull request #19849: [FLINK-27767][sql-gateway] Introduce Endpoint API and utils
fsk119 merged PR #19849: URL: https://github.com/apache/flink/pull/19849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28426) PyFlink provides M1 wheel package
[ https://issues.apache.org/jira/browse/FLINK-28426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28426: --- Labels: pull-request-available (was: ) > PyFlink provides M1 wheel package > - > > Key: FLINK-28426 > URL: https://issues.apache.org/jira/browse/FLINK-28426 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > In [FLINK-25188|https://issues.apache.org/jira/browse/FLINK-25188], pyflink > has provided the support of M1 on MacOS. We also need to provide M1 wheel > package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] HuangXingBo opened a new pull request, #20190: [FLINK-28426][python] Supports m1 wheel package in PyFlink
HuangXingBo opened a new pull request, #20190: URL: https://github.com/apache/flink/pull/20190 ## What is the purpose of the change *This pull request will support m1 wheel package in PyFlink* ## Brief change log - *Supports m1 wheel package in PyFlink* ## Verifying this change This change added tests and can be verified as follows: - *Manual tests in Private Azure Pipeline* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-docker] xintongsong merged pull request #122: Update key for 1.15.1 to full fingerprint for official-images
xintongsong merged PR #122: URL: https://github.com/apache/flink-docker/pull/122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-docker] xintongsong merged pull request #123: Update key for 1.15.1 to use full fingerprint
xintongsong merged PR #123: URL: https://github.com/apache/flink-docker/pull/123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] luoyuxia commented on pull request #20071: [FLINK-28247][hive] fix "no any field presented in group by" exception when over window contains grouping function in Hive dialect
luoyuxia commented on PR #20071: URL: https://github.com/apache/flink/pull/20071#issuecomment-1176948801 @lirui-apache The CI is passed now~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28414) Using flink to query the hive table, an exception occurred, SQL validation failed. Failed to get table schema from deserializer
[ https://issues.apache.org/jira/browse/FLINK-28414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563523#comment-17563523 ] xcrossed commented on FLINK-28414: -- ok > Using flink to query the hive table, an exception occurred, SQL validation > failed. Failed to get table schema from deserializer > > > Key: FLINK-28414 > URL: https://issues.apache.org/jira/browse/FLINK-28414 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.14.2 > Environment: hive version:2.1.1 > flink version:1.14.2 > hive table info > CREATE EXTERNAL TABLE `xxx_events`( > `ver` string COMMENT 'from deserializer', > `recv_time`bigint COMMENT 'from deserializer', > `device_id` string COMMENT 'from deserializer', > `app_version` string COMMENT 'from deserializer', > `item_id` string COMMENT 'from deserializer', > `is_new`int COMMENT 'from deserializer', > `ptype`int COMMENT 'from deserializer', > `platform` string COMMENT 'from deserializer', > `scene` string COMMENT 'from deserializer', > `event_type` string COMMENT 'from deserializer') > PARTITIONED BY ( > `dt` string) > ROW FORMAT SERDE > 'org.openx.data.jsonserde.JsonSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' > LOCATION > 's3://.com/user/hive/warehouse/xxx/events' > TBLPROPERTIES ( > 'transient_lastDdlTime'='1623066811') > Time taken: 0.087 seconds, Fetched: 41 row(s) > > flink table code > > val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir) > tableEnv.registerCatalog(name, hive) > // > set the HiveCatalog as the current catalog of the session > tableEnv.useCatalog(name) > > //query sql > statSql="select * from mobile_events where dt = '20220706' " > > //get result > > val rs = tableEnv.sqlQuery(statSql) > > and then occurred, SQL validation failed. Failed to get table schema from > deserializer >Reporter: xcrossed >Priority: Major > Labels: flink, hive-connector, serializable, sql > > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: SQL validation failed. Failed to get table schema from > deserializer > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: org.apache.flink.table.api.ValidationException: SQL validation > failed. Failed to get table schema from deserializer > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:164) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:107) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:215) > at > org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:716) > at > com.zhhainiao.wp.stat.PaidConversationRate$.main(PaidConversationRate.scala:153) > at com.zhhainiao.wp.stat.PaidConversationRate.main(PaidConversationRate.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke
[jira] [Created] (FLINK-28433) Allow connection to mysql through mariadb driver
PengLei created FLINK-28433: --- Summary: Allow connection to mysql through mariadb driver Key: FLINK-28433 URL: https://issues.apache.org/jira/browse/FLINK-28433 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: PengLei Attachments: image-2022-07-07-09-29-06-834.png Flink connector support connection to mysql. But the url must be started with "jdbc:mysql". Some user need to use mariadb dirver to connect to mysql. It can be achieved by setting the driver parameter in jdbcOptions. Unfortunately， the url verification fails. as follows: !image-2022-07-07-09-29-06-834.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28413) Support ParquetColumnarRowInputFormat in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-28413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-28413: Affects Version/s: (was: 1.15.1) > Support ParquetColumnarRowInputFormat in PyFlink > > > Key: FLINK-28413 > URL: https://issues.apache.org/jira/browse/FLINK-28413 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28413) Support ParquetColumnarRowInputFormat in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-28413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned FLINK-28413: --- Assignee: Juntao Hu > Support ParquetColumnarRowInputFormat in PyFlink > > > Key: FLINK-28413 > URL: https://issues.apache.org/jira/browse/FLINK-28413 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.15.1 >Reporter: Juntao Hu >Assignee: Juntao Hu >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28364) Python Job support for Kubernetes Operator
[ https://issues.apache.org/jira/browse/FLINK-28364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563521#comment-17563521 ] Dian Fu commented on FLINK-28364: - [~gyfora] [~thw] +1 to start with a simple example. > Python Job support for Kubernetes Operator > -- > > Key: FLINK-28364 > URL: https://issues.apache.org/jira/browse/FLINK-28364 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: wu3396 >Priority: Major > > *Describe the solution* > Job types that I want to support pyflink for > *Describe alternatives* > like [here|https://github.com/spotify/flink-on-k8s-operator/pull/165] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28369) Support PARSE_URL bulit-in function in Table API
[ https://issues.apache.org/jira/browse/FLINK-28369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-28369. --- Resolution: Fixed Merged to master via a5b088da231be0d844ef2bc74c148e3bafffb2e1 > Support PARSE_URL bulit-in function in Table API > > > Key: FLINK-28369 > URL: https://issues.apache.org/jira/browse/FLINK-28369 > Project: Flink > Issue Type: Improvement > Components: API / Python, Table SQL / API >Reporter: LuNing Wang >Assignee: LuNing Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20189: [FLINK-28388][docs][tests] Fix documentation build
flinkbot commented on PR #20189: URL: https://github.com/apache/flink/pull/20189#issuecomment-1176923788 ## CI report: * eb3a95316259e4c79775c8b813ab2eae498950f6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu merged pull request #20146: [FLINK-28369][table][python] Support PARSE_URL in Table API
dianfu merged PR #20146: URL: https://github.com/apache/flink/pull/20146 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] swuferhong commented on pull request #20007: [FLINK-27989][table-planner] Csv format supports reporting statistics
swuferhong commented on PR #20007: URL: https://github.com/apache/flink/pull/20007#issuecomment-1176921874 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28388) Python doc build breaking nightly docs
[
https://issues.apache.org/jira/browse/FLINK-28388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-28388:
---
Labels: pull-request-available (was: )
> Python doc build breaking nightly docs
> --
>
> Key: FLINK-28388
> URL: https://issues.apache.org/jira/browse/FLINK-28388
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Documentation
>Affects Versions: shaded-7.0
>Reporter: Márton Balassi
>Assignee: Dian Fu
>Priority: Blocker
> Labels: pull-request-available
>
> For the past 5 days the nightly doc builds via GHA are broken:
> https://github.com/apache/flink/actions/workflows/docs.yml
> {noformat}
> Exception occurred:
> File "/root/flink/flink-python/pyflink/java_gateway.py", line 86, in
> launch_gateway
> raise Exception("It's launching the PythonGatewayServer during Python UDF
> execution "
> Exception: It's launching the PythonGatewayServer during Python UDF execution
> which is unexpected. It usually happens when the job codes are in the top
> level of the Python script file and are not enclosed in a `if name == 'main'`
> statement.
> The full traceback has been saved in /tmp/sphinx-err-3thh_wi2.log, if you
> want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error
> message can be provided next time.
> A bug report can be filed in the tracker at
> . Thanks!
> Makefile:76: recipe for target 'html' failed
> make: *** [html] Error 2
> ==sphinx checks... [FAILED]===
> Error: Process completed with exit code 1.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] dianfu opened a new pull request, #20189: [FLINK-28388][docs][tests] Fix documentation build
dianfu opened a new pull request, #20189: URL: https://github.com/apache/flink/pull/20189 ## What is the purpose of the change *This pull request fixes documentation build* This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no ) - The S3 file system connector: (no ) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28388) Python doc build breaking nightly docs
[
https://issues.apache.org/jira/browse/FLINK-28388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17563515#comment-17563515
]
Dian Fu commented on FLINK-28388:
-
[~mbalassi] Thanks for suggestions. Will try it.
> Python doc build breaking nightly docs
> --
>
> Key: FLINK-28388
> URL: https://issues.apache.org/jira/browse/FLINK-28388
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Documentation
>Affects Versions: shaded-7.0
>Reporter: Márton Balassi
>Assignee: Dian Fu
>Priority: Blocker
>
> For the past 5 days the nightly doc builds via GHA are broken:
> https://github.com/apache/flink/actions/workflows/docs.yml
> {noformat}
> Exception occurred:
> File "/root/flink/flink-python/pyflink/java_gateway.py", line 86, in
> launch_gateway
> raise Exception("It's launching the PythonGatewayServer during Python UDF
> execution "
> Exception: It's launching the PythonGatewayServer during Python UDF execution
> which is unexpected. It usually happens when the job codes are in the top
> level of the Python script file and are not enclosed in a `if name == 'main'`
> statement.
> The full traceback has been saved in /tmp/sphinx-err-3thh_wi2.log, if you
> want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error
> message can be provided next time.
> A bug report can be filed in the tracker at
> . Thanks!
> Makefile:76: recipe for target 'html' failed
> make: *** [html] Error 2
> ==sphinx checks... [FAILED]===
> Error: Process completed with exit code 1.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28287) Should TaskManagerRunner need a ShutdownHook
[ https://issues.apache.org/jira/browse/FLINK-28287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JieFang.He updated FLINK-28287: --- Affects Version/s: 1.14.0 > Should TaskManagerRunner need a ShutdownHook > > > Key: FLINK-28287 > URL: https://issues.apache.org/jira/browse/FLINK-28287 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.14.0, 1.14.5, 1.15.1 >Reporter: JieFang.He >Priority: Major > > TaskManagerRunner has a close method，but did not call when it stop. > Some resources in TaskManagerRunner come with ShutdownHook, but some > resources do not, such as rpcSystem, which causes the temporary file > flink-rpc-akka_*.jar to not be deleted when stop. > Should TaskManagerRunner need a ShutdownHook to call the close method to > release all resources > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28287) Should TaskManagerRunner need a ShutdownHook
[ https://issues.apache.org/jira/browse/FLINK-28287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JieFang.He updated FLINK-28287: --- Affects Version/s: 1.15.1 1.14.5 (was: 1.14.0) > Should TaskManagerRunner need a ShutdownHook > > > Key: FLINK-28287 > URL: https://issues.apache.org/jira/browse/FLINK-28287 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.14.5, 1.15.1 >Reporter: JieFang.He >Priority: Major > > TaskManagerRunner has a close method，but did not call when it stop. > Some resources in TaskManagerRunner come with ShutdownHook, but some > resources do not, such as rpcSystem, which causes the temporary file > flink-rpc-akka_*.jar to not be deleted when stop. > Should TaskManagerRunner need a ShutdownHook to call the close method to > release all resources > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25450) JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-25450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25450:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable on
> azure
> -
>
> Key: FLINK-25450
> URL: https://issues.apache.org/jira/browse/FLINK-25450
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Yun Gao
>Priority: Major
> Labels: stale-major, test-stability
>
> {code:java}
> 2021-12-25T03:54:10.3320203Z java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (9a4dafa6cd128b6113ec046941eb3842)
> 2021-12-25T03:54:10.3321140Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2021-12-25T03:54:10.3321759Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2021-12-25T03:54:10.3341006Z at
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCacheTest.testImmediateCacheInvalidationAfterFailure(DefaultExecutionGraphCacheTest.java:147)
> 2021-12-25T03:54:10.3342333Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-12-25T03:54:10.3343083Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-12-25T03:54:10.3343890Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-12-25T03:54:10.3344613Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-12-25T03:54:10.3345328Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-12-25T03:54:10.3346116Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-12-25T03:54:10.3347173Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-12-25T03:54:10.3347951Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-12-25T03:54:10.3348720Z at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-12-25T03:54:10.3349434Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-25T03:54:10.3350121Z at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-25T03:54:10.3358745Z at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-12-25T03:54:10.3359459Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-12-25T03:54:10.3360100Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-12-25T03:54:10.3360863Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-12-25T03:54:10.3361486Z at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-12-25T03:54:10.3362065Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-12-25T03:54:10.3362668Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-12-25T03:54:10.3363436Z at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-12-25T03:54:10.3364020Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-12-25T03:54:10.3364582Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-12-25T03:54:10.3365146Z at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-25T03:54:10.3365667Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-12-25T03:54:10.3366202Z at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-12-25T03:54:10.3366684Z at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-12-25T03:54:10.3367245Z at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> 2021-12-25T03:54:10.3367904Z at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> 2021-12-25T03:54:10.3368579Z at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
> 2021-12-25T03:54:10.3369303Z at
> o
[jira] [Updated] (FLINK-22255) AdaptiveScheduler improvements/bugs
[
https://issues.apache.org/jira/browse/FLINK-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22255:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> AdaptiveScheduler improvements/bugs
> ---
>
> Key: FLINK-22255
> URL: https://issues.apache.org/jira/browse/FLINK-22255
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: pull-request-available, stale-major
>
> This ticket collects the improvements/bugs for the {{AdaptiveScheduler}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-25456) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint
[
https://issues.apache.org/jira/browse/FLINK-25456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25456:
---
Labels: auto-deprioritized-critical stale-major test-stability (was:
auto-deprioritized-critical test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint
> ---
>
> Key: FLINK-25456
> URL: https://issues.apache.org/jira/browse/FLINK-25456
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.2, 1.15.0
>Reporter: Till Rohrmann
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major, test-stability
>
> The test {{FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint}}
> fails with
> {code}
> 2021-12-27T02:54:54.8464375Z Dec 27 02:54:54 [ERROR] Tests run: 15, Failures:
> 1, Errors: 0, Skipped: 0, Time elapsed: 285.279 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2021-12-27T02:54:54.8465354Z Dec 27 02:54:54 [ERROR]
> testScaleDownBeforeFirstCheckpoint Time elapsed: 85.514 s <<< FAILURE!
> 2021-12-27T02:54:54.8468827Z Dec 27 02:54:54 java.lang.AssertionError:
> Detected producer leak. Thread name: kafka-producer-network-thread |
> producer-MockTask-002a002c-18
> 2021-12-27T02:54:54.8469779Z Dec 27 02:54:54 at
> org.junit.Assert.fail(Assert.java:89)
> 2021-12-27T02:54:54.8470485Z Dec 27 02:54:54 at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.checkProducerLeak(FlinkKafkaProducerITCase.java:847)
> 2021-12-27T02:54:54.8471842Z Dec 27 02:54:54 at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint(FlinkKafkaProducerITCase.java:381)
> 2021-12-27T02:54:54.8472724Z Dec 27 02:54:54 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-12-27T02:54:54.8473509Z Dec 27 02:54:54 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-12-27T02:54:54.8474704Z Dec 27 02:54:54 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-12-27T02:54:54.8475523Z Dec 27 02:54:54 at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2021-12-27T02:54:54.8476258Z Dec 27 02:54:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-12-27T02:54:54.8476949Z Dec 27 02:54:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-12-27T02:54:54.8477632Z Dec 27 02:54:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-12-27T02:54:54.8478451Z Dec 27 02:54:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-12-27T02:54:54.8479282Z Dec 27 02:54:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-12-27T02:54:54.8479976Z Dec 27 02:54:54 at
> org.apache.flink.testutils.junit.RetryRule$RetryOnFailureStatement.evaluate(RetryRule.java:135)
> 2021-12-27T02:54:54.8480696Z Dec 27 02:54:54 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-12-27T02:54:54.8481410Z Dec 27 02:54:54 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-27T02:54:54.8482009Z Dec 27 02:54:54 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-27T02:54:54.8482636Z Dec 27 02:54:54 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-12-27T02:54:54.8483267Z Dec 27 02:54:54 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-12-27T02:54:54.8483900Z Dec 27 02:54:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-12-27T02:54:54.8484574Z Dec 27 02:54:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-12-27T02:54:54.8485214Z Dec 27 02:54:54 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-12-27T02:54:54.8485838Z Dec 27 02:54:54 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-12-27T02:54:54.8486441Z Dec 27 02:54:54 at
> org.junit
[jira] [Updated] (FLINK-25939) PyFlink YARN per-job on Docker test fails on AZP because it could not acquire all required slots
[
https://issues.apache.org/jira/browse/FLINK-25939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25939:
---
Labels: auto-deprioritized-critical stale-major test-stability (was:
auto-deprioritized-critical test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> PyFlink YARN per-job on Docker test fails on AZP because it could not acquire
> all required slots
>
>
> Key: FLINK-25939
> URL: https://issues.apache.org/jira/browse/FLINK-25939
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Deployment / YARN
>Affects Versions: 1.14.3
>Reporter: Till Rohrmann
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major, test-stability
>
> The test {{PyFlink YARN per-job on Docker test}} fails on AZP with:
> {code}
> 2022-02-03T03:32:41.0643749Z Feb 03 03:32:40 2022-02-03 03:31:12,281 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
> Values(tuples=[[{ 1 }, { 2 }, { 3 }]]) -> Calc(select=[f0]) ->
> PythonCalc(select=[add_one(f0) AS a]) -> Sink: Collect table sink (1/1)
> (1c3df1a51de35e44664305a57f4047a7) switched from SCHEDULED to FAILED on
> [unassigned resource].
> 2022-02-03T03:32:41.0645152Z Feb 03 03:32:40
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout
> 2022-02-03T03:32:41.0647529Z Feb 03 03:32:40 at
> org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86)
> ~[flink-dist_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0648830Z Feb 03 03:32:40 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_312]
> 2022-02-03T03:32:41.0649605Z Feb 03 03:32:40 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_312]
> 2022-02-03T03:32:41.0651215Z Feb 03 03:32:40 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455)
> ~[flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0652935Z Feb 03 03:32:40 at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
> ~[flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0654715Z Feb 03 03:32:40 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455)
> ~[flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0656399Z Feb 03 03:32:40 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213)
> ~[flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0657987Z Feb 03 03:32:40 at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
> ~[flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0659528Z Feb 03 03:32:40 at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
> ~[flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0661186Z Feb 03 03:32:40 at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> [flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0662745Z Feb 03 03:32:40 at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> [flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0664203Z Feb 03 03:32:40 at
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> [flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0665591Z Feb 03 03:32:40 at
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> [flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0667179Z Feb 03 03:32:40 at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> [flink-rpc-akka_92d95797-5383-46fa-abb4-628d6de63216.jar:1.14-SNAPSHOT]
> 2022-02-03T03:32:41.0668708Z Feb 03 03:3
[jira] [Updated] (FLINK-24345) Translate "File Systems" page of "Internals" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-24345: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Translate "File Systems" page of "Internals" into Chinese > - > > Key: FLINK-24345 > URL: https://issues.apache.org/jira/browse/FLINK-24345 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Reporter: Liebing Yu >Priority: Major > Labels: pull-request-available, stale-major > > The page url is > https://nightlies.apache.org/flink/flink-docs-master/docs/internals/filesystems/ > The markdown file is located in > flink/docs/content/docs/internals/filesystems.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25624) KafkaSinkITCase.testRecoveryWithExactlyOnceGuarantee blocked on azure pipeline
[
https://issues.apache.org/jira/browse/FLINK-25624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25624:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> KafkaSinkITCase.testRecoveryWithExactlyOnceGuarantee blocked on azure pipeline
> --
>
> Key: FLINK-25624
> URL: https://issues.apache.org/jira/browse/FLINK-25624
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: stale-major, test-stability
>
> {code:java}
> "main" #1 prio=5 os_prio=0 tid=0x7fda8c00b000 nid=0x21b2 waiting on
> condition [0x7fda92dd7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x826165c0> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1989)
> at
> org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:69)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1951)
> at
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase.testRecoveryWithAssertion(KafkaSinkITCase.java:335)
> at
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase.testRecoveryWithExactlyOnceGuarantee(KafkaSinkITCase.java:190)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=29285&view=logs&j=c5612577-f1f7-5977-6ff6-7432788526f7&t=ffa8837a-b445-534e-cdf4-db364cf8235d&l=42106
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-25217) FLIP-190: Support Version Upgrades for Table API & SQL Programs
[ https://issues.apache.org/jira/browse/FLINK-25217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-25217: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > FLIP-190: Support Version Upgrades for Table API & SQL Programs > --- > > Key: FLINK-25217 > URL: https://issues.apache.org/jira/browse/FLINK-25217 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Major > Labels: pull-request-available, stale-major > > Nowadays, the Table & SQL API is as important to Flink as the DataStream API. > It is one of the main abstractions for expressing pipelines that perform > stateful stream processing. Users expect the same backwards compatibility > guarantees when upgrading to a newer Flink version as with the DataStream API. > In particular, this means: > * once the operator topology is defined, it remains static and does not > change between Flink versions, unless resulting in better performance, > * business logic (defined using expressions and functions in queries) behaves > identical as before the version upgrade, > * the state of a Table & SQL API program can be restored from a savepoint of > a previous version, > * adding or removing stateful operators should be made possible in the > DataStream API. > The same query can remain up and running after upgrades. > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25585) JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure failed on the azure
[
https://issues.apache.org/jira/browse/FLINK-25585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25585:
---
Labels: auto-deprioritized-critical stale-major test-stability (was:
auto-deprioritized-critical test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure failed
> on the azure
> -
>
> Key: FLINK-25585
> URL: https://issues.apache.org/jira/browse/FLINK-25585
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.2, 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major, test-stability
>
> {code:java}
> 2022-01-08T02:39:54.7103772Z Jan 08 02:39:54 [ERROR] Tests run: 2, Failures:
> 1, Errors: 0, Skipped: 0, Time elapsed: 349.282 s <<< FAILURE! - in
> org.apache.flink.test.recovery.JobManagerHAProcessFailureRecoveryITCase
> 2022-01-08T02:39:54.7105233Z Jan 08 02:39:54 [ERROR]
> testDispatcherProcessFailure[ExecutionMode BATCH] Time elapsed: 302.006 s
> <<< FAILURE!
> 2022-01-08T02:39:54.7106478Z Jan 08 02:39:54 java.lang.AssertionError: The
> program encountered a RuntimeException :
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> while waiting for job to be initialized
> 2022-01-08T02:39:54.7107409Z Jan 08 02:39:54 at
> org.junit.Assert.fail(Assert.java:89)
> 2022-01-08T02:39:54.7108084Z Jan 08 02:39:54 at
> org.apache.flink.test.recovery.JobManagerHAProcessFailureRecoveryITCase.testDispatcherProcessFailure(JobManagerHAProcessFailureRecoveryITCase.java:383)
> 2022-01-08T02:39:54.7108952Z Jan 08 02:39:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-01-08T02:39:54.7109491Z Jan 08 02:39:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-01-08T02:39:54.7110107Z Jan 08 02:39:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-01-08T02:39:54.7110772Z Jan 08 02:39:54 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-01-08T02:39:54.7111594Z Jan 08 02:39:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-01-08T02:39:54.7112510Z Jan 08 02:39:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-01-08T02:39:54.7113734Z Jan 08 02:39:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-01-08T02:39:54.7114673Z Jan 08 02:39:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-01-08T02:39:54.7115423Z Jan 08 02:39:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-01-08T02:39:54.7116011Z Jan 08 02:39:54 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-01-08T02:39:54.7116586Z Jan 08 02:39:54 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-01-08T02:39:54.7117154Z Jan 08 02:39:54 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-01-08T02:39:54.7117686Z Jan 08 02:39:54 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-01-08T02:39:54.7118448Z Jan 08 02:39:54 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-01-08T02:39:54.7119020Z Jan 08 02:39:54 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-01-08T02:39:54.7119571Z Jan 08 02:39:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-01-08T02:39:54.7120180Z Jan 08 02:39:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-01-08T02:39:54.7120754Z Jan 08 02:39:54 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-01-08T02:39:54.7121286Z Jan 08 02:39:54 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-01-08T02:39:54.7121832Z Jan 08 02:39:54 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-01-08T02:39:54.7122376Z Jan 08 02:39:54 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-01-08T02:39:54.7123179Z Jan 08 02:39
