[jira] [Commented] (FLINK-17330) Avoid scheduling deadlocks caused by cyclic input dependencies between regions
[
https://issues.apache.org/jira/browse/FLINK-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091251#comment-17091251
]
Zhu Zhu commented on FLINK-17330:
-
>> Would it work to say that in the first version we don't support pipelined
>> regions which contain a blocking data exchange?
I think it's possible but it may be hard for users to identify whether there
are cyclic dependencies. Most users will have to choose the mode to set all
edges BLOCKING to be safe and lose the benefit of pipelined region scheduling.
So if we'd like to take it this way, I think it's better we do it automatically
for users, i.e. override GlobalDataExchangeMode to be ALL_EDGES_BLOCKING if
cyclic dependency is detected.
>> I think we would have to detect cyclic dependencies between pipelined
>> regions and merge all regions which are part of the cycle into the same
>> pipelined region
This sounds good to me. It's more flexible without the assumption of the
mapping of logical topology and execution topology. The main concern is the
computation complexity but we can review it later.
I just realized my previous question actually consists of 2 parts:
1. whether to merge regions into one if they have cyclic dependencies? This
seems to be a must if we want to avoid resource deadlocks in initial scheduling
and failure recovery
2. how to detect cyclic dependencies? Checking whether there are intra-region
all-to-all blocking edges can be a performance efficient solution but is not
the only choice, and it also requires attention to POINTWISE edges. If we can
have a common way to find out cyclic dependencies in O(V^2), I think it's even
better. This question can be answered later when we have a deeper look at all
the options.
> Avoid scheduling deadlocks caused by cyclic input dependencies between regions
> --
>
> Key: FLINK-17330
> URL: https://issues.apache.org/jira/browse/FLINK-17330
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> Imagine a job like this:
> A -- (pipelined FORWARD) --> B -- (blocking ALL-to-ALL) --> D
> A -- (pipelined FORWARD) --> C -- (pipelined FORWARD) --> D
> parallelism=2 for all vertices.
> We will have 2 execution pipelined regions:
> R1 = {A1, B1, C1, D1}
> R2 = {A2, B2, C2, D2}
> R1 has a cross-region input edge (B2->D1).
> R2 has a cross-region input edge (B1->D2).
> Scheduling deadlock will happen since we schedule a region only when all its
> inputs are consumable (i.e. blocking partitions to be finished). This is
> because R1 can be scheduled only if R2 finishes, while R2 can be scheduled
> only if R1 finishes.
> To avoid this, one solution is to force a logical pipelined region with
> intra-region ALL-to-ALL blocking edges to form one only execution pipelined
> region, so that there would not be cyclic input dependency between regions.
> Besides that, we should also pay attention to avoid cyclic cross-region
> POINTWISE blocking edges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11774: [FLINK-17020][runtime] Introduce GlobalDataExchangeMode for JobGraph generation
flinkbot edited a comment on pull request #11774: URL: https://github.com/apache/flink/pull/11774#issuecomment-614576955 ## CI report: * b1316cec224ccda73e1eed8226c0b5b61f2c6e21 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161758669) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=162) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #11863: [FLINK-17048][mesos] Add memory related JVM args to Mesos JM startup scripts
zentol commented on a change in pull request #11863:
URL: https://github.com/apache/flink/pull/11863#discussion_r413672750
##
File path: flink-dist/src/main/flink-bin/mesos-bin/mesos-appmaster.sh
##
@@ -17,29 +17,10 @@
# limitations under the License.
-bin=`dirname "$0"`
-bin=`cd "$bin"; pwd`
+bin=$(dirname "$0")
+bin=$(cd "${bin}" || exit; pwd)
# get Flink config
-. "$bin"/config.sh
+. "${bin}"/config.sh
-if [ "$FLINK_IDENT_STRING" = "" ]; then
-FLINK_IDENT_STRING="$USER"
-fi
-
-CC_CLASSPATH=`manglePathList
$(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPATHS`
-
-log="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.log"
-log_setting="-Dlog.file="$log"
-Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j.properties
-Dlog4j.configurationFile=file:"$FLINK_CONF_DIR"/log4j.properties
-Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml"
-
-ENTRY_POINT=org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint
-
-exec $JAVA_RUN $JVM_ARGS -classpath "$CC_CLASSPATH" $log_setting
${ENTRY_POINT} "$@"
-
-rc=$?
-
-if [[ $rc -ne 0 ]]; then
-echo "Error while starting the mesos application master. Please check
${log} for more details."
-fi
-
-exit $rc
+exec "${FLINK_BIN_DIR}"/mesos-jobmanager.sh
"org.apache.flink.mesos.entrypoint.MesosSessionClusterEntrypoint" "$@"
Review comment:
is the `exec` actually necessary? I see both approaches in
`taskmanager.sh`; would be good to make a conscious decision to pick one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11876: [FLINK-17334] HIVE UDF BUGFIX
flinkbot edited a comment on pull request #11876: URL: https://github.com/apache/flink/pull/11876#issuecomment-618262946 ## CI report: * ec30671bf09b056ef47b4b89001bd8118ab23513 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161604503) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=127) * 5dc081058c881b0165b44c37e53607ac891814f5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] docete commented on a change in pull request #11837: [FLINK-16160][table-planner-blink] Fix proctime()/rowtime() doesn't w…
docete commented on a change in pull request #11837:
URL: https://github.com/apache/flink/pull/11837#discussion_r414342841
##
File path:
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/utils/testTableSourceSinks.scala
##
@@ -200,6 +199,65 @@ class TestTableSourceWithTime[T](
}
}
+class TestTableSourceWithTimeFactory[T] extends StreamTableSourceFactory[T] {
+ override def createStreamTableSource(properties: JMap[String, String]):
StreamTableSource[T] = {
+val dp = new DescriptorProperties()
+dp.putProperties(properties)
+
+val isBounded = dp.getOptionalBoolean("is-bounded").orElse(false)
+val tableSchema = dp.getTableSchema(Schema.SCHEMA)
+val serializedData = dp.getOptionalString("data").orElse(null)
+val data = if (serializedData != null) {
+ EncodingUtils.decodeStringToObject(serializedData, classOf[List[T]])
+} else {
+ Seq.empty[T]
+}
+val rowtimeAttributes = SchemaValidator.deriveRowtimeAttributes(dp)
+val rowtime = if (rowtimeAttributes.isEmpty) {
+ null
+} else {
+ rowtimeAttributes.head.getAttributeName
+}
+val proctimeAttribute = SchemaValidator.deriveProctimeAttribute(dp)
+val proctime = if (proctimeAttribute.isPresent) {
+ proctimeAttribute.get()
+} else {
+ null
+}
+
+val serializedMapKeys = dp.getOptionalString("map-keys").orElse(null)
+val serializedMapVals = dp.getOptionalString("map-vals").orElse(null)
+val mapping = if (serializedMapKeys != null && serializedMapVals != null) {
+ val mapKeys = EncodingUtils.decodeStringToObject(serializedMapKeys,
classOf[List[String]])
+ val mapVals = EncodingUtils.decodeStringToObject(serializedMapVals,
classOf[List[String]])
+ if (mapKeys.length != mapVals.length) {
+null
+ } else {
+mapKeys.zip(mapVals).toMap
+ }
+} else {
+ null
+}
+
+val existingTs = dp.getOptionalString("existingTs").orElse(null)
+
+new TestTableSourceWithTime[T](
+ isBounded, tableSchema, null, data, rowtime, proctime, mapping,
existingTs)
Review comment:
See TableScanITCase#testProctimeTableSource()
It should use this factory after we remove tableEnv.registerTableSource()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #11774: [FLINK-17020][runtime] Introduce GlobalDataExchangeMode for JobGraph generation
zhuzhurk commented on pull request #11774: URL: https://github.com/apache/flink/pull/11774#issuecomment-618842768 Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17365) Update Dockerfiles to 1.9.3
Chesnay Schepler created FLINK-17365: Summary: Update Dockerfiles to 1.9.3 Key: FLINK-17365 URL: https://issues.apache.org/jira/browse/FLINK-17365 Project: Flink Issue Type: Improvement Components: Dockerfiles Reporter: Chesnay Schepler Assignee: Chesnay Schepler -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] godfreyhe commented on a change in pull request #11869: [FLINK-17111][table] Support SHOW VIEWS in Flink SQL
godfreyhe commented on a change in pull request #11869:
URL: https://github.com/apache/flink/pull/11869#discussion_r414343381
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java
##
@@ -415,6 +415,37 @@ public CatalogBaseTable getTable() {
});
}
+ /**
+* Returns an array of names of all views(both temporary and permanent)
registered in
+* the namespace of the current catalog and database.
+*
+* @return names of all registered views
+*/
+ public Set listViews() {
+ return listViews(getCurrentCatalog(), getCurrentDatabase());
+ }
+
+ /**
+* Returns an array of names of all views(both temporary and permanent)
registered in
+* the namespace of the current catalog and database.
+*
+* @return names of registered views
+*/
+ public Set listViews(String catalogName, String databaseName) {
+ Catalog currentCatalog = catalogs.get(getCurrentCatalog());
+
+ try {
+ return Stream.concat(
+
currentCatalog.listViews(getCurrentDatabase()).stream(),
+ listTemporaryTablesInternal(catalogName,
databaseName)
+ .filter(e -> e.getValue() instanceof
CatalogView)
+ .map(e -> e.getKey().getObjectName())
+ ).collect(Collectors.toSet());
Review comment:
add a method `listTemporaryViews(String catalogName, String
databaseName)`, `listTemporaryViews()` method and here can use this method
##
File path:
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/api/batch/BatchTableEnvironmentTest.scala
##
@@ -257,6 +257,64 @@ class BatchTableEnvironmentTest extends TableTestBase {
util.tableEnv.executeSql("select * from MyTable")
}
+ @Test
+ def testExecuteSqlWithCreateDropView(): Unit = {
+val util = batchTestUtil()
+
+val createTableStmt =
+ """
+|CREATE TABLE tbl1 (
+| a bigint,
+| b int,
+| c varchar
+|) with (
+| 'connector' = 'COLLECTION',
+| 'is-bounded' = 'true'
+|)
+ """.stripMargin
+val tableResult1 = util.tableEnv.executeSql(createTableStmt)
+assertEquals(ResultKind.SUCCESS, tableResult1.getResultKind)
+assertTrue(util.tableEnv.getCatalog(util.tableEnv.getCurrentCatalog).get()
+
.tableExists(ObjectPath.fromString(s"${util.tableEnv.getCurrentDatabase}.tbl1")))
+
+val tableResult2 = util.tableEnv.executeSql("CREATE VIEW view1 AS SELECT *
FROM tbl1")
+assertEquals(ResultKind.SUCCESS, tableResult2.getResultKind)
+assertTrue(util.tableEnv.getCatalog(util.tableEnv.getCurrentCatalog).get()
+
.tableExists(ObjectPath.fromString(s"${util.tableEnv.getCurrentDatabase}.view1")))
+
+val tableResult3 = util.tableEnv.executeSql("DROP VIEW view1")
+assertEquals(ResultKind.SUCCESS, tableResult3.getResultKind)
+assertFalse(util.tableEnv.getCatalog(util.tableEnv.getCurrentCatalog).get()
+
.tableExists(ObjectPath.fromString(s"${util.tableEnv.getCurrentDatabase}.view1")))
+ }
+
+ @Test
+ def testExecuteSqlWithShowViews(): Unit = {
+val util = batchTestUtil()
+val createTableStmt =
+ """
+|CREATE TABLE tbl1 (
+| a bigint,
+| b int,
+| c varchar
+|) with (
+| 'connector' = 'COLLECTION',
+| 'is-bounded' = 'false'
+|)
+ """.stripMargin
+val tableResult1 = util.tableEnv.executeSql(createTableStmt)
+assertEquals(ResultKind.SUCCESS, tableResult1.getResultKind)
+
+val tableResult2 = util.tableEnv.executeSql("CREATE VIEW view1 AS SELECT *
FROM tbl1")
+assertEquals(ResultKind.SUCCESS, tableResult2.getResultKind)
+
+val tableResult3 = util.tableEnv.executeSql("SHOW VIEWS")
+assertEquals(ResultKind.SUCCESS_WITH_CONTENT, tableResult3.getResultKind)
+checkData(
+ util.tableEnv.listViews().map(Row.of(_)).toList.asJava.iterator(),
+ tableResult3.collect())
Review comment:
add a test "CREATE TEMPORARY VIEW ..."
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableEnvironment.java
##
@@ -756,6 +756,13 @@ default Table fromValues(DataType rowType, Object...
values) {
*/
String[] listTables();
+ /**
+* Gets the names of all views available in the current namespace (the
current database of the current catalog).
+* It returns both temporary and permanent views.
+* @return A list of the names of all registered views in the current
database of the current catalog.
+*/
Review comment:
add `@see #listTemporaryViews()`
##
File path: flink-python/pyflink/table/tests/test_envir
[jira] [Closed] (FLINK-17020) Introduce GlobalDataExchangeMode for JobGraph Generation
[ https://issues.apache.org/jira/browse/FLINK-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu closed FLINK-17020. --- Resolution: Fixed Implemented via ef4daeba7881cecc1548e387ab68d829f998dc67 > Introduce GlobalDataExchangeMode for JobGraph Generation > > > Key: FLINK-17020 > URL: https://issues.apache.org/jira/browse/FLINK-17020 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce GlobalDataExchangeMode with 4 modes: > * ALL_EDGES_BLOCKING > * FORWARD_EDGES_PIPELINED > * POINTWISE_EDGES_PIPELINED > * ALL_EDGES_PIPELINED > StreamGraph will be extended with a new field to host the > GlobalDataExchangeMode. In the JobGraph generation stage, this mode will be > used to determine the data exchange type of each job edge. > More details see [FLIP-119#Global Data Exchange > Mode|https://cwiki.apache.org/confluence/display/FLINK/FLIP-119+Pipelined+Region+Scheduling#FLIP-119PipelinedRegionScheduling-GlobalDataExchangeMode] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-docker] zentol opened a new pull request #16: [FLINK-17365] Update Dockerfiles to 1.9.3
zentol opened a new pull request #16: URL: https://github.com/apache/flink-docker/pull/16 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17365) Update Dockerfiles to 1.9.3
[ https://issues.apache.org/jira/browse/FLINK-17365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17365: --- Labels: pull-request-available (was: ) > Update Dockerfiles to 1.9.3 > --- > > Key: FLINK-17365 > URL: https://issues.apache.org/jira/browse/FLINK-17365 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17287) Disable merge commit button
[ https://issues.apache.org/jira/browse/FLINK-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091255#comment-17091255 ] Chesnay Schepler commented on FLINK-17287: -- docker-master: 9aa8079fc95bb8d7fc704a880471035fd5b1af61 > Disable merge commit button > --- > > Key: FLINK-17287 > URL: https://issues.apache.org/jira/browse/FLINK-17287 > Project: Flink > Issue Type: Improvement >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > Make use of the > [.asf.yaml|https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories] > feature to disable the GitHub merge commit button. > Ideally we just drop this into all repos for consistency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu commented on pull request #11884: [FLINK-17345][python][table] Support register and get Python UDF in Catalog.
dianfu commented on pull request #11884: URL: https://github.com/apache/flink/pull/11884#issuecomment-618849121 @WeiZhong94 Thanks a lot for the update. Will merge once the travis/azure turns green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11796: [FLINK-14258][table][filesystem] Integrate file system connector to streaming sink
flinkbot edited a comment on pull request #11796: URL: https://github.com/apache/flink/pull/11796#issuecomment-615254717 ## CI report: * 4d98d46528fab8087fccf79b9184f27a6197c714 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161760487) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=163) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint
flinkbot edited a comment on pull request #11687: URL: https://github.com/apache/flink/pull/11687#issuecomment-611445542 ## CI report: * 5bad018f00f85a7359345187a12d7938aa510d25 UNKNOWN * d51ce7f47381d99b843278cd701dcff223761a0b UNKNOWN * cba096ae3d8eba4a0d39c64659f76ad10a62be27 UNKNOWN * 23818c4b30eb9714154080253a090fc63d983925 UNKNOWN * 9b6278f15a4fc65d78671a145305df2c7310cc6a UNKNOWN * 4f7399680818f5d29e917e17720e00900822a43d UNKNOWN * 12c398dd2dc5b6716a590490575914698573ab76 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161752886) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=157) * 95dc7d6ada34179b51849d905f15c740ca585a8d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11876: [FLINK-17334] HIVE UDF BUGFIX
flinkbot edited a comment on pull request #11876: URL: https://github.com/apache/flink/pull/11876#issuecomment-618262946 ## CI report: * 5dc081058c881b0165b44c37e53607ac891814f5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161774926) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=170) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11837: [FLINK-16160][table-planner-blink] Fix proctime()/rowtime() doesn't w…
flinkbot edited a comment on pull request #11837: URL: https://github.com/apache/flink/pull/11837#issuecomment-616956896 ## CI report: * 9a4391b7c7248cdca1d2536f402891a6b64335b7 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161760513) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=164) * e69e1972a5934aed08c7fa655babd53365243ad9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] alpinegizmo opened a new pull request #11898: [hotfix][docs] link tags in zh docs require .zh.md links
alpinegizmo opened a new pull request #11898:
URL: https://github.com/apache/flink/pull/11898
## What is the purpose of the change
The docs build is currently broken (https://ci.apache.org/builders) because
of this error (see
https://ci.apache.org/builders/flink-docs-master/builds/1811/steps/Build%20docs/logs/stdio):
Liquid Exception: Could not find document
'concepts/timely-stream-processing.md' in tag 'link'. Make sure the document
exists and the path is correct. in dev/event_time.zh.md
This is caused by {% link %} tags used in event_time.zh.md that aren't
linking to the chinese docs.
## Brief change log
Changed links to point to the .zh.md versions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
tillrohrmann commented on a change in pull request #11323:
URL: https://github.com/apache/flink/pull/11323#discussion_r414358295
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
##
@@ -320,5 +333,16 @@ private void internalStopPod(String podName) {
}
}
);
+
+ final KubernetesWorkerNode kubernetesWorkerNode =
workerNodes.remove(resourceId);
+ final WorkerResourceSpec workerResourceSpec =
podWorkerResources.remove(podName);
+
+ // If the stopped pod is requested in the current attempt
(workerResourceSpec is known) and is not yet added,
+ // we need to notify ActiveResourceManager to decrease the
pending worker count.
+ if (workerResourceSpec != null && kubernetesWorkerNode == null)
{
Review comment:
In the current state with calling `requestKubernetesPodIfRequired` it
should now work.
I think we are still lacking a bit of test coverage, though. For example if
a recovered pod is being used by a job and if the pod now fails, it should be
restarted because the `SlotManager` needs the pod.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-13553) KvStateServerHandlerTest.readInboundBlocking unstable on Travis
[
https://issues.apache.org/jira/browse/FLINK-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091273#comment-17091273
]
Andrey Zagrebin commented on FLINK-13553:
-
This looks related:
[https://travis-ci.org/github/azagrebin/flink/jobs/678649572]
{code:java}
[ERROR]
testQueryUnknownKey(org.apache.flink.queryablestate.network.KvStateServerHandlerTest)
Time elapsed: 10.027 s <<< ERROR!
java.util.concurrent.TimeoutException
at
org.apache.flink.queryablestate.network.KvStateServerHandlerTest.readInboundBlocking(KvStateServerHandlerTest.java:732)
at
org.apache.flink.queryablestate.network.KvStateServerHandlerTest.testQueryUnknownKey(KvStateServerHandlerTest.java:277){code}
> KvStateServerHandlerTest.readInboundBlocking unstable on Travis
> ---
>
> Key: FLINK-13553
> URL: https://issues.apache.org/jira/browse/FLINK-13553
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Gary Yao
>Priority: Critical
> Labels: pull-request-available, test-stability
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{KvStateServerHandlerTest.readInboundBlocking}} and
> {{KvStateServerHandlerTest.testQueryExecutorShutDown}} fail on Travis with a
> {{TimeoutException}}.
> https://api.travis-ci.org/v3/job/566420641/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-13553) KvStateServerHandlerTest.readInboundBlocking unstable on Travis
[
https://issues.apache.org/jira/browse/FLINK-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091273#comment-17091273
]
Andrey Zagrebin edited comment on FLINK-13553 at 4/24/20, 7:31 AM:
---
This looks related (failed in my travis, passed in PR):
[https://travis-ci.org/github/azagrebin/flink/jobs/678649572]
{code:java}
[ERROR]
testQueryUnknownKey(org.apache.flink.queryablestate.network.KvStateServerHandlerTest)
Time elapsed: 10.027 s <<< ERROR!
java.util.concurrent.TimeoutException
at
org.apache.flink.queryablestate.network.KvStateServerHandlerTest.readInboundBlocking(KvStateServerHandlerTest.java:732)
at
org.apache.flink.queryablestate.network.KvStateServerHandlerTest.testQueryUnknownKey(KvStateServerHandlerTest.java:277){code}
was (Author: azagrebin):
This looks related:
[https://travis-ci.org/github/azagrebin/flink/jobs/678649572]
{code:java}
[ERROR]
testQueryUnknownKey(org.apache.flink.queryablestate.network.KvStateServerHandlerTest)
Time elapsed: 10.027 s <<< ERROR!
java.util.concurrent.TimeoutException
at
org.apache.flink.queryablestate.network.KvStateServerHandlerTest.readInboundBlocking(KvStateServerHandlerTest.java:732)
at
org.apache.flink.queryablestate.network.KvStateServerHandlerTest.testQueryUnknownKey(KvStateServerHandlerTest.java:277){code}
> KvStateServerHandlerTest.readInboundBlocking unstable on Travis
> ---
>
> Key: FLINK-13553
> URL: https://issues.apache.org/jira/browse/FLINK-13553
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Gary Yao
>Priority: Critical
> Labels: pull-request-available, test-stability
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{KvStateServerHandlerTest.readInboundBlocking}} and
> {{KvStateServerHandlerTest.testQueryExecutorShutDown}} fail on Travis with a
> {{TimeoutException}}.
> https://api.travis-ci.org/v3/job/566420641/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #11898: [hotfix][docs] link tags in zh docs require .zh.md links
flinkbot commented on pull request #11898: URL: https://github.com/apache/flink/pull/11898#issuecomment-618853417 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d09fd9aa759eab5107de9c147ff6d70bfa3a29a4 (Fri Apr 24 07:31:43 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17314) 【Flink Kafka Connector】The Config（connector.topic）support list topic
[ https://issues.apache.org/jira/browse/FLINK-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091289#comment-17091289 ] zhisheng commented on FLINK-17314: -- [~jark] Kafka connector also support list topics, and is also easily expose it in SQL DDL, eg: one topic：'connector.topic'='test-topic' list topic：test1-topic;test2-topic list topic：test-topic-[0-9] > 【Flink Kafka Connector】The Config（connector.topic）support list topic > > > Key: FLINK-17314 > URL: https://issues.apache.org/jira/browse/FLINK-17314 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Table SQL / Ecosystem >Reporter: zhisheng >Priority: Major > > sometime we may consume from more than one topic, and the data schema in all > topic is same -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17365) Update Dockerfiles to 1.9.3
[ https://issues.apache.org/jira/browse/FLINK-17365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-17365. Fix Version/s: docker-1.9.3.1 Resolution: Fixed docker-master: dbbbfbaa129037ae37a102da85554436b083691b > Update Dockerfiles to 1.9.3 > --- > > Key: FLINK-17365 > URL: https://issues.apache.org/jira/browse/FLINK-17365 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: docker-1.9.3.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17214) Docker entrypoint scripts may print credentials
[
https://issues.apache.org/jira/browse/FLINK-17214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-17214.
Resolution: Fixed
> Docker entrypoint scripts may print credentials
> ---
>
> Key: FLINK-17214
> URL: https://issues.apache.org/jira/browse/FLINK-17214
> Project: Flink
> Issue Type: Bug
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: docker-1.10.1.1, docker-1.11.0, docker-1.9.3.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The docker entrypoint scripts run
> {code}
> echo "config file: " && grep '^[^\n#]' "${CONF_FILE}"}}
> {code}
> which prints the entire configuration.
> This may leak credentials set in the configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17214) Docker entrypoint scripts may print credentials
[
https://issues.apache.org/jira/browse/FLINK-17214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-17214:
-
Fix Version/s: (was: docker-1.9.X)
docker-1.9.3.1
> Docker entrypoint scripts may print credentials
> ---
>
> Key: FLINK-17214
> URL: https://issues.apache.org/jira/browse/FLINK-17214
> Project: Flink
> Issue Type: Bug
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: docker-1.10.1.1, docker-1.11.0, docker-1.9.3.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The docker entrypoint scripts run
> {code}
> echo "config file: " && grep '^[^\n#]' "${CONF_FILE}"}}
> {code}
> which prints the entire configuration.
> This may leak credentials set in the configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17187) Streamline setting of config options
[
https://issues.apache.org/jira/browse/FLINK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-17187.
Fix Version/s: (was: 1.11.0)
docker-1.9.3.0
docker-1.11.0.0
docker-1.10.1.0
Resolution: Fixed
docker-master: 379970e8d2a9e138d1291d83b47e7ea643421b3a
> Streamline setting of config options
>
>
> Key: FLINK-17187
> URL: https://issues.apache.org/jira/browse/FLINK-17187
> Project: Flink
> Issue Type: Improvement
> Components: Dockerfiles
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: docker-1.10.1.0, docker-1.11.0.0, docker-1.9.3.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The docker-entrypoint.sh has some verbose sections for setting config options
> with lots of duplicated logic. We should introduce a utility method for
> setting config options.
>
> {code:java}
> if grep -E "^jobmanager\.rpc\.address:.*" "${CONF_FILE}" > /dev/null; then
> sed -i -e "s/jobmanager\.rpc\.address:.*/jobmanager.rpc.address:
> ${JOB_MANAGER_RPC_ADDRESS}/g" "${CONF_FILE}"
> else
> echo "jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}" >>
> "${CONF_FILE}"
> fiif grep -E "^blob\.server\.port:.*" "${CONF_FILE}" > /dev/null; then
> sed -i -e "s/blob\.server\.port:.*/blob.server.port: 6124/g"
> "${CONF_FILE}"
> else
> echo "blob.server.port: 6124" >> "${CONF_FILE}"
> fi
> ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint
flinkbot edited a comment on pull request #11687: URL: https://github.com/apache/flink/pull/11687#issuecomment-611445542 ## CI report: * 5bad018f00f85a7359345187a12d7938aa510d25 UNKNOWN * d51ce7f47381d99b843278cd701dcff223761a0b UNKNOWN * cba096ae3d8eba4a0d39c64659f76ad10a62be27 UNKNOWN * 23818c4b30eb9714154080253a090fc63d983925 UNKNOWN * 9b6278f15a4fc65d78671a145305df2c7310cc6a UNKNOWN * 4f7399680818f5d29e917e17720e00900822a43d UNKNOWN * 12c398dd2dc5b6716a590490575914698573ab76 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161752886) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=157) * 95dc7d6ada34179b51849d905f15c740ca585a8d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161778166) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=172) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17365) Update Dockerfiles to 1.9.3
[ https://issues.apache.org/jira/browse/FLINK-17365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091296#comment-17091296 ] Chesnay Schepler edited comment on FLINK-17365 at 4/24/20, 7:45 AM: docker-master: dbbbfbaa129037ae37a102da85554436b083691b e7e19032d43fc8d93cb3763faf32f1bff9df891d was (Author: zentol): docker-master: dbbbfbaa129037ae37a102da85554436b083691b > Update Dockerfiles to 1.9.3 > --- > > Key: FLINK-17365 > URL: https://issues.apache.org/jira/browse/FLINK-17365 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: docker-1.9.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11863: [FLINK-17048][mesos] Add memory related JVM args to Mesos JM startup scripts
flinkbot edited a comment on pull request #11863: URL: https://github.com/apache/flink/pull/11863#issuecomment-617687609 ## CI report: * d61df619ede73e8234f2be3cd4adac63c8ce1f19 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161659332) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=139) * b0b080e3a5efb7651d6fa032521985585d9e333e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan opened a new pull request #11899: [FLINK-17342] enqueue savepoint requests in UC mode
rkhachatryan opened a new pull request #11899: URL: https://github.com/apache/flink/pull/11899 ## What is the purpose of the change Current behavior of forcing savepoints causes excess of max-concurrent-checkpoints limit. Which violates current Unaligned Checkpoints (UC) assumption of a single concurrent barrier. ## Brief change log 1. Remove periodicTriggeringSuspended. Instead, if periodic request can't be executed now is ignored. 2. Check queue before executing a new request (not after). 3. Execute queued request when pending checkpoint is completed (instead of resuming timer). 4. Don't set CheckpointRequest.force in UC mode. 5. Change requests queue to PriorityQueue to prioritize savepoints. Limit its size. 6. Move all request-related decision logic into one place (and one sync block). 7. For savepoints, if running in UC mode and limit is reached then enqueue request instead of forcing it 8. Don't throw trigger exceptions. CheckpointFailureManager ignores them ## Verifying this change This change added tests and can be verified as follows: - Added `CheckpointCoordinatorTest.testSavepointScheduledInUnalignedMode` to test new CheckpointCoordinator behavior (savepoints in UC mode) - Added `CheckpointRequestDeciderTest` to test decision logic in isolation ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11829: [FLINK-17021][table-planner-blink] Blink batch planner set GlobalDataExchangeMode
flinkbot edited a comment on pull request #11829: URL: https://github.com/apache/flink/pull/11829#issuecomment-616561190 ## CI report: * c15a8214d2883f5a5c07bb6357dbfa465dd5cd39 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161357311) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18) * 916148ed89d76a9b2fa7659941023f473ecb38a2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11853: [FLINK-15006][table-planner] Add option to shuffle-by-partition when dynamic inserting
flinkbot edited a comment on pull request #11853: URL: https://github.com/apache/flink/pull/11853#issuecomment-617586421 ## CI report: * 258a96c64c409102c679382b054d17602df65953 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161760537) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=165) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11837: [FLINK-16160][table-planner-blink] Fix proctime()/rowtime() doesn't w…
flinkbot edited a comment on pull request #11837: URL: https://github.com/apache/flink/pull/11837#issuecomment-616956896 ## CI report: * 9a4391b7c7248cdca1d2536f402891a6b64335b7 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161760513) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=164) * e69e1972a5934aed08c7fa655babd53365243ad9 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161778295) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=173) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #11895: [FLINK-10911] Flink's flink-scala-shell is not working with Scala 2.12
zentol commented on pull request #11895: URL: https://github.com/apache/flink/pull/11895#issuecomment-618859936 Can you give a quick note on what the goal of this PR is? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #11898: [hotfix][docs] fix broken build: link tags in zh docs require .zh.md links
flinkbot commented on pull request #11898: URL: https://github.com/apache/flink/pull/11898#issuecomment-618860022 ## CI report: * d09fd9aa759eab5107de9c147ff6d70bfa3a29a4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11884: [FLINK-17345][python][table] Support register and get Python UDF in Catalog.
flinkbot edited a comment on pull request #11884: URL: https://github.com/apache/flink/pull/11884#issuecomment-618376627 ## CI report: * 570f7e22326fc2f2d9f7dff4066192b604e7c479 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/161749776) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=154) * 3579637b38484fb5589c309fea8e803ad3c2e4e0 UNKNOWN * 78e23cbb5c0ec424f89a83bffa5146e428237fea UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-10911) Flink's flink-scala-shell is not working with Scala 2.12
[
https://issues.apache.org/jira/browse/FLINK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reassigned FLINK-10911:
Assignee: Jeff Zhang
> Flink's flink-scala-shell is not working with Scala 2.12
>
>
> Key: FLINK-10911
> URL: https://issues.apache.org/jira/browse/FLINK-10911
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Jeff Zhang
>Priority: Major
> Labels: pull-request-available
>
> Flink's {{flink-scala-shell}} module is not working with Scala 2.12.
> Therefore, it is currently excluded from the Scala 2.12 builds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17342) Schedule savepoint if max-inflight-checkpoints limit is reached isntead of forcing (in UC mode)
[ https://issues.apache.org/jira/browse/FLINK-17342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17342: --- Labels: pull-request-available (was: ) > Schedule savepoint if max-inflight-checkpoints limit is reached isntead of > forcing (in UC mode) > --- > > Key: FLINK-17342 > URL: https://issues.apache.org/jira/browse/FLINK-17342 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.11.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #11899: [FLINK-17342] enqueue savepoint requests in UC mode
flinkbot commented on pull request #11899: URL: https://github.com/apache/flink/pull/11899#issuecomment-618860667 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8ced5a3c7d94080f7f34588721886653bf334060 (Fri Apr 24 07:49:22 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-docker] azagrebin commented on a change in pull request #15: [FLINK-17346] De-duplicate process setup
azagrebin commented on a change in pull request #15:
URL: https://github.com/apache/flink-docker/pull/15#discussion_r414367906
##
File path: testing/testing_lib.sh
##
@@ -45,51 +45,42 @@ function build_image() {
docker build -t "$image_name" "$dockerfile_dir"
}
-function run_jobmanager() {
-local dockerfile
-dockerfile="$1"
+function internal_run() {
+local dockerfile="$1"
+local docker_run_command="$2"
+local args="$3"
local image_tag image_name
image_tag="$(image_tag "$dockerfile")"
image_name="$(image_name "$image_tag")"
-echo >&2 "===> Starting ${image_tag} jobmanager..."
-
-# Prints container ID
-docker run \
---rm \
---detach \
---name "jobmanager" \
---network "$NETWORK_NAME" \
---publish 6123:6123 \
---publish 8081:8081 \
--e JOB_MANAGER_RPC_ADDRESS="jobmanager" \
-"$image_name" \
-jobmanager
+echo >&2 "===> Starting ${image_tag} ${args}..."
+
+eval "docker run --rm --detach --network $NETWORK_NAME -e
JOB_MANAGER_RPC_ADDRESS=jobmanager ${docker_run_command} $image_name ${args}"
+}
+
+function internal_run_jobmanager() {
+internal_run "$1" "--name jobmanager --publish 6123:6123 --publish
8081:8081 $2" jobmanager
+}
+
+function run_jobmanager() {
+internal_run_jobmanager "$1" ""
Review comment:
do we need `""` at the end?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-17366) Implement listViews
Flavio Pompermaier created FLINK-17366: -- Summary: Implement listViews Key: FLINK-17366 URL: https://issues.apache.org/jira/browse/FLINK-17366 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: Flavio Pompermaier But how to set as read-only Table..? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17367) Support generating dockerfiles against custom URL
Chesnay Schepler created FLINK-17367: Summary: Support generating dockerfiles against custom URL Key: FLINK-17367 URL: https://issues.apache.org/jira/browse/FLINK-17367 Project: Flink Issue Type: Improvement Components: Dockerfiles Reporter: Chesnay Schepler Assignee: Chesnay Schepler The Dockerfiles are currently hard-wired to work against the apache mirrors to retrieve official releases. In order to test features against Flink-SNAPSHOT, and as a generally useful dev feature, we should allow generating the images against an arbitrary url. In the case of Flink-SNAPSHOT, these would for example be the S3 bucket we upload the snapshot binaries to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise commented on a change in pull request #11725: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
AHeise commented on a change in pull request #11725:
URL: https://github.com/apache/flink/pull/11725#discussion_r414363269
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaShuffleProducer.java
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.annotation.Internal;
+import
org.apache.flink.api.common.serialization.TypeInformationSerializationSchema;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.KeyGroupRangeAssignment;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.PropertiesUtil;
+
+import org.apache.kafka.clients.producer.ProducerRecord;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Properties;
+
+import static
org.apache.flink.streaming.connectors.kafka.FlinkKafkaShuffle.PARTITION_NUMBER;
+
+/**
+ * Flink Kafka Shuffle Producer Function.
+ * It is different from {@link FlinkKafkaProducer} in the way handling
elements and watermarks
+ */
+@Internal
+public class FlinkKafkaShuffleProducer extends FlinkKafkaProducer
{
+ private final KafkaSerializer kafkaSerializer;
+ private final KeySelector keySelector;
+ private final int numberOfPartitions;
+
+ FlinkKafkaShuffleProducer(
+ String defaultTopicId,
+ TypeInformationSerializationSchema schema,
+ Properties props,
+ KeySelector keySelector,
+ Semantic semantic,
+ int kafkaProducersPoolSize) {
+ super(defaultTopicId, (element, timestamp) -> null, props,
semantic, kafkaProducersPoolSize);
+
+ this.kafkaSerializer = new
KafkaSerializer<>(schema.getSerializer());
+ this.keySelector = keySelector;
+
+ Preconditions.checkArgument(
+ props.getProperty(PARTITION_NUMBER) != null,
+ "Missing partition number for Kafka Shuffle");
+ numberOfPartitions = PropertiesUtil.getInt(props,
PARTITION_NUMBER, Integer.MIN_VALUE);
+ }
+
+ /**
+* This is the function invoked to handle each element.
+* @param transaction transaction state;
+*elements are written to Kafka in transactions to
guarantee different level of data consistency
+* @param next element to handle
+* @param context context needed to handle the element
+* @throws FlinkKafkaException
+*/
+ @Override
+ public void invoke(KafkaTransactionState transaction, IN next, Context
context) throws FlinkKafkaException {
+ checkErroneous();
+
+ // write timestamp to Kafka if timestamp is available
+ Long timestamp = context.timestamp();
+
+ int[] partitions = getPartitions(transaction);
+ int partitionIndex;
+ try {
+ partitionIndex = KeyGroupRangeAssignment
+
.assignKeyToParallelOperator(keySelector.getKey(next), partitions.length,
partitions.length);
+ } catch (Exception e) {
+ throw new RuntimeException("Fail to assign a partition
number to record");
+ }
+
+ ProducerRecord record = new ProducerRecord<>(
+ defaultTopicId, partitionIndex, timestamp, null,
kafkaSerializer.serializeRecord(next, timestamp));
+ pendingRecords.incrementAndGet();
+ transaction.producer.send(record, callback);
+ }
+
+ /**
+* This is the function invoked to handle each watermark.
+* @param transaction transaction state;
+*watermark are written to Kafka (if needed) in
transactions
+* @param watermark watermark to handle
+* @throws FlinkKafkaException
+*/
+ @Over
[jira] [Commented] (FLINK-17312) Support sql client savepoint
[
https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091310#comment-17091310
]
lun zhang commented on FLINK-17312:
---
Thanks for your replay.I have build a `sql client` python program where you
can write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in `sql client`. You can stop with
savepoint in `sql client` by running `flink stop -s savepoint jobid`. But we
can't use this `savepoint` in `sql client` command line. So I pull a request
add `savepoint` support in yml file. like
envirment.yml
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
{{}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
> Support sql client savepoint
>
>
> Key: FLINK-17312
> URL: https://issues.apache.org/jira/browse/FLINK-17312
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0, 1.11.0
>Reporter: lun zhang
>Priority: Major
> Labels: pull-request-available
>
> Sql client not support sql job savepoint current. It's important when you
> use this in really world.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zjffdu commented on pull request #11895: [FLINK-10911] Flink's flink-scala-shell is not working with Scala 2.12 [WIP]
zjffdu commented on pull request #11895: URL: https://github.com/apache/flink/pull/11895#issuecomment-618865450 @zentol It is WIP PR, I can run scala shell 2.12 successfully in my local machine. I just want to run it in CI to check whether it can pass CI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17368) exception message in PrometheusPushGatewayReporter
zl created FLINK-17368:
--
Summary: exception message in PrometheusPushGatewayReporter
Key: FLINK-17368
URL: https://issues.apache.org/jira/browse/FLINK-17368
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics
Affects Versions: 1.10.0
Reporter: zl
when sending flink metrics to prometheus pushgateway by using
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter, there are a
lof of exception message in taskmanager log. Here is the exception stack:
{code:java}
2020-04-23 18:16:44,927 WARN
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter - Failed to
push metrics to PushGateway with jobName a517f2f8bb79b59abb5e596f34adca27,
groupingKey {}.
java.io.IOException: Response code from
http://10.3.71.136:9091/metrics/job/a517f2f8bb79b59abb5e596f34adca27 was 200
at
org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
at
org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:127)
at
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:109)
at
org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:441)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
After investigation, I found that it's
io.prometheus:simpleclient_pushgateway:0.3.0 which casues the exception. before
io.prometheus:simpleclient_pushgateway:0.8.0,
io.prometheus.client.exporter.PushGateway#doRequest use response code 202 to
decide whether a metric is successfully sended or not, so response code 200
indicates a failed transmission. In
io.prometheus:simpleclient_pushgateway:0.8.0, response code 2xx is used to
indicates a successful transmission.
After we change the version of io.prometheus:simpleclient_pushgateway to 0.8.0
in flink-metrics-prometheus module, there have been no more such exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17312) Support sql client savepoint
[
https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091310#comment-17091310
]
lun zhang edited comment on FLINK-17312 at 4/24/20, 8:02 AM:
-
Thanks for your reply.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
[~ykt836]
was (Author: zhanglun):
Thanks for your replay.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
[~ykt836]
> Support sql client savepoint
>
>
> Key: FLINK-17312
> URL: https://issues.apache.org/jira/browse/FLINK-17312
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0, 1.11.0
>Reporter: lun zhang
>Priority: Major
> Labels: pull-request-available
>
> Sql client not support sql job savepoint current. It's important when you
> use this in really world.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16423) test_ha_per_job_cluster_datastream.sh gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091312#comment-17091312
]
Robert Metzger commented on FLINK-16423:
Another case: https://api.travis-ci.org/v3/job/678609505/log.txt
> test_ha_per_job_cluster_datastream.sh gets stuck
>
>
> Key: FLINK-16423
> URL: https://issues.apache.org/jira/browse/FLINK-16423
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Attachments: 20200408.1.tgz
>
>
> This was seen in
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5905&view=logs&j=b1623ac9-0979-5b0d-2e5e-1377d695c991&t=e7804547-1789-5225-2bcf-269eeaa37447
> ... the relevant part of the logs is here:
> {code}
> 2020-03-04T11:27:25.4819486Z
> ==
> 2020-03-04T11:27:25.4820470Z Running 'Running HA per-job cluster (rocks,
> non-incremental) end-to-end test'
> 2020-03-04T11:27:25.4820922Z
> ==
> 2020-03-04T11:27:25.4840177Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-25482960156
> 2020-03-04T11:27:25.6712478Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-03-04T11:27:25.6830402Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-03-04T11:27:26.2988914Z Starting zookeeper daemon on host fv-az655.
> 2020-03-04T11:27:26.3001237Z Running on HA mode: parallelism=4,
> backend=rocks, asyncSnapshots=true, and incremSnapshots=false.
> 2020-03-04T11:27:27.4206924Z Starting standalonejob daemon on host fv-az655.
> 2020-03-04T11:27:27.4217066Z Start 1 more task managers
> 2020-03-04T11:27:30.8412541Z Starting taskexecutor daemon on host fv-az655.
> 2020-03-04T11:27:38.1779980Z Job () is
> running.
> 2020-03-04T11:27:38.1781375Z Running JM watchdog @ 89778
> 2020-03-04T11:27:38.1781858Z Running TM watchdog @ 89779
> 2020-03-04T11:27:38.1783272Z Waiting for text Completed checkpoint [1-9]* for
> job to appear 2 of times in logs...
> 2020-03-04T13:21:29.9076797Z ##[error]The operation was canceled.
> 2020-03-04T13:21:29.9094090Z ##[section]Finishing: Run e2e tests
> {code}
> The last three lines indicate that the test is waiting forever for a
> checkpoint to appear.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17312) Support sql client savepoint
[
https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091310#comment-17091310
]
lun zhang edited comment on FLINK-17312 at 4/24/20, 8:02 AM:
-
Thanks for your replay.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
[~ykt836]
was (Author: zhanglun):
Thanks for your replay.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
> Support sql client savepoint
>
>
> Key: FLINK-17312
> URL: https://issues.apache.org/jira/browse/FLINK-17312
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0, 1.11.0
>Reporter: lun zhang
>Priority: Major
> Labels: pull-request-available
>
> Sql client not support sql job savepoint current. It's important when you
> use this in really world.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17312) Support sql client savepoint
[
https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091310#comment-17091310
]
lun zhang edited comment on FLINK-17312 at 4/24/20, 8:02 AM:
-
Thanks for your replay.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
was (Author: zhanglun):
Thanks for your replay.I have build a `sql client` python program where you
can write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in `sql client`. You can stop with
savepoint in `sql client` by running `flink stop -s savepoint jobid`. But we
can't use this `savepoint` in `sql client` command line. So I pull a request
add `savepoint` support in yml file. like
envirment.yml
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
{{}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
> Support sql client savepoint
>
>
> Key: FLINK-17312
> URL: https://issues.apache.org/jira/browse/FLINK-17312
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0, 1.11.0
>Reporter: lun zhang
>Priority: Major
> Labels: pull-request-available
>
> Sql client not support sql job savepoint current. It's important when you
> use this in really world.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17312) Support sql client savepoint
[
https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091310#comment-17091310
]
lun zhang edited comment on FLINK-17312 at 4/24/20, 8:03 AM:
-
Thanks for your reply.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help sql client build a high availability sql
job. I've already tests my code in flink-1.10.It's very helpful when you can
stop and restart you `sql job` by savepoint.
[~ykt836]
was (Author: zhanglun):
Thanks for your reply.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help `sql client` build a high availability
`sql job`. I'm already tests my code in `flink-1.10`.It's very helpful when you
can stop and restart you `sql job` by savepoint.
[~ykt836]
> Support sql client savepoint
>
>
> Key: FLINK-17312
> URL: https://issues.apache.org/jira/browse/FLINK-17312
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0, 1.11.0
>Reporter: lun zhang
>Priority: Major
> Labels: pull-request-available
>
> Sql client not support sql job savepoint current. It's important when you
> use this in really world.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17368) exception message in PrometheusPushGatewayReporter
[
https://issues.apache.org/jira/browse/FLINK-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-17368.
Resolution: Duplicate
> exception message in PrometheusPushGatewayReporter
> ---
>
> Key: FLINK-17368
> URL: https://issues.apache.org/jira/browse/FLINK-17368
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
>Affects Versions: 1.10.0
>Reporter: zl
>Priority: Minor
>
> when sending flink metrics to prometheus pushgateway by using
> org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter, there are
> a lof of exception message in taskmanager log. Here is the exception stack:
> {code:java}
> 2020-04-23 18:16:44,927 WARN
> org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter - Failed
> to push metrics to PushGateway with jobName a517f2f8bb79b59abb5e596f34adca27,
> groupingKey {}.
> java.io.IOException: Response code from
> http://10.3.71.136:9091/metrics/job/a517f2f8bb79b59abb5e596f34adca27 was 200
> at
> org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:297)
> at
> org.apache.flink.shaded.io.prometheus.client.exporter.PushGateway.push(PushGateway.java:127)
> at
> org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter.report(PrometheusPushGatewayReporter.java:109)
> at
> org.apache.flink.runtime.metrics.MetricRegistryImpl$ReporterTask.run(MetricRegistryImpl.java:441)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
>
> After investigation, I found that it's
> io.prometheus:simpleclient_pushgateway:0.3.0 which casues the exception.
> before io.prometheus:simpleclient_pushgateway:0.8.0,
> io.prometheus.client.exporter.PushGateway#doRequest use response code 202 to
> decide whether a metric is successfully sended or not, so response code 200
> indicates a failed transmission. In
> io.prometheus:simpleclient_pushgateway:0.8.0, response code 2xx is used to
> indicates a successful transmission.
>
> After we change the version of io.prometheus:simpleclient_pushgateway to
> 0.8.0 in flink-metrics-prometheus module, there have been no more such
> exception.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] KarmaGYZ commented on pull request #11854: [WIP] Introduce external resource framework
KarmaGYZ commented on pull request #11854: URL: https://github.com/apache/flink/pull/11854#issuecomment-618867585 Thanks for the review @xintongsong . I've updated the PR and replied to some of the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11829: [FLINK-17021][table-planner-blink] Blink batch planner set GlobalDataExchangeMode
flinkbot edited a comment on pull request #11829: URL: https://github.com/apache/flink/pull/11829#issuecomment-616561190 ## CI report: * c15a8214d2883f5a5c07bb6357dbfa465dd5cd39 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161357311) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18) * 916148ed89d76a9b2fa7659941023f473ecb38a2 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161780756) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=174) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17330) Avoid scheduling deadlocks caused by cyclic input dependencies between regions
[
https://issues.apache.org/jira/browse/FLINK-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091315#comment-17091315
]
Till Rohrmann commented on FLINK-17330:
---
>> I think it's possible but it may be hard for users to identify whether there
>> are cyclic dependencies. Most users will have to choose the mode to set all
>> edges BLOCKING to be safe and lose the benefit of pipelined region
>> scheduling. So if we'd like to take it this way, I think it's better we do
>> it automatically for users, i.e. override GlobalDataExchangeMode to be
>> ALL_EDGES_BLOCKING if cyclic dependency is detected.
I would actually suggest to throw an exception with the hint to set
{{GlobalDataExchangeMode}} to {{ALL_EDGES_BLOCKING}}. It is not a perfect
solution but it reduces complexity because we don't have to implement some
magic which might be surprising and won't last very long.
I think the main question is whether we consider this feature to be required
for the MVP or not. I believe that even with this limitation we will add
additional value for our users because in many cases they won't be affected. If
they are affected, then they have clear instructions how to work around the
problem. Moreover, it could also be possible that we are actually able to solve
this problem after the MVP has been completed and before the release in which
the MVP will be shipped. That way nobody will be affected. If we do it the
other way around (fixing this problem for the MVP to complete) we might risk
missing a release and hence not shipping improvements to the user. I guess I'm
mainly arguing from a project management point of view here by trying to keep
the scope as small as possible and advocating for incremental steps.
>> 2. how to detect cyclic dependencies? Checking whether there are
>> intra-region all-to-all blocking edges can be a performance efficient
>> solution but is not the only choice, and it also requires attention to
>> POINTWISE edges. If we can have a common way to find out cyclic dependencies
>> in O(V^2), I think it's even better. This question can be answered later
>> when we have a deeper look at all the options.
I agree. I think we need to take a look at possible algorithms. Maybe [Tarjan's
strongly connected components
algorithm|https://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm]
could be a good fit for the task.
> Avoid scheduling deadlocks caused by cyclic input dependencies between regions
> --
>
> Key: FLINK-17330
> URL: https://issues.apache.org/jira/browse/FLINK-17330
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> Imagine a job like this:
> A -- (pipelined FORWARD) --> B -- (blocking ALL-to-ALL) --> D
> A -- (pipelined FORWARD) --> C -- (pipelined FORWARD) --> D
> parallelism=2 for all vertices.
> We will have 2 execution pipelined regions:
> R1 = {A1, B1, C1, D1}
> R2 = {A2, B2, C2, D2}
> R1 has a cross-region input edge (B2->D1).
> R2 has a cross-region input edge (B1->D2).
> Scheduling deadlock will happen since we schedule a region only when all its
> inputs are consumable (i.e. blocking partitions to be finished). This is
> because R1 can be scheduled only if R2 finishes, while R2 can be scheduled
> only if R1 finishes.
> To avoid this, one solution is to force a logical pipelined region with
> intra-region ALL-to-ALL blocking edges to form one only execution pipelined
> region, so that there would not be cyclic input dependency between regions.
> Besides that, we should also pay attention to avoid cyclic cross-region
> POINTWISE blocking edges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11898: [hotfix][docs] fix broken build: link tags in zh docs require .zh.md links
flinkbot edited a comment on pull request #11898: URL: https://github.com/apache/flink/pull/11898#issuecomment-618860022 ## CI report: * d09fd9aa759eab5107de9c147ff6d70bfa3a29a4 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161780872) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=177) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11895: [FLINK-10911] Flink's flink-scala-shell is not working with Scala 2.12 [WIP]
flinkbot edited a comment on pull request #11895: URL: https://github.com/apache/flink/pull/11895#issuecomment-618799215 ## CI report: * 191da9e8a608bde3657193779bc9f0a48350f36d Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161760585) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=166) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=161) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11796: [FLINK-14258][table][filesystem] Integrate file system connector to streaming sink
flinkbot edited a comment on pull request #11796: URL: https://github.com/apache/flink/pull/11796#issuecomment-615254717 ## CI report: * 4d98d46528fab8087fccf79b9184f27a6197c714 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161760487) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=163) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11863: [FLINK-17048][mesos] Add memory related JVM args to Mesos JM startup scripts
flinkbot edited a comment on pull request #11863: URL: https://github.com/apache/flink/pull/11863#issuecomment-617687609 ## CI report: * d61df619ede73e8234f2be3cd4adac63c8ce1f19 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/161659332) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=139) * b0b080e3a5efb7651d6fa032521985585d9e333e Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161780807) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=175) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11884: [FLINK-17345][python][table] Support register and get Python UDF in Catalog.
flinkbot edited a comment on pull request #11884: URL: https://github.com/apache/flink/pull/11884#issuecomment-618376627 ## CI report: * 570f7e22326fc2f2d9f7dff4066192b604e7c479 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/161749776) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=154) * 3579637b38484fb5589c309fea8e803ad3c2e4e0 UNKNOWN * 78e23cbb5c0ec424f89a83bffa5146e428237fea Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161780846) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=176) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17330) Avoid scheduling deadlocks caused by cyclic input dependencies between regions
[
https://issues.apache.org/jira/browse/FLINK-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091315#comment-17091315
]
Till Rohrmann edited comment on FLINK-17330 at 4/24/20, 8:06 AM:
-
>> I think it's possible but it may be hard for users to identify whether there
>> are cyclic dependencies. Most users will have to choose the mode to set all
>> edges BLOCKING to be safe and lose the benefit of pipelined region
>> scheduling. So if we'd like to take it this way, I think it's better we do
>> it automatically for users, i.e. override GlobalDataExchangeMode to be
>> ALL_EDGES_BLOCKING if cyclic dependency is detected.
I would actually suggest to throw an exception with the hint to set
{{GlobalDataExchangeMode}} to {{ALL_EDGES_BLOCKING}}. It is not a perfect
solution but it reduces complexity because we don't have to implement some
magic which might be surprising and won't last very long. It depends also a bit
on how difficult/complex it would be to implement such an automatic fall back.
I think the main question is whether we consider this feature to be required
for the MVP or not. I believe that even with this limitation we will add
additional value for our users because in many cases they won't be affected. If
they are affected, then they have clear instructions how to work around the
problem. Moreover, it could also be possible that we are actually able to solve
this problem after the MVP has been completed and before the release in which
the MVP will be shipped. That way nobody will be affected. If we do it the
other way around (fixing this problem for the MVP to complete) we might risk
missing a release and hence not shipping improvements to the user. I guess I'm
mainly arguing from a project management point of view here by trying to keep
the scope as small as possible and advocating for incremental steps.
>> 2. how to detect cyclic dependencies? Checking whether there are
>> intra-region all-to-all blocking edges can be a performance efficient
>> solution but is not the only choice, and it also requires attention to
>> POINTWISE edges. If we can have a common way to find out cyclic dependencies
>> in O(V^2), I think it's even better. This question can be answered later
>> when we have a deeper look at all the options.
I agree. I think we need to take a look at possible algorithms. Maybe [Tarjan's
strongly connected components
algorithm|https://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm]
could be a good fit for the task.
was (Author: till.rohrmann):
>> I think it's possible but it may be hard for users to identify whether there
>> are cyclic dependencies. Most users will have to choose the mode to set all
>> edges BLOCKING to be safe and lose the benefit of pipelined region
>> scheduling. So if we'd like to take it this way, I think it's better we do
>> it automatically for users, i.e. override GlobalDataExchangeMode to be
>> ALL_EDGES_BLOCKING if cyclic dependency is detected.
I would actually suggest to throw an exception with the hint to set
{{GlobalDataExchangeMode}} to {{ALL_EDGES_BLOCKING}}. It is not a perfect
solution but it reduces complexity because we don't have to implement some
magic which might be surprising and won't last very long.
I think the main question is whether we consider this feature to be required
for the MVP or not. I believe that even with this limitation we will add
additional value for our users because in many cases they won't be affected. If
they are affected, then they have clear instructions how to work around the
problem. Moreover, it could also be possible that we are actually able to solve
this problem after the MVP has been completed and before the release in which
the MVP will be shipped. That way nobody will be affected. If we do it the
other way around (fixing this problem for the MVP to complete) we might risk
missing a release and hence not shipping improvements to the user. I guess I'm
mainly arguing from a project management point of view here by trying to keep
the scope as small as possible and advocating for incremental steps.
>> 2. how to detect cyclic dependencies? Checking whether there are
>> intra-region all-to-all blocking edges can be a performance efficient
>> solution but is not the only choice, and it also requires attention to
>> POINTWISE edges. If we can have a common way to find out cyclic dependencies
>> in O(V^2), I think it's even better. This question can be answered later
>> when we have a deeper look at all the options.
I agree. I think we need to take a look at possible algorithms. Maybe [Tarjan's
strongly connected components
algorithm|https://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm]
could be a good fit for the task.
> Avoid scheduling deadlocks caused by cyclic input dependencies between r
[GitHub] [flink] flinkbot commented on pull request #11899: [FLINK-17342] enqueue savepoint requests in UC mode
flinkbot commented on pull request #11899: URL: https://github.com/apache/flink/pull/11899#issuecomment-618868456 ## CI report: * 8ced5a3c7d94080f7f34588721886653bf334060 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17312) Support sql client start with savepoint
[ https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lun zhang updated FLINK-17312: -- Summary: Support sql client start with savepoint (was: Support sql client savepoint) > Support sql client start with savepoint > --- > > Key: FLINK-17312 > URL: https://issues.apache.org/jira/browse/FLINK-17312 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0, 1.11.0 >Reporter: lun zhang >Priority: Major > Labels: pull-request-available > > Sql client not support *insert sql job* restart with *savepoint* > current.It's very helpful when you can stop your flink *insert sql job* and > restart with savepoint . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13553) KvStateServerHandlerTest.readInboundBlocking unstable on Travis
[
https://issues.apache.org/jira/browse/FLINK-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091317#comment-17091317
]
Till Rohrmann commented on FLINK-13553:
---
If this is the case, then we need to re-open this issue.
> KvStateServerHandlerTest.readInboundBlocking unstable on Travis
> ---
>
> Key: FLINK-13553
> URL: https://issues.apache.org/jira/browse/FLINK-13553
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Gary Yao
>Priority: Critical
> Labels: pull-request-available, test-stability
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{KvStateServerHandlerTest.readInboundBlocking}} and
> {{KvStateServerHandlerTest.testQueryExecutorShutDown}} fail on Travis with a
> {{TimeoutException}}.
> https://api.travis-ci.org/v3/job/566420641/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17312) Support sql client savepoint
[ https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lun zhang updated FLINK-17312: -- Description: Sql client not support *insert sql job* restart with *savepoint* current.It's very helpful when you can stop your flink *insert sql job* and restart with savepoint . (was: Sql client not support sql job savepoint current. It's important when you use this in really world. ) > Support sql client savepoint > > > Key: FLINK-17312 > URL: https://issues.apache.org/jira/browse/FLINK-17312 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Client >Affects Versions: 1.10.0, 1.11.0 >Reporter: lun zhang >Priority: Major > Labels: pull-request-available > > Sql client not support *insert sql job* restart with *savepoint* > current.It's very helpful when you can stop your flink *insert sql job* and > restart with savepoint . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17313) Validation error when insert decimal/varchar with precision into sink using TypeInformation of row
[
https://issues.apache.org/jira/browse/FLINK-17313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terry Wang updated FLINK-17313:
---
Summary: Validation error when insert decimal/varchar with precision into
sink using TypeInformation of row (was: Validation error when insert
decimal/timestamp/varchar with precision into sink using TypeInformation of row)
> Validation error when insert decimal/varchar with precision into sink using
> TypeInformation of row
> --
>
> Key: FLINK-17313
> URL: https://issues.apache.org/jira/browse/FLINK-17313
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Terry Wang
>Priority: Major
> Labels: pull-request-available
>
> Test code like follwing(in blink planner):
> {code:java}
> tEnv.sqlUpdate("create table randomSource (" +
> " a varchar(10),"
> +
> " b
> decimal(20,2)" +
> " ) with (" +
> " 'type' =
> 'random'," +
> " 'count' = '10'"
> +
> " )");
> tEnv.sqlUpdate("create table printSink (" +
> " a varchar(10),"
> +
> " b
> decimal(22,2)," +
> " c
> timestamp(3)," +
> " ) with (" +
> " 'type' = 'print'" +
> " )");
> tEnv.sqlUpdate("insert into printSink select *,
> current_timestamp from randomSource");
> tEnv.execute("");
> {code}
> Print TableSink implements UpsertStreamTableSink and it's getReocrdType is as
> following:
> {code:java}
> public TypeInformation getRecordType() {
> return getTableSchema().toRowType();
> }
> {code}
> Varchar column validation exception is:
> org.apache.flink.table.api.ValidationException: Type VARCHAR(10) of table
> field 'a' does not match with the physical type STRING of the 'a' field of
> the TableSink consumed type.
> at
> org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:165)
> at
> org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:278)
> at
> org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:255)
> at
> org.apache.flink.table.types.logical.utils.LogicalTypeDefaultVisitor.visit(LogicalTypeDefaultVisitor.java:67)
> at
> org.apache.flink.table.types.logical.VarCharType.accept(VarCharType.java:157)
> at
> org.apache.flink.table.utils.TypeMappingUtils.checkIfCompatible(TypeMappingUtils.java:255)
> at
> org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:161)
> at
> org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$validateLogicalPhysicalTypesCompatible$1.apply$mcVI$sp(TableSinkUtils.scala:315)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
> at
> org.apache.flink.table.planner.sinks.TableSinkUtils$.validateLogicalPhysicalTypesCompatible(TableSinkUtils.scala:308)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:195)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:191)
> at scala.Option.map(Option.scala:146)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(PlannerBase.scala:191)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:150)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:150)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.Abstrac
[jira] [Updated] (FLINK-17313) Validation error when insert decimal/varchar with precision into sink using TypeInformation of row
[
https://issues.apache.org/jira/browse/FLINK-17313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terry Wang updated FLINK-17313:
---
Description:
Test code like follwing(in blink planner):
{code:java}
tEnv.sqlUpdate("create table randomSource (" +
" a varchar(10),"
+
" b
decimal(20,2)" +
" ) with (" +
" 'type' =
'random'," +
" 'count' = '10'"
+
" )");
tEnv.sqlUpdate("create table printSink (" +
" a varchar(10),"
+
" b
decimal(22,2)," +
" ) with (" +
" 'type' = 'print'" +
" )");
tEnv.sqlUpdate("insert into printSink select * from
randomSource");
tEnv.execute("");
{code}
Print TableSink implements UpsertStreamTableSink and it's getReocrdType is as
following:
{code:java}
public TypeInformation getRecordType() {
return getTableSchema().toRowType();
}
{code}
Varchar column validation exception is:
org.apache.flink.table.api.ValidationException: Type VARCHAR(10) of table field
'a' does not match with the physical type STRING of the 'a' field of the
TableSink consumed type.
at
org.apache.flink.table.utils.TypeMappingUtils.lambda$checkPhysicalLogicalTypeCompatible$4(TypeMappingUtils.java:165)
at
org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:278)
at
org.apache.flink.table.utils.TypeMappingUtils$1.defaultMethod(TypeMappingUtils.java:255)
at
org.apache.flink.table.types.logical.utils.LogicalTypeDefaultVisitor.visit(LogicalTypeDefaultVisitor.java:67)
at
org.apache.flink.table.types.logical.VarCharType.accept(VarCharType.java:157)
at
org.apache.flink.table.utils.TypeMappingUtils.checkIfCompatible(TypeMappingUtils.java:255)
at
org.apache.flink.table.utils.TypeMappingUtils.checkPhysicalLogicalTypeCompatible(TypeMappingUtils.java:161)
at
org.apache.flink.table.planner.sinks.TableSinkUtils$$anonfun$validateLogicalPhysicalTypesCompatible$1.apply$mcVI$sp(TableSinkUtils.scala:315)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
at
org.apache.flink.table.planner.sinks.TableSinkUtils$.validateLogicalPhysicalTypesCompatible(TableSinkUtils.scala:308)
at
org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:195)
at
org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$2.apply(PlannerBase.scala:191)
at scala.Option.map(Option.scala:146)
at
org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(PlannerBase.scala:191)
at
org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:150)
at
org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:150)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:150)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:863)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.translateAndClearBuffer(TableEnvironmentImpl.java:855)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.execute(TableEnvironmentImpl.java:822)
Other type validation exception is similar, I dig into and think it's caused by
TypeMappingUtils#checkPhysicalLogicalTypeCompatible. It seems that the method
doesn't consider the different physical and logical type validation logic of
source and sink: logical type should be able to cover the physical type in
source, but physical type should be able to cover the logic type in sink vice
verse. Besides, the decimal ty
[GitHub] [flink] tillrohrmann commented on pull request #11887: [DRAFT][FLINK-14816] Add thread dump feature for taskmanager
tillrohrmann commented on pull request #11887: URL: https://github.com/apache/flink/pull/11887#issuecomment-618870957 Thanks for the pointer @lamber-ken. I will update the PR to fix the `RestAPIStabilityTest`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17312) Support sql client start with savepoint
[
https://issues.apache.org/jira/browse/FLINK-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091310#comment-17091310
]
lun zhang edited comment on FLINK-17312 at 4/24/20, 8:14 AM:
-
Thanks for your reply.I have build a *sql client* *platform* where you can
write 、 manage and debug your `flink sql`. You can see more in my github
project [fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for
use it in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help sql client build a high availability sql
job. I've already tests my code in flink-1.10.It's very helpful when you can
stop and restart you `sql job` by savepoint.
[~ykt836]
was (Author: zhanglun):
Thanks for your reply.I have build a *sql client* *platform* where you can
write and manage your `flink sql`. You can see more in my github project
[fsqlfly|[https://github.com/mrzhangboss/fsqlfly]] .Now I'm ready for use it
in really world .
But I found a missing significant future in *sql client*. You can stop with
savepoint in *sql client* by running *_flink stop -s savepoint jobid_*. But we
can't use this *savepoint* in *sql client* command line. So I pull a request
add savepoint support when start flink sql job in yml file. You can use like :
1. first stop your *insert sql job* to get savepoint dir
2. then start your sql job again just need add one line in your *envirment.yml*
{{execution:}}
{{ planner}}{{: blink}}
{{ type: streaming}}
{{ savepoint-path: hdfs:///tmp/savepoints/jasdf # the location of you latest
stoppoint file}}
By support this future.This will help sql client build a high availability sql
job. I've already tests my code in flink-1.10.It's very helpful when you can
stop and restart you `sql job` by savepoint.
[~ykt836]
> Support sql client start with savepoint
> ---
>
> Key: FLINK-17312
> URL: https://issues.apache.org/jira/browse/FLINK-17312
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Client
>Affects Versions: 1.10.0, 1.11.0
>Reporter: lun zhang
>Priority: Major
> Labels: pull-request-available
>
> Sql client not support *insert sql job* restart with *savepoint*
> current.It's very helpful when you can stop your flink *insert sql job* and
> restart with savepoint .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-17058) Adding TimeoutTrigger support nested triggers
[ https://issues.apache.org/jira/browse/FLINK-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-17058: Assignee: Roey Shem Tov > Adding TimeoutTrigger support nested triggers > - > > Key: FLINK-17058 > URL: https://issues.apache.org/jira/browse/FLINK-17058 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Roey Shem Tov >Assignee: Roey Shem Tov >Priority: Minor > Attachments: ProcessingTimeoutTrigger.java, > ProcessingTimeoutTrigger.java > > > Hello, > first Jira ticket that im opening here so if there is any mistakes of how > doing it, please recorrect me. > My suggestion is to add a TimeoutTrigger that apply as nestedTrigger(as > example how the PurgeTrigger does). > The TimeoutTrigger will support ProcessTime/EventTime and will have 2 timeous: > # Const timeout - when the first element of the window is arriving it is > opening a timeout of X millis - after that the window will be evaluate. > # Continual timeout - each record arriving will increase the timeout of the > evaluation of the window. > > I found it very useful in our case when using flink, and i would like to work > on it (if it is possible). > what do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17058) Adding TimeoutTrigger support nested triggers
[
https://issues.apache.org/jira/browse/FLINK-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091329#comment-17091329
]
Aljoscha Krettek commented on FLINK-17058:
--
For going on, could you add a test similar to {{CountTriggerTest}} and then
open a pull request?
> Adding TimeoutTrigger support nested triggers
> -
>
> Key: FLINK-17058
> URL: https://issues.apache.org/jira/browse/FLINK-17058
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Roey Shem Tov
>Priority: Minor
> Attachments: ProcessingTimeoutTrigger.java,
> ProcessingTimeoutTrigger.java
>
>
> Hello,
> first Jira ticket that im opening here so if there is any mistakes of how
> doing it, please recorrect me.
> My suggestion is to add a TimeoutTrigger that apply as nestedTrigger(as
> example how the PurgeTrigger does).
> The TimeoutTrigger will support ProcessTime/EventTime and will have 2 timeous:
> # Const timeout - when the first element of the window is arriving it is
> opening a timeout of X millis - after that the window will be evaluate.
> # Continual timeout - each record arriving will increase the timeout of the
> evaluation of the window.
>
> I found it very useful in our case when using flink, and i would like to work
> on it (if it is possible).
> what do you think?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #11879: [FLINK-17308] Add regular cleanup task for ExecutionGraphCache
tillrohrmann commented on a change in pull request #11879:
URL: https://github.com/apache/flink/pull/11879#discussion_r414390217
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/AbstractExecutionGraphHandler.java
##
@@ -95,7 +96,7 @@ protected AbstractExecutionGraphHandler(
/**
* Called for each request after the corresponding {@link
AccessExecutionGraph} has been retrieved from the
-* {@link ExecutionGraphCache}.
+* {@link DefaultExecutionGraphCache}.
Review comment:
Good point. Will revert it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #11879: [FLINK-17308] Add regular cleanup task for ExecutionGraphCache
tillrohrmann commented on a change in pull request #11879:
URL: https://github.com/apache/flink/pull/11879#discussion_r414391835
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/metrics/JobVertexWatermarksHandlerTest.java
##
@@ -87,7 +87,7 @@ public void before() throws Exception {
Time.seconds(1),
Collections.emptyMap(),
metricFetcher,
- Mockito.mock(ExecutionGraphCache.class),
+ Mockito.mock(DefaultExecutionGraphCache.class),
Review comment:
I'll check it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-17236) Add new Tutorials section to Documentation
[ https://issues.apache.org/jira/browse/FLINK-17236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Anderson updated FLINK-17236: --- Release Note: Added a section of Hands-on Tutorials to the documentation. (was: Introduction to the new Hands-on Tutorials added to the documentation.) > Add new Tutorials section to Documentation > -- > > Key: FLINK-17236 > URL: https://issues.apache.org/jira/browse/FLINK-17236 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Documentation / Training >Reporter: David Anderson >Assignee: David Anderson >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > This section will contain pages of content contributed from Ververica's Flink > training website. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #11879: [FLINK-17308] Add regular cleanup task for ExecutionGraphCache
tillrohrmann commented on a change in pull request #11879:
URL: https://github.com/apache/flink/pull/11879#discussion_r414392648
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/legacy/ExecutionGraphCacheTest.java
##
@@ -59,7 +59,7 @@
import static org.junit.Assert.fail;
/**
- * Tests for the {@link ExecutionGraphCache}.
+ * Tests for the {@link DefaultExecutionGraphCache}.
*/
public class ExecutionGraphCacheTest extends TestLogger {
Review comment:
Makes sense. I will change it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #11879: [FLINK-17308] Add regular cleanup task for ExecutionGraphCache
tillrohrmann commented on a change in pull request #11879:
URL: https://github.com/apache/flink/pull/11879#discussion_r414392422
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/metrics/JobVertexWatermarksHandlerTest.java
##
@@ -87,7 +87,7 @@ public void before() throws Exception {
Time.seconds(1),
Collections.emptyMap(),
metricFetcher,
- Mockito.mock(ExecutionGraphCache.class),
+ Mockito.mock(DefaultExecutionGraphCache.class),
Review comment:
Yes, it works.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #11848: [FLINK-17313]fix type validation error when sink table ddl exists column with precision of decimal/varchar
dawidwys commented on a change in pull request #11848:
URL: https://github.com/apache/flink/pull/11848#discussion_r414392699
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/TypeMappingUtils.java
##
@@ -169,8 +169,7 @@ public static void checkPhysicalLogicalTypeCompatible(
logicalFieldName,
physicalFieldType,
physicalFieldName,
-
"TableSource return type"),
- cause));
+
"TableSource return type"), cause));
Review comment:
What I meant is that we can inline the `TableSource return type` into
the string. There is no reason for having it as an argument to
`String.format()`.:
```
(cause) -> new ValidationException(
String.format("Type %s of table field '%s' does
not match with " +
"the physical type %s of the '%s' field
of the "TableSource return type".",
logicalFieldType,
logicalFieldName,
physicalFieldType,
physicalFieldName),
cause));
```
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/TypeMappingUtils.java
##
@@ -169,8 +169,7 @@ public static void checkPhysicalLogicalTypeCompatible(
logicalFieldName,
physicalFieldType,
physicalFieldName,
-
"TableSource return type"),
- cause));
+
"TableSource return type"), cause));
Review comment:
What I meant is that we can inline the `TableSource return type` into
the string. There is no reason for having it as an argument to
`String.format()`.:
```
(cause) -> new ValidationException(
String.format("Type %s of table field '%s' does
not match with " +
"the physical type %s of the '%s' field
of the TableSource return type.",
logicalFieldType,
logicalFieldName,
physicalFieldType,
physicalFieldName),
cause));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #11848: [FLINK-17313]fix type validation error when sink table ddl exists column with precision of decimal/varchar
dawidwys commented on a change in pull request #11848:
URL: https://github.com/apache/flink/pull/11848#discussion_r414392699
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/TypeMappingUtils.java
##
@@ -169,8 +169,7 @@ public static void checkPhysicalLogicalTypeCompatible(
logicalFieldName,
physicalFieldType,
physicalFieldName,
-
"TableSource return type"),
- cause));
+
"TableSource return type"), cause));
Review comment:
What I meant is that we can inline the `TableSource return type` into
the string. There is no reason for having it as an argument to
`String.format()`.:
```
(cause) -> new ValidationException(
String.format("Type %s of table
field '%s' does not match with " +
"the physical type %s
of the '%s' field of the "TableSource return type".",
logicalFieldType,
logicalFieldName,
physicalFieldType,
physicalFieldName),
cause));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #11848: [FLINK-17313]fix type validation error when sink table ddl exists column with precision of decimal/varchar
dawidwys commented on a change in pull request #11848:
URL: https://github.com/apache/flink/pull/11848#discussion_r414392699
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/TypeMappingUtils.java
##
@@ -169,8 +169,7 @@ public static void checkPhysicalLogicalTypeCompatible(
logicalFieldName,
physicalFieldType,
physicalFieldName,
-
"TableSource return type"),
- cause));
+
"TableSource return type"), cause));
Review comment:
What I meant is that we can inline the `TableSource return type` into
the string. There is no reason for having it as an argument to
`String.format()`.:
```
(cause) -> new ValidationException(
String.format("Type %s of table field '%s' does not match with " +
"the physical type %s of the '%s' field of the TableSource
return type.",
logicalFieldType,
logicalFieldName,
physicalFieldType,
physicalFieldName),
cause));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #11848: [FLINK-17313]fix type validation error when sink table ddl exists column with precision of decimal/varchar
dawidwys commented on a change in pull request #11848:
URL: https://github.com/apache/flink/pull/11848#discussion_r414392699
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/TypeMappingUtils.java
##
@@ -169,8 +169,7 @@ public static void checkPhysicalLogicalTypeCompatible(
logicalFieldName,
physicalFieldType,
physicalFieldName,
-
"TableSource return type"),
- cause));
+
"TableSource return type"), cause));
Review comment:
What I meant is that we can inline the `TableSource return type` into
the string. There is no reason for having it as an argument to
`String.format()`.:
```
(cause) -> new ValidationException(
String.format("Type %s of table field '%s' does not match with
" +
"the physical type %s of the '%s' field of the
TableSource return type.",
logicalFieldType,
logicalFieldName,
physicalFieldType,
physicalFieldName),
cause));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] curcur commented on pull request #11725: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur commented on pull request #11725: URL: https://github.com/apache/flink/pull/11725#issuecomment-618879789 > So I had an offline discussion with Stephan to clarify the scope. Indeed, `KafkaShuffle` has been requested by users and serves as a bridge until we get fully persistent channels. > > We both agree that it would have been nice to also support reading from non-Flink shuffles (a.k.a from any partitioned Kafka topic) by making the serializer pluggable at the composition level. Please have a look at `StreamElementSerializer` and see if we can use it. If that doesn't work for some reason, then I can live with a pure `KafkaShuffle` in the first iteration. > > Implementation-wise, we are both a bit skeptical that an API change (to `SinkFunction`) is the best course as that requires more coordination and should have probably been triggered already if you want this feature in 1.11. Using custom operators would give you all freedom without that the need of coordiation. It would also avoid the changes to `KafkaProducer`/`KafkaConsumer` on the cost of replicating some logic. > > Lastly, I have strong headaches on how checkpoints and savepoints are working with `KafkaShuffle`. I think for storing checkpoints and recovery in terms of fault tolerance, the approach is good as-is. However, for savepoints, we should probably ensure that no unconsumed data is still in lingering in the shuffle topic as that would translate to in-flight data. Hence, restoring from an old savepoint would completely screw up the data. At this point, we also need to ensure that the topic is purged (probably with some assertion). Not supporting going back in checkpoints should be save from current guarantees. Alternatively, we also need to implement some recovery logic for older check/savepoints that ignores "future" data somehow (so some higher level Kafka offset management). Hey Arvid, thanks so much for the quick response! I think you have several concerns about 1. why `StreamElementSerializer` can not be reused 2. why I have to have a different `KafkaProducer/KafkaConsumer` 3. Have a better way instead of changing `SinkFunction` (that's exactly my concern, and why I want to get early feedback, I am hesitating as well) 4. `save points`, which I do not completely get it. For the first two, I have reasons; for the third one, I have concerns as well. For the forth, not completely sure I understand it correctly. Do, do you have time to chat a bit on these four points on Monday? BTW, I am not insisting to get this in 1.11. Instead, I really want to do it in the right way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11687: [FLINK-16536][network][checkpointing] Implement InputChannel state recovery for unaligned checkpoint
flinkbot edited a comment on pull request #11687: URL: https://github.com/apache/flink/pull/11687#issuecomment-611445542 ## CI report: * 5bad018f00f85a7359345187a12d7938aa510d25 UNKNOWN * d51ce7f47381d99b843278cd701dcff223761a0b UNKNOWN * cba096ae3d8eba4a0d39c64659f76ad10a62be27 UNKNOWN * 23818c4b30eb9714154080253a090fc63d983925 UNKNOWN * 9b6278f15a4fc65d78671a145305df2c7310cc6a UNKNOWN * 4f7399680818f5d29e917e17720e00900822a43d UNKNOWN * 95dc7d6ada34179b51849d905f15c740ca585a8d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161778166) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=172) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11854: [WIP] Introduce external resource framework
flinkbot edited a comment on pull request #11854: URL: https://github.com/apache/flink/pull/11854#issuecomment-617586491 ## CI report: * 6f27933a89fef05f7b65868d898780e9beb6b231 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161379447) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=37) * 8a589190554914c4a8886f3630468b56c18eca94 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11848: [FLINK-17313]fix type validation error when sink table ddl exists column with precision of decimal/varchar
flinkbot edited a comment on pull request #11848: URL: https://github.com/apache/flink/pull/11848#issuecomment-617531785 ## CI report: * 9b24a40f43a101b63ab73d1fa1e0c0f31178b36d Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161363103) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22) * 88d8d3f5e30e69f324adac96e4db471f91fe490b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11855: [FLINK-13639] Refactor the IntermediateResultPartitionID to consist o…
flinkbot edited a comment on pull request #11855: URL: https://github.com/apache/flink/pull/11855#issuecomment-617596227 ## CI report: * 77720a95c9fb8163487dbec5bc82681f1e7f9fde Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161400024) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=54) * 32d5ffa4730232c0ae2d978c4a9537604e5510db UNKNOWN * 206279cb7c4bbb95605ce57a71d5f43568a2649b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11899: [FLINK-17342] enqueue savepoint requests in UC mode
flinkbot edited a comment on pull request #11899: URL: https://github.com/apache/flink/pull/11899#issuecomment-618868456 ## CI report: * 8ced5a3c7d94080f7f34588721886653bf334060 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161783542) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=178) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-docker] zentol commented on a change in pull request #15: [FLINK-17346] De-duplicate process setup
zentol commented on a change in pull request #15:
URL: https://github.com/apache/flink-docker/pull/15#discussion_r414395476
##
File path: testing/testing_lib.sh
##
@@ -45,51 +45,42 @@ function build_image() {
docker build -t "$image_name" "$dockerfile_dir"
}
-function run_jobmanager() {
-local dockerfile
-dockerfile="$1"
+function internal_run() {
+local dockerfile="$1"
+local docker_run_command="$2"
+local args="$3"
local image_tag image_name
image_tag="$(image_tag "$dockerfile")"
image_name="$(image_name "$image_tag")"
-echo >&2 "===> Starting ${image_tag} jobmanager..."
-
-# Prints container ID
-docker run \
---rm \
---detach \
---name "jobmanager" \
---network "$NETWORK_NAME" \
---publish 6123:6123 \
---publish 8081:8081 \
--e JOB_MANAGER_RPC_ADDRESS="jobmanager" \
-"$image_name" \
-jobmanager
+echo >&2 "===> Starting ${image_tag} ${args}..."
+
+eval "docker run --rm --detach --network $NETWORK_NAME -e
JOB_MANAGER_RPC_ADDRESS=jobmanager ${docker_run_command} $image_name ${args}"
+}
+
+function internal_run_jobmanager() {
+internal_run "$1" "--name jobmanager --publish 6123:6123 --publish
8081:8081 $2" jobmanager
+}
+
+function run_jobmanager() {
+internal_run_jobmanager "$1" ""
Review comment:
probably not, but I think it is better to be explicit about it. If
another argument is added you're less likely to pass the new argument as the
second one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-17346) Deduplicate process setup in docker tests
[ https://issues.apache.org/jira/browse/FLINK-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-17346. Fix Version/s: docker-1.9.3.0 docker-1.11.0.0 docker-1.10.1.0 Resolution: Fixed docker-master: 887ac36c2b9611095f989eb0cff527fa196f0456 > Deduplicate process setup in docker tests > - > > Key: FLINK-17346 > URL: https://issues.apache.org/jira/browse/FLINK-17346 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: docker-1.10.1.0, docker-1.11.0.0, docker-1.9.3.0 > > > There's lots of duplication in how the docker processes is being setup, which > would become even worse with FLINK-17164. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur edited a comment on pull request #11725: [FLINK-15670][API] Provide a Kafka Source/Sink pair as KafkaShuffle
curcur edited a comment on pull request #11725: URL: https://github.com/apache/flink/pull/11725#issuecomment-618879789 > So I had an offline discussion with Stephan to clarify the scope. Indeed, `KafkaShuffle` has been requested by users and serves as a bridge until we get fully persistent channels. > > We both agree that it would have been nice to also support reading from non-Flink shuffles (a.k.a from any partitioned Kafka topic) by making the serializer pluggable at the composition level. Please have a look at `StreamElementSerializer` and see if we can use it. If that doesn't work for some reason, then I can live with a pure `KafkaShuffle` in the first iteration. > > Implementation-wise, we are both a bit skeptical that an API change (to `SinkFunction`) is the best course as that requires more coordination and should have probably been triggered already if you want this feature in 1.11. Using custom operators would give you all freedom without that the need of coordiation. It would also avoid the changes to `KafkaProducer`/`KafkaConsumer` on the cost of replicating some logic. > > Lastly, I have strong headaches on how checkpoints and savepoints are working with `KafkaShuffle`. I think for storing checkpoints and recovery in terms of fault tolerance, the approach is good as-is. However, for savepoints, we should probably ensure that no unconsumed data is still in lingering in the shuffle topic as that would translate to in-flight data. Hence, restoring from an old savepoint would completely screw up the data. At this point, we also need to ensure that the topic is purged (probably with some assertion). Not supporting going back in checkpoints should be save from current guarantees. Alternatively, we also need to implement some recovery logic for older check/savepoints that ignores "future" data somehow (so some higher level Kafka offset management). Hey Arvid, thanks so much for the quick response! I think you have several concerns about 1. why `StreamElementSerializer` can not be reused 2. why I have to have a different `KafkaProducer/KafkaConsumer` 3. Have a better way instead of changing `SinkFunction` (that's exactly my concern, and why I want to get early feedback, I am hesitating as well) 4. `save points`, which I do not completely get it. For the first two, I have reasons; for the third one, I have concerns as well. For the forth, not completely sure I understand it correctly. Do, do you have time to chat a bit on these four points on Monday? BTW, I am not insisting to get this in 1.11 if the scope is bigger than expected. Instead, I really want to do it in the right way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #11879: [FLINK-17308] Add regular cleanup task for ExecutionGraphCache
tillrohrmann commented on a change in pull request #11879:
URL: https://github.com/apache/flink/pull/11879#discussion_r414399874
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorEndpoint.java
##
@@ -725,13 +731,28 @@ public WebMonitorEndpoint(
@Override
public void startInternal() throws Exception {
leaderElectionService.start(this);
+ startExecutionGraphCacheCleanupTask();
+
if (hasWebUI) {
log.info("Web frontend listening at {}.",
getRestBaseUrl());
}
}
+ private void startExecutionGraphCacheCleanupTask() {
+ final long cleanupInterval = 2 *
restConfiguration.getRefreshInterval();
Review comment:
The idea was that the cleanup does not have to be as aggressive as the
refresh interval. Any other multiple of `getRefreshInterval` would have done
the same job I guess.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17330) Avoid scheduling deadlocks caused by cyclic input dependencies between regions
[
https://issues.apache.org/jira/browse/FLINK-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091338#comment-17091338
]
Zhu Zhu commented on FLINK-17330:
-
Thanks for the explanation an d suggestion! This fix is not a blocker of other
work items of FLIP-119. So I think we can focus on the required work items
first and decide how we do this fix based on the time we have after the other
items are done.
If there is not enough time, we can just add cyclic dependency detection and
throw exception to ask users to set edges to BLOCKING.
If we have sufficient time, we can directly go to the solution to "merge
regions with cyclic dependencies". I think it's possible since the feature
freeze is highly possibly to be extended for 2 weeks.
WDYT?
> Avoid scheduling deadlocks caused by cyclic input dependencies between regions
> --
>
> Key: FLINK-17330
> URL: https://issues.apache.org/jira/browse/FLINK-17330
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Zhu Zhu
>Priority: Major
> Fix For: 1.11.0
>
>
> Imagine a job like this:
> A -- (pipelined FORWARD) --> B -- (blocking ALL-to-ALL) --> D
> A -- (pipelined FORWARD) --> C -- (pipelined FORWARD) --> D
> parallelism=2 for all vertices.
> We will have 2 execution pipelined regions:
> R1 = {A1, B1, C1, D1}
> R2 = {A2, B2, C2, D2}
> R1 has a cross-region input edge (B2->D1).
> R2 has a cross-region input edge (B1->D2).
> Scheduling deadlock will happen since we schedule a region only when all its
> inputs are consumable (i.e. blocking partitions to be finished). This is
> because R1 can be scheduled only if R2 finishes, while R2 can be scheduled
> only if R1 finishes.
> To avoid this, one solution is to force a logical pipelined region with
> intra-region ALL-to-ALL blocking edges to form one only execution pipelined
> region, so that there would not be cyclic input dependency between regions.
> Besides that, we should also pay attention to avoid cyclic cross-region
> POINTWISE blocking edges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16423) test_ha_per_job_cluster_datastream.sh gets stuck
[
https://issues.apache.org/jira/browse/FLINK-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091340#comment-17091340
]
Till Rohrmann commented on FLINK-16423:
---
[~rmetzger] shouldn't your comment go to FLINK-16770 since this is a duplicate
which has been closed?
> test_ha_per_job_cluster_datastream.sh gets stuck
>
>
> Key: FLINK-16423
> URL: https://issues.apache.org/jira/browse/FLINK-16423
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Attachments: 20200408.1.tgz
>
>
> This was seen in
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5905&view=logs&j=b1623ac9-0979-5b0d-2e5e-1377d695c991&t=e7804547-1789-5225-2bcf-269eeaa37447
> ... the relevant part of the logs is here:
> {code}
> 2020-03-04T11:27:25.4819486Z
> ==
> 2020-03-04T11:27:25.4820470Z Running 'Running HA per-job cluster (rocks,
> non-incremental) end-to-end test'
> 2020-03-04T11:27:25.4820922Z
> ==
> 2020-03-04T11:27:25.4840177Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-25482960156
> 2020-03-04T11:27:25.6712478Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-03-04T11:27:25.6830402Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-03-04T11:27:26.2988914Z Starting zookeeper daemon on host fv-az655.
> 2020-03-04T11:27:26.3001237Z Running on HA mode: parallelism=4,
> backend=rocks, asyncSnapshots=true, and incremSnapshots=false.
> 2020-03-04T11:27:27.4206924Z Starting standalonejob daemon on host fv-az655.
> 2020-03-04T11:27:27.4217066Z Start 1 more task managers
> 2020-03-04T11:27:30.8412541Z Starting taskexecutor daemon on host fv-az655.
> 2020-03-04T11:27:38.1779980Z Job () is
> running.
> 2020-03-04T11:27:38.1781375Z Running JM watchdog @ 89778
> 2020-03-04T11:27:38.1781858Z Running TM watchdog @ 89779
> 2020-03-04T11:27:38.1783272Z Waiting for text Completed checkpoint [1-9]* for
> job to appear 2 of times in logs...
> 2020-03-04T13:21:29.9076797Z ##[error]The operation was canceled.
> 2020-03-04T13:21:29.9094090Z ##[section]Finishing: Run e2e tests
> {code}
> The last three lines indicate that the test is waiting forever for a
> checkpoint to appear.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-docker] zentol opened a new pull request #17: [FLINK-17367] Add configurable download URL
zentol opened a new pull request #17: URL: https://github.com/apache/flink-docker/pull/17 Adds a hook into `add-version.sh` to generate dockerfiles that download the distribution from a configurable URL. Since these likely won't match any GPG signature (as is the case for our nightly snapshots) this check will be skipped. This required that we pass the URLs directly into the `Dockerfile`, instead of having it generate them from the flink/scala versions. A new `GPG_SKIP` flag controls whether the signature should be verified. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17367) Support generating dockerfiles against custom URL
[ https://issues.apache.org/jira/browse/FLINK-17367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17367: --- Labels: pull-request-available (was: ) > Support generating dockerfiles against custom URL > - > > Key: FLINK-17367 > URL: https://issues.apache.org/jira/browse/FLINK-17367 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > The Dockerfiles are currently hard-wired to work against the apache mirrors > to retrieve official releases. > In order to test features against Flink-SNAPSHOT, and as a generally useful > dev feature, we should allow generating the images against an arbitrary url. > In the case of Flink-SNAPSHOT, these would for example be the S3 bucket we > upload the snapshot binaries to. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11848: [FLINK-17313]fix type validation error when sink table ddl exists column with precision of decimal/varchar
flinkbot edited a comment on pull request #11848: URL: https://github.com/apache/flink/pull/11848#issuecomment-617531785 ## CI report: * 9b24a40f43a101b63ab73d1fa1e0c0f31178b36d Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161363103) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22) * 88d8d3f5e30e69f324adac96e4db471f91fe490b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161787406) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=180) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11854: [WIP] Introduce external resource framework
flinkbot edited a comment on pull request #11854: URL: https://github.com/apache/flink/pull/11854#issuecomment-617586491 ## CI report: * 6f27933a89fef05f7b65868d898780e9beb6b231 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161379447) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=37) * 8a589190554914c4a8886f3630468b56c18eca94 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161787444) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=181) * bddb0e274da11bbe99d15c6e0bb55e8d8c0e658a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11855: [FLINK-13639] Refactor the IntermediateResultPartitionID to consist o…
flinkbot edited a comment on pull request #11855: URL: https://github.com/apache/flink/pull/11855#issuecomment-617596227 ## CI report: * 77720a95c9fb8163487dbec5bc82681f1e7f9fde Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161400024) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=54) * 32d5ffa4730232c0ae2d978c4a9537604e5510db UNKNOWN * 206279cb7c4bbb95605ce57a71d5f43568a2649b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/161787490) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=182) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11883: [BP-1.9][FLINK-17308] Add regular cleanup task for ExecutionGraphCache
flinkbot edited a comment on pull request #11883: URL: https://github.com/apache/flink/pull/11883#issuecomment-618317168 ## CI report: * b3f2123931105e098b03e66236349e47c7aa5e7e UNKNOWN * f924b3374104c3ca7cc334c1c0201cfd7098d7cd Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/161604561) * 735f28ff4c14af0634059716851f6fad29c5a0ad UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] docete commented on a change in pull request #11869: [FLINK-17111][table] Support SHOW VIEWS in Flink SQL
docete commented on a change in pull request #11869:
URL: https://github.com/apache/flink/pull/11869#discussion_r414408162
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java
##
@@ -415,6 +415,37 @@ public CatalogBaseTable getTable() {
});
}
+ /**
+* Returns an array of names of all views(both temporary and permanent)
registered in
+* the namespace of the current catalog and database.
+*
+* @return names of all registered views
+*/
+ public Set listViews() {
+ return listViews(getCurrentCatalog(), getCurrentDatabase());
+ }
+
+ /**
+* Returns an array of names of all views(both temporary and permanent)
registered in
+* the namespace of the current catalog and database.
+*
+* @return names of registered views
+*/
+ public Set listViews(String catalogName, String databaseName) {
+ Catalog currentCatalog = catalogs.get(getCurrentCatalog());
+
+ try {
+ return Stream.concat(
+
currentCatalog.listViews(getCurrentDatabase()).stream(),
+ listTemporaryTablesInternal(catalogName,
databaseName)
+ .filter(e -> e.getValue() instanceof
CatalogView)
+ .map(e -> e.getKey().getObjectName())
+ ).collect(Collectors.toSet());
Review comment:
IMO the private `listTemporaryTablesInternal` is enough. or we should
add `listTemporaryTables(String catalogName, String databaseName)` for
`listTemporaryTables()`. That's probably a bit superfluous.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
