[GitHub] [flink] kezhuw commented on a change in pull request #14936: [FLINK-20580][core] Don't support nullable value for SerializedValue

[GitBox]

kezhuw commented on a change in pull request #14936:
URL: https://github.com/apache/flink/pull/14936#discussion_r578501942



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/rpc/akka/AkkaRpcSerializedValue.java
##
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rpc.akka;
+
+import org.apache.flink.util.InstantiationUtil;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Arrays;
+
+/** A self-contained serialized value to decouple from user values and 
transfer on wire. */
+class AkkaRpcSerializedValue implements Serializable {
+@Nullable private final byte[] serializedData;

Review comment:
   @tillrohrmann Missed `serialVersionUID`.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14959: [FLINK-16947][Azure] Attempt to fix maven network issues

[GitBox]

flinkbot edited a comment on pull request #14959:
URL: https://github.com/apache/flink/pull/14959#issuecomment-781241815


   
   ## CI report:
   
   * 3e3889ee5cc45f7d8dd654b9c5296a80ab6e88e0 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13448)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14916: [FLINK-21345][Table SQL / Planner] Fix BUG of Union All join Temporal…

[GitBox]

flinkbot edited a comment on pull request #14916:
URL: https://github.com/apache/flink/pull/14916#issuecomment-776614443


   
   ## CI report:
   
   * 13b22ab740f81a326927caa4f344bbafab336a54 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13447)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] kezhuw commented on pull request #14936: [FLINK-20580][core] Don't support nullable value for SerializedValue

[GitBox]

kezhuw commented on pull request #14936:
URL: https://github.com/apache/flink/pull/14936#issuecomment-781410232


   @tillrohrmann Thanks for reviewing, the comments make sense to me. Then I 
will give up rebasing to you.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-21382) Standalone K8s documentation does not explain usage of standby JobManagers

[Yang Wang (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286514#comment-17286514
 ] 

Yang Wang commented on FLINK-21382:
---

Make sense. I used to leave a similar solution 
[here|https://issues.apache.org/jira/browse/FLINK-20982?focusedCommentId=17265715=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17265715].

 

I would suggest to set the jobmanager rpc address to pod ip instead of 
Kubernetes service name even though there is only one JobManager. And also we 
do not need to create the internal service any more when the HA is enabled.

> Standalone K8s documentation does not explain usage of standby JobManagers
> --
>
> Key: FLINK-21382
> URL: https://issues.apache.org/jira/browse/FLINK-21382
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes, Documentation
>Affects Versions: 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
>
> Our [standalone K8s 
> documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#high-availability-with-standalone-kubernetes]
>  mentions how to configure K8s HA services. It does not mention that this 
> only works with a single JobManager. When using standby JobManagers, then the 
> given deployment yamls won't work because the {{jobmanager.rpc.address}} is 
> configured to be the {{jobmanager}} service.
> Changing the configuration to work is surprisingly difficult because of a 
> lack of documentation. Moreover, it is quite difficult to pass in custom 
> configuration values when using a ConfigMap for sharing Flink's 
> {{flink-conf.yaml}}. The problem is that mounted ConfigMaps are not writable 
> from a pod perspective. See [this 
> answer|https://stackoverflow.com/a/66228073/4815083] for how one could 
> achieve it.
> I think we could improve our documentation to explain our users how to 
> configure a standalone HA cluster with standby JobManagers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] tillrohrmann commented on a change in pull request #14936: [FLINK-20580][core] Don't support nullable value for SerializedValue

[GitBox]

tillrohrmann commented on a change in pull request #14936:
URL: https://github.com/apache/flink/pull/14936#discussion_r578464596



##
File path: 
flink-core/src/test/java/org/apache/flink/util/SerializedValueTest.java
##
@@ -47,26 +50,47 @@ public void testSimpleValue() {
 assertNotNull(v.toString());
 assertNotNull(copy.toString());
 
+assertNotEquals(0, v.getByteArray().length);
+assertArrayEquals(v.getByteArray(), copy.getByteArray());
+
+byte[] bytes = v.getByteArray();
+SerializedValue saved =
+SerializedValue.fromBytes(Arrays.copyOf(bytes, 
bytes.length));
+assertEquals(v, saved);
+assertArrayEquals(v.getByteArray(), saved.getByteArray());
 } catch (Exception e) {
 e.printStackTrace();
 fail(e.getMessage());
 }
 }
 
 @Test

Review comment:
   Might be a bit shorter:
   
   ```suggestion
   @Test(expected = NullPointerException.class)
   ```

##
File path: 
flink-core/src/test/java/org/apache/flink/util/SerializedValueTest.java
##
@@ -47,26 +50,47 @@ public void testSimpleValue() {
 assertNotNull(v.toString());
 assertNotNull(copy.toString());
 
+assertNotEquals(0, v.getByteArray().length);
+assertArrayEquals(v.getByteArray(), copy.getByteArray());
+
+byte[] bytes = v.getByteArray();
+SerializedValue saved =
+SerializedValue.fromBytes(Arrays.copyOf(bytes, 
bytes.length));
+assertEquals(v, saved);
+assertArrayEquals(v.getByteArray(), saved.getByteArray());
 } catch (Exception e) {
 e.printStackTrace();
 fail(e.getMessage());
 }
 }
 
 @Test
-public void testNullValue() {
+public void testNullValue() throws Exception {
 try {
-SerializedValue v = new SerializedValue<>(null);
-SerializedValue copy = 
CommonTestUtils.createCopySerializable(v);
+new SerializedValue<>(null);
+fail("expect NullPointerException");
+} catch (NullPointerException e) {
+// ignore
+}
+}
 
-assertNull(copy.deserializeValue(getClass().getClassLoader()));
+@Test
+public void testFromNullBytes() {
+try {
+SerializedValue.fromBytes(null);
+fail("expect NullPointerException");
+} catch (NullPointerException e) {
+// ignore
+}
+}
 
-assertEquals(v, copy);
-assertEquals(v.hashCode(), copy.hashCode());
-assertEquals(v.toString(), copy.toString());
-} catch (Exception e) {
-e.printStackTrace();
-fail(e.getMessage());
+@Test

Review comment:
   Same here with `@Test(exepcted = ...)`.

##
File path: 
flink-core/src/test/java/org/apache/flink/util/SerializedValueTest.java
##
@@ -47,26 +50,47 @@ public void testSimpleValue() {
 assertNotNull(v.toString());
 assertNotNull(copy.toString());
 
+assertNotEquals(0, v.getByteArray().length);
+assertArrayEquals(v.getByteArray(), copy.getByteArray());
+
+byte[] bytes = v.getByteArray();
+SerializedValue saved =
+SerializedValue.fromBytes(Arrays.copyOf(bytes, 
bytes.length));
+assertEquals(v, saved);
+assertArrayEquals(v.getByteArray(), saved.getByteArray());
 } catch (Exception e) {
 e.printStackTrace();
 fail(e.getMessage());
 }
 }
 
 @Test
-public void testNullValue() {
+public void testNullValue() throws Exception {
 try {
-SerializedValue v = new SerializedValue<>(null);
-SerializedValue copy = 
CommonTestUtils.createCopySerializable(v);
+new SerializedValue<>(null);
+fail("expect NullPointerException");
+} catch (NullPointerException e) {
+// ignore
+}
+}
 
-assertNull(copy.deserializeValue(getClass().getClassLoader()));
+@Test

Review comment:
   Same here with `@Test(expected = ...)`.

##
File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/rpc/akka/AkkaRpcSerializedValueTest.java
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * 

[GitHub] [flink] rmetzger commented on a change in pull request #14923: [DRAFT] Remove State#onEnter

[GitBox]

rmetzger commented on a change in pull request #14923:
URL: https://github.com/apache/flink/pull/14923#discussion_r578386398



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/declarative/DeclarativeScheduler.java
##
@@ -887,19 +897,22 @@ public void runIfState(State expectedState, Runnable 
action, Duration delay) {
 
 // 
 
-private void transitionToState(State newState) {
-if (state != newState) {
-LOG.debug(
-"Transition from state {} to {}.",
-state.getClass().getSimpleName(),
-newState.getClass().getSimpleName());
+@VisibleForTesting
+ void transitionToState(

Review comment:
   why did you change the semantics of "transition to same state" from 
"ignore" to "fail"?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14928: [FLINK-21360][coordination] Make resource timeout configurable

[GitBox]

flinkbot edited a comment on pull request #14928:
URL: https://github.com/apache/flink/pull/14928#issuecomment-14828


   
   ## CI report:
   
   * 6be910fe164322d9862bad9c0d80f8ff4ee5fe62 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13444)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] tillrohrmann commented on a change in pull request #14951: [FLINK-21380][coordination] Hide terminal ExecutionGraph in StateWithExecutionGraph

[GitBox]

tillrohrmann commented on a change in pull request #14951:
URL: https://github.com/apache/flink/pull/14951#discussion_r578441952



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionGraph.java
##
@@ -335,9 +352,16 @@ public static ArchivedExecutionGraph 
createFrom(ExecutionGraph executionGraph) {
 
 final long[] timestamps = new long[JobStatus.values().length];
 
+// if the state is overridden with a non-globally-terminal state then 
we need to erase
+// traces of globally-terminal states
+final boolean clearGloballyTerminalStateTimestamps =
+statusOverride != null && 
!statusOverride.isGloballyTerminalState();

Review comment:
   I think we check in the beginning of this method whether 
`statusOverride` is not a globally terminal state if it is not `null`.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] rkhachatryan commented on a change in pull request #14943: [FLINK-21354] Implement a StateChangelogStateBackend to forward state changes to St…

[GitBox]

rkhachatryan commented on a change in pull request #14943:
URL: https://github.com/apache/flink/pull/14943#discussion_r578434972



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##
@@ -162,14 +163,19 @@ public static CheckpointStorage load(
 Preconditions.checkNotNull(classLoader, "classLoader");
 Preconditions.checkNotNull(configuredStateBackend, "statebackend");
 
-if (configuredStateBackend instanceof CheckpointStorage) {
+StateBackend rootStateBackend =
+(configuredStateBackend instanceof DelegateStateBackend)
+? ((DelegateStateBackend) 
configuredStateBackend).getDelegatedStateBackend()
+: configuredStateBackend;

Review comment:
   This change will be duplicated once we add some other 
delegating/proxying backend.
   Can we avoid duplication somehow?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] rkhachatryan commented on a change in pull request #14943: [FLINK-21354] Implement a StateChangelogStateBackend to forward state changes to St…

[GitBox]

rkhachatryan commented on a change in pull request #14943:
URL: https://github.com/apache/flink/pull/14943#discussion_r578414166



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/delegate/DelegatedStateBackend.java
##
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.delegate;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateBackend;
+
+@Internal
+/**
+ * An interface for DelegatedStateBackend. A state backend to be delegated 
must implement this
+ * interface.
+ */
+public interface DelegatedStateBackend extends StateBackend {}

Review comment:
   I'm curious why didn't you add `unwrap` method to the `StateBackend` 
interface?
   
   I see some issues with the current approach:
   1. If we add some new delegatee then it's impossible to distinguish for 
which delegatee given backend can be delegated. For example, we add new 
delegating backend with extended logging. How RocksDbBackend can signal that it 
can NOT work with it?
   2. Every new backend must be marked explicitly

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/StateBackendLoader.java
##
@@ -204,6 +222,11 @@ public static StateBackend 
fromApplicationOrConfigOrDefault(
 
 // (1) the application defined state backend has precedence
 if (fromApplication != null) {
+
+checkArgument(
+!(fromApplication instanceof DelegateStateBackend),
+"DelegateStateBackend can not be delegated!");

Review comment:
   Is the intention here to prevent wrapping the backend twice?
   If so, I think such a check should be placed just before the actual wrapping.
   
   Another concern: can we get `DelegateStateBackend` here after 
deserialization?

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/delegate/DelegateStateBackend.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.delegate;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.state.StateBackend;
+
+/**
+ * An abstract base class for delegated state backend.
+ *
+ * As its name, it should include a state backend to delegate, which can be 
one of the following:
+ * {@code MemoryStateBackend}, {code FsStateBackend} and {@code 
RocksDBStateBackend}.
+ */
+@Internal
+public abstract class DelegateStateBackend implements StateBackend, 
java.io.Serializable {

Review comment:
   Could you explain why do we need this class?

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##
@@ -162,14 +163,19 @@ public static CheckpointStorage load(
 Preconditions.checkNotNull(classLoader, "classLoader");
 Preconditions.checkNotNull(configuredStateBackend, "statebackend");
 
-if (configuredStateBackend instanceof CheckpointStorage) {
+StateBackend rootStateBackend =
+(configuredStateBackend instanceof DelegateStateBackend)
+? ((DelegateStateBackend) 
configuredStateBackend).getDelegatedStateBackend()
+: configuredStateBackend;

Review comment:
   This change will duplicated once we add some other delegating/proxying 
backend.
   Can we avoid it 

[GitHub] [flink] flinkbot edited a comment on pull request #14950: [FLINK-21347][coordination] Extract interface from ExecutionGraph

[GitBox]

flinkbot edited a comment on pull request #14950:
URL: https://github.com/apache/flink/pull/14950#issuecomment-780431068


   
   ## CI report:
   
   * 1b43a411d22af19f12f14a79c1094c8dc5cfeb42 UNKNOWN
   * e6aae42cf093d4f44b93b79a78bdc64538df8725 Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13445)
 
   * a8191e6800961467c726695c8ef7575d157a Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13449)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14943: [FLINK-21354] Implement a StateChangelogStateBackend to forward state changes to St…

[GitBox]

flinkbot edited a comment on pull request #14943:
URL: https://github.com/apache/flink/pull/14943#issuecomment-779315285


   
   ## CI report:
   
   * f80b97c479ec82663e971553c54c8a6cda3122ff UNKNOWN
   * 3e1cf6d7f7f645e991c0b788f653bab7b1baa797 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13453)
 
   * c9be932fec6390ffb103c000550cbc85747c266e Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13461)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API

[GitBox]

flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058


   
   ## CI report:
   
   * f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
   * 000fa3e28def66c51ebf7b5975d8b6106988 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13446)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lllkkklll commented on pull request #13353: [FLINK-17104] Support registering custom JobStatusListeners from config

[GitBox]

lllkkklll commented on pull request #13353:
URL: https://github.com/apache/flink/pull/13353#issuecomment-781356093


   why not merge? better idea?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] tillrohrmann commented on a change in pull request #14798: [FLINK-21187] Provide exception history for root causes

[GitBox]

tillrohrmann commented on a change in pull request #14798:
URL: https://github.com/apache/flink/pull/14798#discussion_r578407995



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##
@@ -153,7 +152,7 @@
 
 private LogicalSlot assignedResource;
 
-private Throwable failureCause; // once assigned, never changes
+private Optional failureCause = Optional.empty(); // once 
assigned, never changes

Review comment:
   The comment seems to be wrong now.

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##
@@ -1133,7 +1128,10 @@ private void processFail(
 checkState(transitionState(current, FAILED, t));
 
 // success (in a manner of speaking)
-this.failureCause = t;
+if (t != null) {
+// we only set the failureCause if an error is passed (see 
FLINK-21376)
+this.failureCause = Optional.of(new ErrorInfo(t, 
getStateTimestamp(FAILED)));
+}

Review comment:
   Can't we say that if `t == null`, then we create an unknown failure 
cause? `new FlinkException("Unknown cause for Execution failure. This might be 
caused by FLINK-21376")`?

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java
##
@@ -635,6 +641,33 @@ public void cancel() {
 return 
executionGraph.getTerminationFuture().thenApply(FunctionUtils.nullFn());
 }
 
+protected void archiveGlobalFailure(Throwable failure) {
+taskFailureHistory.add(
+new ErrorInfo(failure, 
executionGraph.getStatusTimestamp(JobStatus.FAILED)));
+log.debug("Archive global failure.", failure);
+}
+
+protected void archiveFromFailureHandlingResult(FailureHandlingResult 
failureHandlingResult) {
+Optional executionOptional =
+failureHandlingResult
+.getExecutionVertexIdOfFailedTask()
+.map(this::getExecutionVertex)
+.map(ExecutionVertex::getCurrentExecutionAttempt);

Review comment:
   I think `archiveFromFailureHandlingResult` can also be called when 
handling a global failover. In this case `executionOptional` would be empty and 
as a consequence we won't record a failure cause.

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java
##
@@ -635,6 +641,33 @@ public void cancel() {
 return 
executionGraph.getTerminationFuture().thenApply(FunctionUtils.nullFn());
 }
 
+protected void archiveGlobalFailure(Throwable failure) {
+taskFailureHistory.add(
+new ErrorInfo(failure, 
executionGraph.getStatusTimestamp(JobStatus.FAILED)));
+log.debug("Archive global failure.", failure);
+}
+
+protected void archiveFromFailureHandlingResult(FailureHandlingResult 
failureHandlingResult) {
+Optional executionOptional =
+failureHandlingResult
+.getExecutionVertexIdOfFailedTask()
+.map(this::getExecutionVertex)
+.map(ExecutionVertex::getCurrentExecutionAttempt);
+
+executionOptional.ifPresent(
+execution ->
+execution
+.getFailureInfo()
+.ifPresent(
+failureInfo -> {
+
taskFailureHistory.add(failureInfo);
+log.debug(
+"Archive local failure 
causing attempt {} to fail: {}",
+execution.getAttemptId(),
+
failureInfo.getExceptionAsString());
+}));

Review comment:
   Why do we have to ask the execution for the failure cause? If we are 
only interested in the root cause, doesn't `failureHandlingResult` contain all 
the required information?

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/flip1/FailureHandlingResult.java
##
@@ -38,6 +42,12 @@
 /** Delay before the restarting can be conducted. */
 private final long restartDelayMS;
 
+/**
+ * The ExecutionVertexID refering to the ExecutionVertex the failure is 
originating from or
+ * {@code null} if it's a global failure.
+ */
+@Nullable private final ExecutionVertexID failingExecutionVertexId;
+
 /** Reason why the failure is not recoverable. */

Review comment:
   The JavaDoc seems wrong. If I am not mistaken, then we will also set 
this field if the failure is recoverable, right?

##
File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java
##

[GitHub] [flink] flinkbot edited a comment on pull request #14946: [FLINK-21381][docs] Add information about service account permissions to K8s HA service documentation

[GitBox]

flinkbot edited a comment on pull request #14946:
URL: https://github.com/apache/flink/pull/14946#issuecomment-779866080


   
   ## CI report:
   
   * eacbdcd5b9a6d60747e81f91208b102b53a5cafa Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13450)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14943: [FLINK-21354] Implement a StateChangelogStateBackend to forward state changes to St…

[GitBox]

flinkbot edited a comment on pull request #14943:
URL: https://github.com/apache/flink/pull/14943#issuecomment-779315285


   
   ## CI report:
   
   * f80b97c479ec82663e971553c54c8a6cda3122ff UNKNOWN
   * 3e1cf6d7f7f645e991c0b788f653bab7b1baa797 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13453)
 
   * c9be932fec6390ffb103c000550cbc85747c266e UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-14869) Force an extended resource to be null if it's value is 0

[Xintong Song (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xintong Song closed FLINK-14869.

Fix Version/s: 1.13.0
   Resolution: Fixed

Fixed via
* master (1.13): 72841d77802746dc12bffb5927ae0e43a681cb98

> Force an extended resource to be null if it's value is 0
> 
>
> Key: FLINK-14869
> URL: https://issues.apache.org/jira/browse/FLINK-14869
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Xintong Song
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.13.0
>
>
> {{Resource}} accepts value 0 at the moment because 0 {{CPUResource}} is used 
> in production to indicate no strict requirement for CPU cores. 
> However, keeping extended resources with value 0 is causing troubles. 
> e.g. There can be 2 ResourceSpecs, rs1 and rs2, with the same resources 
> except that rs1 contains a GPUResource with value 0 while rs2 does not 
> contains a specified GPUResource. In this case, rs1.equals(rs2)==false and 
> rs1.isLessOrEqualThan(rs2)==false, which is not correct.
> Accepting both 0 and null extended resources is also making things more 
> complex, since we need to consider it in all 
> comparisons(equals/lessThanOrEqual/isMatch) and aggregations(merge/subtract).
> To be simple, I'd propose to change it like this: in 
> ResourceSpec/ResourceProfile we only keep extended resource with positive 
> values. To be specific, if in the constructors of 
> ResourceSpec/ResourceProfile, a specified extended resource is found to be 0, 
> it must be dropped so that 0 resource would always be null.
> cc [~chesnay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] xintongsong closed pull request #14955: [FLINK-14869][core] Discard zero-valued extended resources in ResourceSpec/ResourceProfile

[GitBox]

xintongsong closed pull request #14955:
URL: https://github.com/apache/flink/pull/14955


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14955: [FLINK-14869][core] Discard zero-valued extended resources in ResourceSpec/ResourceProfile

[GitBox]

flinkbot edited a comment on pull request #14955:
URL: https://github.com/apache/flink/pull/14955#issuecomment-781090588


   
   ## CI report:
   
   * a4330f962aafed35041166c9a718e95fe5fcc214 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13442)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14944: [FLINK-21297] Support 'LOAD/UNLOAD MODULE' syntax

[GitBox]

flinkbot edited a comment on pull request #14944:
URL: https://github.com/apache/flink/pull/14944#issuecomment-779315396


   
   ## CI report:
   
   * e265855ee20db57e62a53be0f80486ff2feaef29 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13428)
 
   * 2ee9ac64332d3547bf934d7cd17f73ee5404c8b1 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13459)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14958: [FLINK-18550][sql-client] use TableResult#collect to get select result for sql client

[GitBox]

flinkbot edited a comment on pull request #14958:
URL: https://github.com/apache/flink/pull/14958#issuecomment-781106758


   
   ## CI report:
   
   * 15a2ba54d452e3f88ca92ed2820b8e58285a7058 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13441)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14944: [FLINK-21297] Support 'LOAD/UNLOAD MODULE' syntax

[GitBox]

flinkbot edited a comment on pull request #14944:
URL: https://github.com/apache/flink/pull/14944#issuecomment-779315396


   
   ## CI report:
   
   * e265855ee20db57e62a53be0f80486ff2feaef29 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13428)
 
   * 2ee9ac64332d3547bf934d7cd17f73ee5404c8b1 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14740: [FLINK-21067][runtime][checkpoint] Modify the logic of computing which tasks to trigger/ack/commit to support finished tasks

[GitBox]

flinkbot edited a comment on pull request #14740:
URL: https://github.com/apache/flink/pull/14740#issuecomment-766340750


   
   ## CI report:
   
   * c7e6b28b249f85cf52740d5201a769e0982a60aa UNKNOWN
   * bebd298009b12a9d5ac6518902f5534f8e00ff32 UNKNOWN
   * c69c79df69c5fa912b6d16e3cfa40edc3b9b82f2 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13438)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-21393) Implement ParquetAvroInputFormat

[Jira]

[ 
https://issues.apache.org/jira/browse/FLINK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286461#comment-17286461
 ] 

Dominik Wosiński commented on FLINK-21393:
--

Happy to add that.

> Implement ParquetAvroInputFormat 
> -
>
> Key: FLINK-21393
> URL: https://issues.apache.org/jira/browse/FLINK-21393
> Project: Flink
>  Issue Type: Improvement
>Reporter: Dominik Wosiński
>Priority: Minor
> Fix For: 1.13.0
>
>
>  Currently, there are several classes extending ParquetInputFormat like 
> `ParquetPojoInputFormat` or `ParquetRowInputFormat`, but there is no class 
> that would allow us to read the parquet Avro without creating additional 
> mappings and so on.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] echauchot commented on pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

echauchot commented on pull request #14961:
URL: https://github.com/apache/flink/pull/14961#issuecomment-781322363


   @fhueske as you merged the previous related PR, can you please review this 
one ?
   Thanks.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] echauchot edited a comment on pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

echauchot edited a comment on pull request #14961:
URL: https://github.com/apache/flink/pull/14961#issuecomment-781272437


   @HuangZhenQiu FYI as discussed



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-21351) Incremental checkpoint data would be lost once a non-stop savepoint completed

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286454#comment-17286454
 ] 

Till Rohrmann commented on FLINK-21351:
---

I think that this should work a bit better at the cost of higher complexity.

> Incremental checkpoint data would be lost once a non-stop savepoint completed
> -
>
> Key: FLINK-21351
> URL: https://issues.apache.org/jira/browse/FLINK-21351
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Checkpointing
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Yun Tang
>Assignee: Roman Khachatryan
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.11.4, 1.13.0, 1.12.3
>
>
> FLINK-10354 counted savepoint as retained checkpoint so that job could 
> failover from latest position. I think this operation is reasonable, however, 
> current implementation would let incremental checkpoint data lost immediately 
> once a non-stop savepoint completed.
> Current general phase of incremental checkpoints: once a newer checkpoint 
> completed, it would be added to checkpoint store. And if the size of 
> completed checkpoints larger than max retained limit, it would subsume the 
> oldest one. This lead to the reference of incremental data decrease one and 
> data would be deleted once reference reached to zero. As we always ensure to 
> register newer checkpoint and then unregister older checkpoint, current phase 
> works fine as expected.
> However, if a non-stop savepoint (a median manual trigger savepoint) is 
> completed, it would be also added into checkpoint store and just subsume 
> previous added checkpoint (in default retain one checkpoint case), which 
> would unregister older checkpoint without newer checkpoint registered, 
> leading to data lost.
> Thanks for [~banmoy] reporting this problem first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-21351) Incremental checkpoint data would be lost once a non-stop savepoint completed

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286455#comment-17286455
 ] 

Till Rohrmann commented on FLINK-21351:
---

[~pnowojski] can you help with the review?

> Incremental checkpoint data would be lost once a non-stop savepoint completed
> -
>
> Key: FLINK-21351
> URL: https://issues.apache.org/jira/browse/FLINK-21351
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Checkpointing
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Yun Tang
>Assignee: Roman Khachatryan
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.11.4, 1.13.0, 1.12.3
>
>
> FLINK-10354 counted savepoint as retained checkpoint so that job could 
> failover from latest position. I think this operation is reasonable, however, 
> current implementation would let incremental checkpoint data lost immediately 
> once a non-stop savepoint completed.
> Current general phase of incremental checkpoints: once a newer checkpoint 
> completed, it would be added to checkpoint store. And if the size of 
> completed checkpoints larger than max retained limit, it would subsume the 
> oldest one. This lead to the reference of incremental data decrease one and 
> data would be deleted once reference reached to zero. As we always ensure to 
> register newer checkpoint and then unregister older checkpoint, current phase 
> works fine as expected.
> However, if a non-stop savepoint (a median manual trigger savepoint) is 
> completed, it would be also added into checkpoint store and just subsume 
> previous added checkpoint (in default retain one checkpoint case), which 
> would unregister older checkpoint without newer checkpoint registered, 
> leading to data lost.
> Thanks for [~banmoy] reporting this problem first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] rmetzger commented on a change in pull request #14923: [DRAFT] Remove State#onEnter

[GitBox]

rmetzger commented on a change in pull request #14923:
URL: https://github.com/apache/flink/pull/14923#discussion_r578386398



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/declarative/DeclarativeScheduler.java
##
@@ -887,19 +897,22 @@ public void runIfState(State expectedState, Runnable 
action, Duration delay) {
 
 // 
 
-private void transitionToState(State newState) {
-if (state != newState) {
-LOG.debug(
-"Transition from state {} to {}.",
-state.getClass().getSimpleName(),
-newState.getClass().getSimpleName());
+@VisibleForTesting
+ void transitionToState(

Review comment:
   why did you change the semantics of "transition to same state" from 
"ignore" to "fail"?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14962: [FLINK-18789][sql-client] Use TableEnvironment#executeSql method to execute insert statement in sql client

[GitBox]

flinkbot edited a comment on pull request #14962:
URL: https://github.com/apache/flink/pull/14962#issuecomment-781306516


   
   ## CI report:
   
   * fda5571674ea74fb5c104180e3790d45c600fbea Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13458)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14822: [FLINK-21005][table-api] Introduce new provider for unified Sink API and implement in planner

[GitBox]

flinkbot edited a comment on pull request #14822:
URL: https://github.com/apache/flink/pull/14822#issuecomment-770394744


   
   ## CI report:
   
   * 4b0b591d6cc06d0eacc83f9277147d55af6ddf78 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13439)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-20427) Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to data loss

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286453#comment-17286453
 ] 

Till Rohrmann commented on FLINK-20427:
---

[~rex-remind], the sources are not affected by this problem. You have problems 
if you require your Flink job to have exactly once processing guarantees wrt to 
your sink or external systems you are interacting. If your setup tolerates 
duplicates in ES, then this is not a problem.

> Remove CheckpointConfig.setPreferCheckpointForRecovery because it can lead to 
> data loss
> ---
>
> Key: FLINK-20427
> URL: https://issues.apache.org/jira/browse/FLINK-20427
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream, Runtime / Checkpointing
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.13.0
>
>
> The {{CheckpointConfig.setPreferCheckpointForRecovery}} allows to configure 
> whether Flink prefers checkpoints for recovery if the 
> {{CompletedCheckpointStore}} contains savepoints and checkpoints. This is 
> problematic because due to this feature, Flink might prefer older checkpoints 
> over newer savepoints for recovery. Since some components expect that the 
> always the latest checkpoint/savepoint is used (e.g. the 
> {{SourceCoordinator}}), it breaks assumptions and can lead to 
> {{SourceSplits}} which are not read. This effectively means that the system 
> loses data. Similarly, this behaviour can cause that exactly once sinks might 
> output results multiple times which violates the processing guarantees. 
> Hence, I believe that we should remove this setting because it changes 
> Flink's behaviour in some very significant way potentially w/o the user 
> noticing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-21082) Document how to start multiple JobManagers in HA mode for Kubernetes deployment

[Till Rohrmann (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-21082.
-
Resolution: Duplicate

> Document how to start multiple JobManagers in HA mode for Kubernetes 
> deployment
> ---
>
> Key: FLINK-21082
> URL: https://issues.apache.org/jira/browse/FLINK-21082
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Yang Wang
>Priority: Major
>
> Some users ask in the user ML[1] for how to start multiple JobManagers in HA 
> mode for K8s deployment. Since we could not easily start JobManager pods to 
> achieve that, it is valuable to document in Flink official documentation.
>  
> [1]. 
> https://lists.apache.org/thread.html/r1b0d771eb34f871865a75b96f59dc5daf45929fe36a505ffc13c8573%40%3Cuser.flink.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-21382) Standalone K8s documentation does not explain usage of standby JobManagers

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286452#comment-17286452
 ] 

Till Rohrmann commented on FLINK-21382:
---

That's very good [~fly_in_gis]. I've actually overlooked this ticket. I will 
FLINK-21082 in favour of this ticket because here we already link to a possible 
way how to do it.

> Standalone K8s documentation does not explain usage of standby JobManagers
> --
>
> Key: FLINK-21382
> URL: https://issues.apache.org/jira/browse/FLINK-21382
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes, Documentation
>Affects Versions: 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
>
> Our [standalone K8s 
> documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html#high-availability-with-standalone-kubernetes]
>  mentions how to configure K8s HA services. It does not mention that this 
> only works with a single JobManager. When using standby JobManagers, then the 
> given deployment yamls won't work because the {{jobmanager.rpc.address}} is 
> configured to be the {{jobmanager}} service.
> Changing the configuration to work is surprisingly difficult because of a 
> lack of documentation. Moreover, it is quite difficult to pass in custom 
> configuration values when using a ConfigMap for sharing Flink's 
> {{flink-conf.yaml}}. The problem is that mounted ConfigMaps are not writable 
> from a pod perspective. See [this 
> answer|https://stackoverflow.com/a/66228073/4815083] for how one could 
> achieve it.
> I think we could improve our documentation to explain our users how to 
> configure a standalone HA cluster with standby JobManagers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (FLINK-18171) Do not take client side config options to cluster

[Till Rohrmann (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-18171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-18171.
-
Resolution: Later

> Do not take client side config options to cluster
> -
>
> Key: FLINK-18171
> URL: https://issues.apache.org/jira/browse/FLINK-18171
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes, Deployment / YARN, Runtime / 
> Configuration
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Priority: Major
>
> Follow the discussion in this PR[1], some client side config options should 
> not be taken into the cluster. Once it takes effect, it will cause some 
> issue(e.g. FLINK-18149).
> For K8s deployment, we have explicitly remove 
> {{KubernetesConfigOptions.KUBE_CONFIG_FILE}} and 
> {{DeploymentOptionsInternal.CONF_DIR}}. 
> For Yarn deployment, we have at least two options 
> {{DeploymentOptionsInternal.CONF_DIR}} and 
> {{YarnConfigOptionsInternal.APPLICATION_LOG_CONFIG_FILE}} could be removed in 
> client side.
>  
> Benefit from this, we could avoid some unexpected configuration loading or 
> secret issues. Also we could avoid the logs in the jobmanager confusing the 
> users since it is a client local path.
> {code:java}
> 2020-06-05 14:38:38,656 INFO  
> org.apache.flink.configuration.GlobalConfiguration   [] - Loading 
> configuration property: $internal.deployment.config-dir, 
> /home/danrtsey.wy/flink-1.11-SNAPSHOT/conf
> 2020-06-05 14:38:38,656 INFO  
> org.apache.flink.configuration.GlobalConfiguration   [] - Loading 
> configuration property: $internal.yarn.log-config-file, 
> /home/danrtsey.wy/flink-1.11-SNAPSHOT/conf/log4j.properties
> {code}
>  
> [1]. https://github.com/apache/flink/pull/12501#pullrequestreview-425452351



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-21383) Docker image does not play well together with ConfigMap based flink-conf.yamls

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286450#comment-17286450
 ] 

Till Rohrmann commented on FLINK-21383:
---

The failure messages are unfortunately not very descriptive. Maybe we can 
improve this a bit.

Yes making the configuration configurable within the Flink process should solve 
the process. One way would be to use the dynamic properties for that. A related 
idea is to implement 
[FLIP-161|https://cwiki.apache.org/confluence/display/FLINK/FLIP-161%3A+Configuration+through+envrionment+variables?focusedCommentId=173083588].

> Docker image does not play well together with ConfigMap based flink-conf.yamls
> --
>
> Key: FLINK-21383
> URL: https://issues.apache.org/jira/browse/FLINK-21383
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / Kubernetes, flink-docker
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Till Rohrmann
>Priority: Major
>  Labels: usability
>
> Flink's Docker image does not play well together with ConfigMap based 
> flink-conf.yamls. The {{docker-entrypoint.sh}} script offers a few env 
> variables to overwrite configuration values (e.g. {{FLINK_PROPERTIES}}, 
> {{JOB_MANAGER_RPC_ADDRESS}}, etc.). The problem is that the entrypoint script 
> assumes that it can modify the existing {{flink-conf.yaml}}. This is not the 
> case if the {{flink-conf.yaml}} is based on a {{ConfigMap}}.
> Making things worse, failures updating the {{flink-conf.yaml}} are not 
> reported. Moreover, the called {{jobmanager.sh}} and {{taskmanager.sh}} 
> scripts don't support to pass in dynamic configuration properties into the 
> processes.
> I think the problem is that our assumption that we can modify the 
> {{flink-conf.yaml}} does not always hold true. If we updated the final 
> configuration from within the Flink process (dynamic properties and env 
> variables), then this problem could be avoided.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-18171) Do not take client side config options to cluster

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-18171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286451#comment-17286451
 ] 

Till Rohrmann commented on FLINK-18171:
---

Alright, then I will close this issue and mark it as "later".

> Do not take client side config options to cluster
> -
>
> Key: FLINK-18171
> URL: https://issues.apache.org/jira/browse/FLINK-18171
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Kubernetes, Deployment / YARN, Runtime / 
> Configuration
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Priority: Major
>
> Follow the discussion in this PR[1], some client side config options should 
> not be taken into the cluster. Once it takes effect, it will cause some 
> issue(e.g. FLINK-18149).
> For K8s deployment, we have explicitly remove 
> {{KubernetesConfigOptions.KUBE_CONFIG_FILE}} and 
> {{DeploymentOptionsInternal.CONF_DIR}}. 
> For Yarn deployment, we have at least two options 
> {{DeploymentOptionsInternal.CONF_DIR}} and 
> {{YarnConfigOptionsInternal.APPLICATION_LOG_CONFIG_FILE}} could be removed in 
> client side.
>  
> Benefit from this, we could avoid some unexpected configuration loading or 
> secret issues. Also we could avoid the logs in the jobmanager confusing the 
> users since it is a client local path.
> {code:java}
> 2020-06-05 14:38:38,656 INFO  
> org.apache.flink.configuration.GlobalConfiguration   [] - Loading 
> configuration property: $internal.deployment.config-dir, 
> /home/danrtsey.wy/flink-1.11-SNAPSHOT/conf
> 2020-06-05 14:38:38,656 INFO  
> org.apache.flink.configuration.GlobalConfiguration   [] - Loading 
> configuration property: $internal.yarn.log-config-file, 
> /home/danrtsey.wy/flink-1.11-SNAPSHOT/conf/log4j.properties
> {code}
>  
> [1]. https://github.com/apache/flink/pull/12501#pullrequestreview-425452351



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #14962: [FLINK-18789][sql-client] Use TableEnvironment#executeSql method to execute insert statement in sql client

[GitBox]

flinkbot commented on pull request #14962:
URL: https://github.com/apache/flink/pull/14962#issuecomment-781306516


   
   ## CI report:
   
   * fda5571674ea74fb5c104180e3790d45c600fbea UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14943: [FLINK-21354] Implement a StateChangelogStateBackend to forward state changes to St…

[GitBox]

flinkbot edited a comment on pull request #14943:
URL: https://github.com/apache/flink/pull/14943#issuecomment-779315285


   
   ## CI report:
   
   * f80b97c479ec82663e971553c54c8a6cda3122ff UNKNOWN
   * 3e1cf6d7f7f645e991c0b788f653bab7b1baa797 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13453)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14962: [FLINK-18789][sql-client] Use TableEnvironment#executeSql method to execute insert statement in sql client

[GitBox]

flinkbot commented on pull request #14962:
URL: https://github.com/apache/flink/pull/14962#issuecomment-781302347


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit fda5571674ea74fb5c104180e3790d45c600fbea (Thu Feb 18 
12:15:38 UTC 2021)
   
   **Warnings:**
* **1 pom.xml files were touched**: Check for build and licensing issues.
* Documentation files were touched, but no `.zh.md` files: Update Chinese 
documentation or file Jira ticket.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] fsk119 opened a new pull request #14962: [FLINK-18789][sql-client] Use TableEnvironment#executeSql method to execute insert statement in sql client

[GitBox]

fsk119 opened a new pull request #14962:
URL: https://github.com/apache/flink/pull/14962


   
   
   ## What is the purpose of the change
   
   *sql client has a lot of logic to execute an insert job, which can be 
simplified through executeSql method. This pr aims to do the improvement*
   
   *This pr is finished by godfrey. I just rebase this PR to the master.*
   
   ## Brief change log
   
   - remove `executeUpdate` method from Executor
   - execute insert statement through `executeSql` method in CliClient
   - remove the unused code
   
   ## Verifying this change
   
   *This change is already covered by existing tests, such as 
`LocalExecutorITCase`.*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / 
don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (**not applicable** / docs / 
JavaDocs / not documented)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

flinkbot edited a comment on pull request #14961:
URL: https://github.com/apache/flink/pull/14961#issuecomment-781278553


   
   ## CI report:
   
   * a35e534ab18f7d4c80a8ee52d7dd70b7e3a0f2a0 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13455)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14954: [FLINK-17061][runtime] Unset TM total process/flink memory size for fine-grained resource management

[GitBox]

flinkbot edited a comment on pull request #14954:
URL: https://github.com/apache/flink/pull/14954#issuecomment-781063841


   
   ## CI report:
   
   * de5409c4a93a18d7c664e6483036222985e673c4 Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13436)
 
   * 709ba9478b5c13b183bf66d73294d8133438a8f5 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13454)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14943: [FLINK-21354] Implement a StateChangelogStateBackend to forward state changes to St…

[GitBox]

flinkbot edited a comment on pull request #14943:
URL: https://github.com/apache/flink/pull/14943#issuecomment-779315285


   
   ## CI report:
   
   * f80b97c479ec82663e971553c54c8a6cda3122ff UNKNOWN
   * fb2643d0aa19f9a4ea3106061c1f9c113abbd5d8 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13393)
 
   * 3e1cf6d7f7f645e991c0b788f653bab7b1baa797 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] dianfu closed pull request #14957: [FLINK-20534] Add Flink 1.12 MigrationVersion and add Flink 1.12 snapshots for TypeSerializer upgrade tests

[GitBox]

dianfu closed pull request #14957:
URL: https://github.com/apache/flink/pull/14957


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] dianfu commented on pull request #14957: [FLINK-20534] Add Flink 1.12 MigrationVersion and add Flink 1.12 snapshots for TypeSerializer upgrade tests

[GitBox]

dianfu commented on pull request #14957:
URL: https://github.com/apache/flink/pull/14957#issuecomment-781291040


   Closed via 570e40c21560f5f2a2e270739f507708abeafb17 and 
68e82653ce21a291ea61edccd0557ddfc5f91c05



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Closed] (FLINK-20534) Add Flink 1.12 MigrationVersion

[Dian Fu (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dian Fu closed FLINK-20534.
---
Fix Version/s: (was: 1.12.0)
   1.12.3
   1.13.0
   Resolution: Fixed

Merged to 
- master via 2fb46aa54d8b95b1ca467eab0120e4279f1df2fa and 
6875421cfdc0d4d68e6dcd55809d61c188c647a1
- release-1.12 via 570e40c21560f5f2a2e270739f507708abeafb17 and 
68e82653ce21a291ea61edccd0557ddfc5f91c05

> Add Flink 1.12 MigrationVersion
> ---
>
> Key: FLINK-20534
> URL: https://issues.apache.org/jira/browse/FLINK-20534
> Project: Flink
>  Issue Type: Task
>  Components: API / Core
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.13.0, 1.12.3
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21397) BufferUnderflowException when read parquet

[lihe ma (Jira)]

lihe ma created FLINK-21397:
---

 Summary: BufferUnderflowException when read parquet 
 Key: FLINK-21397
 URL: https://issues.apache.org/jira/browse/FLINK-21397
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.12.1
Reporter: lihe ma
 Attachments: 
part-f33924c5-99c3-4177-9a9a-e2d5c71a799a-1-2324.snappy.parquet

error when read parquet file . 

when the encoding of all pages in parquet file is PLAIN_DICTIONARY , it works 
well , but  if   parquet file contains 3 pages, and the encoding of page0 and 
page1 is PLAIN_DICTIONARY, page2 is PLAIN  , then flink throw exception after 
page0 and page1 read finish.
the souurce parquet file is write by flink 1.11.

 

the parquet file info :

{{row group 0}}
{{}}
{{oid: BINARY SNAPPY DO:0 FPO:4 SZ:625876/1748820/2.79 VC:95192 ENC:BIT 
[more]...}}{{oid TV=95192 RL=0 DL=1 DS: 36972 DE:PLAIN_DICTIONARY}}
{{ 
}}
{{ page 0: DLE:RLE RLE:BIT_PACKED VLE:PLAIN_DICTIONARY [more]... SZ:70314}}
{{ page 1: DLE:RLE RLE:BIT_PACKED VLE:PLAIN_DICTIONARY [more]... SZ:74850}}
{{ page 2: DLE:RLE RLE:BIT_PACKED VLE:PLAIN ST:[m [more]... SZ:568184 }}
{{BINARY oid}}

exception msg:
{code:java}
Caused by: java.nio.BufferUnderflowExceptionCaused by: 
java.nio.BufferUnderflowException at 
java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151) at 
java.nio.ByteBuffer.get(ByteBuffer.java:715) at 
org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes(Binary.java:422)
 at 
org.apache.flink.formats.parquet.vector.reader.BytesColumnReader.readBatchFromDictionaryIds(BytesColumnReader.java:77)
 at 
org.apache.flink.formats.parquet.vector.reader.BytesColumnReader.readBatchFromDictionaryIds(BytesColumnReader.java:31)
 at 
org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:186)
 at 
org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.nextBatch(ParquetVectorizedInputFormat.java:363)
 at 
org.apache.flink.formats.parquet.ParquetVectorizedInputFormat$ParquetReader.readBatch(ParquetVectorizedInputFormat.java:334)
 at 
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:71)
 at 
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56)
 at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:138)
 ... 6 more
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] dianfu closed pull request #14956: [FLINK-20534] Add Flink 1.12 MigrationVersion and add Flink 1.12 snapshots for TypeSerializer upgrade tests

[GitBox]

dianfu closed pull request #14956:
URL: https://github.com/apache/flink/pull/14956


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.

[Xiaoguang Sun (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286428#comment-17286428
 ] 

Xiaoguang Sun commented on FLINK-21108:
---

Hi [~trohrmann]

I have done some work to make rest client possible to authenticate itself with 
proxy against both basic and digest challenges. I still need some time to add 
more test cases however in case there are some changes need to be revised, I 
have pushed the changes to [my 
repository|https://github.com/sunxiaoguang/flink/commit/83b136784b706b06f4b36d2bb02d546ad0685538]
 so you can take a look and give me some feedbacks.

Please take a look at where it restart requests after receiving authentication 
challenges, I feel particularly not good about it but have no better idea at 
this time. Thanks 

> Flink runtime rest server and history server webmonitor do not require 
> authentication.
> --
>
> Key: FLINK-21108
> URL: https://issues.apache.org/jira/browse/FLINK-21108
> Project: Flink
>  Issue Type: New Feature
>  Components: Runtime / REST, Runtime / Web Frontend
>Reporter: Xiaoguang Sun
>Assignee: Xiaoguang Sun
>Priority: Major
>  Labels: pull-request-available
>
> Flink runtime rest server and history server webmonitor do not require 
> authentication. At certain scenarios, prohibiting unauthorized access is 
> desired. Http basic authentication can be used here.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] tillrohrmann commented on a change in pull request #14897: [FLINK-21221][runtime] Deduplication for multiple ResourceCounters

[GitBox]

tillrohrmann commented on a change in pull request #14897:
URL: https://github.com/apache/flink/pull/14897#discussion_r578344017



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/DeclarativeSlotManager.java
##
@@ -600,61 +603,69 @@ private void allocateSlot(
 FutureUtils.assertNoException(slotAllocationResponseProcessingFuture);
 }
 
-private void tryFulfillRequirementsWithPendingSlots(
+private ResourceCounter tryFulfillRequirementsWithPendingSlots(
 JobID jobId,
-Map missingResources,
+Collection> missingResources,
 ResourceCounter pendingSlots) {
-for (Map.Entry missingResource : 
missingResources.entrySet()) {
+for (Map.Entry missingResource : 
missingResources) {
 ResourceProfile profile = missingResource.getKey();
 for (int i = 0; i < missingResource.getValue(); i++) {
-if (!tryFulfillWithPendingSlots(profile, pendingSlots)) {
-boolean couldAllocateWorkerAndReserveSlot =
+final Tuple2 matchingResult =
+tryFulfillWithPendingSlots(profile, pendingSlots);
+pendingSlots = matchingResult.f1;
+if (!matchingResult.f0) {
+final Tuple2 allocationResult =
 tryAllocateWorkerAndReserveSlot(profile, 
pendingSlots);
+pendingSlots = allocationResult.f1;
+boolean couldAllocateWorkerAndReserveSlot = 
allocationResult.f0;
 if (!couldAllocateWorkerAndReserveSlot && 
sendNotEnoughResourceNotifications) {
 LOG.warn("Could not fulfill resource requirements of 
job {}.", jobId);
 resourceActions.notifyNotEnoughResourcesAvailable(
 jobId, 
resourceTracker.getAcquiredResources(jobId));
-return;
+return pendingSlots;
 }
 }
 }
 }
+return pendingSlots;
 }
 
-private boolean tryFulfillWithPendingSlots(
+private Tuple2 tryFulfillWithPendingSlots(

Review comment:
   I would recommend using an explicit type. Something like 
`MatchingResult` which tells you whether the matching was successful or not. In 
the former case, it also gives you the newly available pendingSlots back.
   
   The problem with `Tuple2` is that it is not very expressive. You only have 
`Tuple2.f0` and `Tuple2.f1`. This is not very helpful when someone new to the 
code wants to understand the code. Having something like `MatchingResult` with 
`MatchingResult.isSuccess` or `MatchingResult.isSuccessfulMatching` tells you 
more about the what the intention is.

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/util/ResourceCounter.java
##
@@ -53,6 +53,16 @@ public int getResourceCount(ResourceProfile resourceProfile) 
{
 return resources.getOrDefault(resourceProfile, 0);
 }
 
+/**
+ * Adds increment to this resource counter value and returns the resulting 
value. Number of
+ * resources of all the {@link ResourceProfile} in this counter.
+ *
+ * @return number of resources of all the {@link ResourceProfile} in this 
counter
+ */
+public int getResourceCount() {

Review comment:
   Maybe call it `getTotalResourceCount`.

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/BiDirectionalResourceToRequirementMapping.java
##
@@ -74,23 +81,26 @@ private static void internalDecrementCount(
 "Attempting to decrement count of %s->%s, but 
primary key was unknown.",
 resourceProfile,
 secondaryKey);
-resourceCounter.decrementCount(secondaryKey, decrement);
-return resourceCounter.isEmpty() ? null : resourceCounter;
+final ResourceCounter newCounter =
+resourceCounter.subtract(secondaryKey, decrement);
+return newCounter.isEmpty() ? null : newCounter;
 });
 }
 
-public Map 
getResourcesFulfilling(ResourceProfile requirement) {
+public Collection> 
getResourcesFulfilling(

Review comment:
   Instead of returning `Collection>` 
couldn't we also return a `ResourceCounter`? Given that this class is immutable 
it should not be a problem to hand it out.

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/BiDirectionalResourceToRequirementMapping.java
##
@@ -74,23 +81,26 @@ private static void internalDecrementCount(
 "Attempting to decrement count of %s->%s, but 
primary key was unknown.",
 resourceProfile,
   

[GitHub] [flink] flinkbot commented on pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

flinkbot commented on pull request #14961:
URL: https://github.com/apache/flink/pull/14961#issuecomment-781278553


   
   ## CI report:
   
   * a35e534ab18f7d4c80a8ee52d7dd70b7e3a0f2a0 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14960: fix scrolling issue in Firefox

[GitBox]

flinkbot edited a comment on pull request #14960:
URL: https://github.com/apache/flink/pull/14960#issuecomment-781267077


   
   ## CI report:
   
   * c195cee17a3fa141bcf497f137495a54e55041f7 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13452)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14957: [FLINK-20534] Add Flink 1.12 MigrationVersion and add Flink 1.12 snapshots for TypeSerializer upgrade tests

[GitBox]

flinkbot edited a comment on pull request #14957:
URL: https://github.com/apache/flink/pull/14957#issuecomment-781106649


   
   ## CI report:
   
   * dc2f8d2db78483a1d21a376ac0ff36da497f977d Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13434)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14956: [FLINK-20534] Add Flink 1.12 MigrationVersion and add Flink 1.12 snapshots for TypeSerializer upgrade tests

[GitBox]

flinkbot edited a comment on pull request #14956:
URL: https://github.com/apache/flink/pull/14956#issuecomment-781106539


   
   ## CI report:
   
   * a05e0c6456256e78658fbfceb4a9dfcb72fe2728 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13433)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14954: [FLINK-17061][runtime] Unset TM total process/flink memory size for fine-grained resource management

[GitBox]

flinkbot edited a comment on pull request #14954:
URL: https://github.com/apache/flink/pull/14954#issuecomment-781063841


   
   ## CI report:
   
   * 6746ea0922384476c2f9f4ad4fcae1273811201f Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13429)
 
   * de5409c4a93a18d7c664e6483036222985e673c4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13436)
 
   * 709ba9478b5c13b183bf66d73294d8133438a8f5 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14953: [FLINK-21351][checkpointing] Don't subsume last checkpoint

[GitBox]

flinkbot edited a comment on pull request #14953:
URL: https://github.com/apache/flink/pull/14953#issuecomment-780752668


   
   ## CI report:
   
   * 941374ecdd03b5ceb9245e4d8a8370671b691a34 UNKNOWN
   * b495b70cd52846372f8e14cd08facd234235f984 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13425)
 
   * 07d008483dd496a4b89fe34e799511e3e2670fb8 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13451)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14946: [FLINK-21381][docs] Add information about service account permissions to K8s HA service documentation

[GitBox]

flinkbot edited a comment on pull request #14946:
URL: https://github.com/apache/flink/pull/14946#issuecomment-779866080


   
   ## CI report:
   
   * 3acf861c7e288f9ce7c3b0fec5bc513c75ea8f18 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13375)
 
   * eacbdcd5b9a6d60747e81f91208b102b53a5cafa Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13450)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14897: [FLINK-21221][runtime] Deduplication for multiple ResourceCounters

[GitBox]

flinkbot edited a comment on pull request #14897:
URL: https://github.com/apache/flink/pull/14897#issuecomment-775050448


   
   ## CI report:
   
   * 35644082e06622b2182b6fb68d9a30df167d5c11 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13432)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

flinkbot commented on pull request #14961:
URL: https://github.com/apache/flink/pull/14961#issuecomment-781272978


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit a35e534ab18f7d4c80a8ee52d7dd70b7e3a0f2a0 (Thu Feb 18 
11:19:17 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] echauchot commented on pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

echauchot commented on pull request #14961:
URL: https://github.com/apache/flink/pull/14961#issuecomment-781272437


   @HuangZhenQiu can you ping a committer as you proposed to review this PR ?
   Thanks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-21388) Parquet DECIMAL logical type is not properly supported in ParquetSchemaConverter

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-21388:
---
Labels: pull-request-available  (was: )

> Parquet DECIMAL logical type is not properly supported in 
> ParquetSchemaConverter
> 
>
> Key: FLINK-21388
> URL: https://issues.apache.org/jira/browse/FLINK-21388
> Project: Flink
>  Issue Type: Bug
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table 
> SQL / Ecosystem
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Labels: pull-request-available
>
> For a parquet field such as this one 
> {code:java}
> optional int32 ss_wholesale_cost (DECIMAL(7,2));{code}
> _ParquetSchemaConverter_ raises the exception:
> {code:java}
> Unsupported original type : DECIMAL for primitive type INT32{code}
> whereas it should be mapped to flink _DECIMAL_ Datatype (_BIG_DEC_ type 
> information) owing to 
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connectors/formats/parquet.html#data-type-mapping
>  
> and 
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/types.html#old-planner



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] echauchot opened a new pull request #14961: [FLINK-21388] [flink-parquet] support DECIMAL parquet logical type when parquet primitive type is INT32

[GitBox]

echauchot opened a new pull request #14961:
URL: https://github.com/apache/flink/pull/14961


   ## What is the purpose of the change
   
   support DECIMAL parquet logical type when parquet primitive type is INT32
   
   ## Brief change log
   
   The support for INT64 was already done (matching _int64/DECIMAL_ to 
_BasicTypeInfo.INT_TYPE_INFO_)
   Simply added support for INT32 (matching _int32/DECIMAL_ to 
_BasicTypeInfo.INT_TYPE_INFO_)
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   ParquetSchemaConverterTest#testDecimalParquetLogicalType
   
   ## Does this pull request potentially affect one of the following parts:
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (FLINK-21396) Update the Catalog APIs

[Timo Walther (Jira)]

Timo Walther created FLINK-21396:


 Summary: Update the Catalog APIs
 Key: FLINK-21396
 URL: https://issues.apache.org/jira/browse/FLINK-21396
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther


Update the Catalog APIs according to FLIP-164 and make sure to maintain 
backwards compatibility.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21395) Implement Schema, ResolvedSchema, SchemaResolver

[Timo Walther (Jira)]

Timo Walther created FLINK-21395:


 Summary: Implement Schema, ResolvedSchema, SchemaResolver
 Key: FLINK-21395
 URL: https://issues.apache.org/jira/browse/FLINK-21395
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther


Introduces the main classes and utilities around schema mentioned in FLIP-164.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21394) FLIP-164: Improve Schema Handling in Catalogs

[Timo Walther (Jira)]

Timo Walther created FLINK-21394:


 Summary: FLIP-164: Improve Schema Handling in Catalogs
 Key: FLINK-21394
 URL: https://issues.apache.org/jira/browse/FLINK-21394
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther


Schema information is necessary at different locations in the Table API for 
defining tables and/or views. In particular, it is necessary to define the 
schema in a programmatic DDL (FLIP-129) and when converting a DataStream to a 
Table (FLIP-136).

We need similar APIs in the Catalog interfaces such that catalog 
implementations can define table/views in a unified way.

This FLIP updates the class hierarchy to achieve the following goals:

- make it visible whether a schema is resolved or unresolved and when the 
resolution happens
- offer a unified API for FLIP-129, FLIP-136, and catalogs
- allow arbitrary data types and expressions in the schema for watermark spec 
or columns
- have access to other catalogs for declaring a data type or expression via 
CatalogManager
- cleaned up TableSchema
- remain backwards compatible in the persisted properties and API



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] flinkbot commented on pull request #14960: fix scrolling issue in Firefox

[GitBox]

flinkbot commented on pull request #14960:
URL: https://github.com/apache/flink/pull/14960#issuecomment-781267077


   
   ## CI report:
   
   * c195cee17a3fa141bcf497f137495a54e55041f7 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14955: [FLINK-14869][core] Discard zero-valued extended resources in ResourceSpec/ResourceProfile

[GitBox]

flinkbot edited a comment on pull request #14955:
URL: https://github.com/apache/flink/pull/14955#issuecomment-781090588


   
   ## CI report:
   
   * 216d83acae21669fd504d4562524a7e9276062a8 Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13437)
 
   * a4330f962aafed35041166c9a718e95fe5fcc214 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13442)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14950: [FLINK-21347][coordination] Extract interface from ExecutionGraph

[GitBox]

flinkbot edited a comment on pull request #14950:
URL: https://github.com/apache/flink/pull/14950#issuecomment-780431068


   
   ## CI report:
   
   * bee61356ebeb5f99c96ce3193b3bd8205e61e44d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13409)
 
   * 1b43a411d22af19f12f14a79c1094c8dc5cfeb42 UNKNOWN
   * e6aae42cf093d4f44b93b79a78bdc64538df8725 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13445)
 
   * a8191e6800961467c726695c8ef7575d157a Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13449)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14946: [FLINK-21381][docs] Add information about service account permissions to K8s HA service documentation

[GitBox]

flinkbot edited a comment on pull request #14946:
URL: https://github.com/apache/flink/pull/14946#issuecomment-779866080


   
   ## CI report:
   
   * 3acf861c7e288f9ce7c3b0fec5bc513c75ea8f18 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13375)
 
   * eacbdcd5b9a6d60747e81f91208b102b53a5cafa UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] curcur commented on a change in pull request #14799: [FLINK-21354][WIP] Wrapped StateBackend to forward state changes to StateChangleLog

[GitBox]

curcur commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r578323771



##
File path: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManagerImpl.java
##
@@ -99,23 +101,30 @@ private InternalTimeServiceManagerImpl(
  * IMPORTANT: Keep in sync with {@link 
InternalTimeServiceManager.Provider}.
  */
 public static  InternalTimeServiceManagerImpl create(
-CheckpointableKeyedStateBackend keyedStatedBackend,
+CheckpointableKeyedStateBackend keyedStateBackend,
 ClassLoader userClassloader,
 KeyContext keyContext,
 ProcessingTimeService processingTimeService,
 Iterable rawKeyedStates)
 throws Exception {
-final KeyGroupRange keyGroupRange = 
keyedStatedBackend.getKeyGroupRange();
+final KeyGroupRange keyGroupRange = 
keyedStateBackend.getKeyGroupRange();
+
+KeyedStateBackend rootKeyedStateBackend =
+keyedStateBackend instanceof ProxyKeyedStateBackend
+? ((ProxyKeyedStateBackend) keyedStateBackend)
+.getProxiedKeyedStateBackend()
+: keyedStateBackend;
+
 final boolean requiresSnapshotLegacyTimers =
-keyedStatedBackend instanceof AbstractKeyedStateBackend
-&& ((AbstractKeyedStateBackend) keyedStatedBackend)
+rootKeyedStateBackend instanceof AbstractKeyedStateBackend
+&& ((AbstractKeyedStateBackend) 
rootKeyedStateBackend)

Review comment:
   Yes, that's a good idea





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] curcur commented on a change in pull request #14799: [FLINK-21354][WIP] Wrapped StateBackend to forward state changes to StateChangleLog

[GitBox]

curcur commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r578323324



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointStorageLoader.java
##
@@ -162,14 +163,19 @@ public static CheckpointStorage load(
 Preconditions.checkNotNull(classLoader, "classLoader");
 Preconditions.checkNotNull(configuredStateBackend, "statebackend");
 
-if (configuredStateBackend instanceof CheckpointStorage) {
+StateBackend rootStateBackend =
+(configuredStateBackend instanceof ProxyStateBackend)
+? ((ProxyStateBackend) 
configuredStateBackend).getProxiedStateBackend()
+: configuredStateBackend;

Review comment:
   This is the only place needing to unwrap for "state backend"; and it is 
because some of the backends are both state backend + checkpoint storage (for 
now).
   
   There are some other places to check whether it is XXXKeyedStateBackend 
might be cofusing with this one.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14959: [FLINK-16947][Azure] Attempt to fix maven network issues

[GitBox]

flinkbot edited a comment on pull request #14959:
URL: https://github.com/apache/flink/pull/14959#issuecomment-781241815


   
   ## CI report:
   
   * 3e3889ee5cc45f7d8dd654b9c5296a80ab6e88e0 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13448)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14950: [FLINK-21347][coordination] Extract interface from ExecutionGraph

[GitBox]

flinkbot edited a comment on pull request #14950:
URL: https://github.com/apache/flink/pull/14950#issuecomment-780431068


   
   ## CI report:
   
   * bee61356ebeb5f99c96ce3193b3bd8205e61e44d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13409)
 
   * 1b43a411d22af19f12f14a79c1094c8dc5cfeb42 UNKNOWN
   * e6aae42cf093d4f44b93b79a78bdc64538df8725 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13445)
 
   * a8191e6800961467c726695c8ef7575d157a UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] tillrohrmann commented on pull request #14946: [FLINK-21381][docs] Add information about service account permissions to K8s HA service documentation

[GitBox]

tillrohrmann commented on pull request #14946:
URL: https://github.com/apache/flink/pull/14946#issuecomment-781252287


   Thanks for the review @XComp and @wangyang0918. I have updated the 
documentation. Please take another look whether it is now clearer.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14953: [FLINK-21351][checkpointing] Don't subsume last checkpoint

[GitBox]

flinkbot edited a comment on pull request #14953:
URL: https://github.com/apache/flink/pull/14953#issuecomment-780752668


   
   ## CI report:
   
   * 941374ecdd03b5ceb9245e4d8a8370671b691a34 UNKNOWN
   * b495b70cd52846372f8e14cd08facd234235f984 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13425)
 
   * 07d008483dd496a4b89fe34e799511e3e2670fb8 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lovelock commented on pull request #14848: [FLINK-21268]Fix scrolling issue in Firefox

[GitBox]

lovelock commented on pull request #14848:
URL: https://github.com/apache/flink/pull/14848#issuecomment-781251547


   @vthinkxie please see this https://github.com/apache/flink/pull/14960



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lovelock closed pull request #14848: [FLINK-21268]Fix scrolling issue in Firefox

[GitBox]

lovelock closed pull request #14848:
URL: https://github.com/apache/flink/pull/14848


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14960: fix scrolling issue in Firefox

[GitBox]

flinkbot commented on pull request #14960:
URL: https://github.com/apache/flink/pull/14960#issuecomment-781251002


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit c195cee17a3fa141bcf497f137495a54e55041f7 (Thu Feb 18 
10:40:35 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **Invalid pull request title: No valid Jira ID provided**
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lovelock opened a new pull request #14960: fix scrolling issue in Firefox

[GitBox]

lovelock opened a new pull request #14960:
URL: https://github.com/apache/flink/pull/14960


   
   
   ## What is the purpose of the change
   
   1. Fix scrolling bug in Firefox. Logs view both in TaskManager and 
JobManager can't scroll with mouse in Firefox because of the bug in 
monaco-editor of version 0.16.0, which is mentioned 
[here](https://github.com/microsoft/monaco-editor/issues/1353)
   
   ## Brief change log
   
   1. Upgrade monaco-editor from `^0.16.0` to `^0.18.0`. The current version of 
monaco-editor is 0.22.3, but it won't compile.
   
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-21362) Remove State#onEnter

[Robert Metzger (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286405#comment-17286405
 ] 

Robert Metzger commented on FLINK-21362:


Draft PR: https://github.com/apache/flink/pull/14923


> Remove State#onEnter
> 
>
> Key: FLINK-21362
> URL: https://issues.apache.org/jira/browse/FLINK-21362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination
>Reporter: Chesnay Schepler
>Assignee: Robert Metzger
>Priority: Major
> Fix For: 1.13.0
>
>
> Since we no longer need to construct the new state before calling 
> {{State#onLeave}} we could remove {{State#onEnter}} and move all contained 
> logic into the constructor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] rmetzger commented on pull request #14950: [FLINK-21347][coordination] Extract interface from ExecutionGraph

[GitBox]

rmetzger commented on pull request #14950:
URL: https://github.com/apache/flink/pull/14950#issuecomment-781246489


   Rebased onto DeclScheduler merge on master



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14959: [FLINK-16947][Azure] Attempt to fix maven network issues

[GitBox]

flinkbot commented on pull request #14959:
URL: https://github.com/apache/flink/pull/14959#issuecomment-781241815


   
   ## CI report:
   
   * 3e3889ee5cc45f7d8dd654b9c5296a80ab6e88e0 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14916: [FLINK-21345][Table SQL / Planner] Fix BUG of Union All join Temporal…

[GitBox]

flinkbot edited a comment on pull request #14916:
URL: https://github.com/apache/flink/pull/14916#issuecomment-776614443


   
   ## CI report:
   
   * bf763e66a66853764f8bb39c07db8d86c98916aa Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13193)
 
   * 13b22ab740f81a326927caa4f344bbafab336a54 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13447)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API

[GitBox]

flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058


   
   ## CI report:
   
   * f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
   * 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420)
 
   * 000fa3e28def66c51ebf7b5975d8b6106988 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13446)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] tillrohrmann commented on a change in pull request #14946: [FLINK-21381][docs] Add information about service account permissions to K8s HA service documentation

[GitBox]

tillrohrmann commented on a change in pull request #14946:
URL: https://github.com/apache/flink/pull/14946#discussion_r578292069



##
File path: docs/content.zh/docs/deployment/ha/kubernetes_ha.md
##
@@ -31,6 +31,14 @@ Flink's Kubernetes HA services use 
[Kubernetes](https://kubernetes.io/) for high
 Kubernetes high availability services can only be used when deploying to 
Kubernetes.
 Consequently, they can be configured when using [standalone Flink on 
Kubernetes]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" 
>}}) or the [native Kubernetes integration]({{< ref 
"docs/deployment/resource-providers/native_kubernetes" >}})
 
+## Prerequisites
+
+In order to use Flink's Kubernetes HA services you must fulfill the following 
prerequisites:
+
+- Kubernetes >= 1.9.
+- `default` service account with [RBAC]({{< ref 
"docs/deployment/resource-providers/native_kubernetes" >}}#rbac) permissions to 
create, edit, delete ConfigMaps.

Review comment:
   Alright, then let me update this PR a bit.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lovelock commented on pull request #14848: [FLINK-21268]Fix scrolling issue in Firefox

[GitBox]

lovelock commented on pull request #14848:
URL: https://github.com/apache/flink/pull/14848#issuecomment-781235336


   > Hi @lovelock
   > `@ant-design/icons-angular` is the dependency of ng-zorro-antd, which 
should be installed automatically.
   > can you check your npm version via run `npm version`?
   
   I've noticed the differences between two `package-lock.json` due to 
`lockFileVersion` so I would change node version and try it later.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] lovelock commented on pull request #14848: [FLINK-21268]Fix scrolling issue in Firefox

[GitBox]

lovelock commented on pull request #14848:
URL: https://github.com/apache/flink/pull/14848#issuecomment-781232691


   I've hidden 3 abused comments by myself because all the errors I run into 
are because network problem. However `git-commit-id-plugin` would cause OOM 
error because of network problem is somewhat confusing.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14954: [FLINK-17061][runtime] Unset TM total process/flink memory size for fine-grained resource management

[GitBox]

flinkbot edited a comment on pull request #14954:
URL: https://github.com/apache/flink/pull/14954#issuecomment-781063841


   
   ## CI report:
   
   * 6746ea0922384476c2f9f4ad4fcae1273811201f Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13429)
 
   * de5409c4a93a18d7c664e6483036222985e673c4 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13436)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14928: [FLINK-21360][coordination] Make resource timeout configurable

[GitBox]

flinkbot edited a comment on pull request #14928:
URL: https://github.com/apache/flink/pull/14928#issuecomment-14828


   
   ## CI report:
   
   * fa4a474ab1addab8caafd64686a1a9d029843556 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13344)
 
   * 6be910fe164322d9862bad9c0d80f8ff4ee5fe62 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13444)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14950: [FLINK-21347][coordination] Extract interface from ExecutionGraph

[GitBox]

flinkbot edited a comment on pull request #14950:
URL: https://github.com/apache/flink/pull/14950#issuecomment-780431068


   
   ## CI report:
   
   * bee61356ebeb5f99c96ce3193b3bd8205e61e44d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13409)
 
   * 1b43a411d22af19f12f14a79c1094c8dc5cfeb42 UNKNOWN
   * e6aae42cf093d4f44b93b79a78bdc64538df8725 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13445)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot edited a comment on pull request #14916: [FLINK-21345][Table SQL / Planner] Fix BUG of Union All join Temporal…

[GitBox]

flinkbot edited a comment on pull request #14916:
URL: https://github.com/apache/flink/pull/14916#issuecomment-776614443


   
   ## CI report:
   
   * bf763e66a66853764f8bb39c07db8d86c98916aa Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13193)
 
   * 13b22ab740f81a326927caa4f344bbafab336a54 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (FLINK-21319) hadoop-mapreduce jars are not loaded into classpath when submiting flink on yarn jobs.

[Till Rohrmann (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-21319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286398#comment-17286398
 ] 

Till Rohrmann commented on FLINK-21319:
---

I don't know what kind of job you are running but Flink itself should not need 
this dependency. Maybe you can share a bit more details about your job with me.

> hadoop-mapreduce jars are not loaded into classpath when submiting flink on 
> yarn jobs.
> --
>
> Key: FLINK-21319
> URL: https://issues.apache.org/jira/browse/FLINK-21319
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / YARN
>Affects Versions: 1.12.1
>Reporter: Tang Yan
>Priority: Major
>
> My code is to query hive:
> {code:java}
> EnvironmentSettings settings = 
> EnvironmentSettings.newInstance().useBlinkPlanner().build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> String name            = "myhive";
> String defaultDatabase = "test"; 
> String hiveConfDir     = "/etc/hive/conf";
> HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir); 
> tableEnv.registerCatalog(name, hive);
> tableEnv.useDatabase(defaultDatabase);
> String testsql="select count(1) from mytable"; 
> tableEnv.executeSql(testsql); 
> {code}
> Env: Flink 1.12.1 + CDH6.3.0
> My submit command:
>  
> {code:java}
> export HADOOP_CLASSPATH=`hadoop classpath`
> /opt/flink-1.12.1/bin/flink run -m yarn-cluster -p 2 -c 
> com..flink.test.HiveConnTestJob /home/path/flinkTestCDH6-0.0.1-SNAPSHOT.jar
> {code}
>  
> Job ERROR:
>  
> {code:java}
> java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf
> {code}
>  
> If I do hadoop classpath on the server, I can see hadoop-mapreduce jars 
> folder is included as below, but when I check the flink job logs, it's not 
> included there.
>  
> {code:java}
> [root@my_server1 lib]# hadoop classpath
> /etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/GPLEXTRAS-6.3.0-1.gplextras6.3.0.p0.1279813/lib/hadoop/lib/COPYING.hadoop-lzo:/opt/cloudera/parcels/GPLEXTRAS-6.3.0-1.gplextras6.3.0.p0.1279813/lib/hadoop/lib/hadoop-lzo-0.4.15-cdh6.3.0.jar:/opt/cloudera/parcels/GPLEXTRAS-6.3.0-1.gplextras6.3.0.p0.1279813/lib/hadoop/lib/hadoop-lzo.jar:/opt/cloudera/parcels/GPLEXTRAS-6.3.0-1.gplextras6.3.0.p0.1279813/lib/hadoop/lib/native
> {code}
>  
> Flink job logs:
>  
> {code:java}
> 2021-02-08 05:26:42,590 INFO  
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] -  Classpath: 
> 

[GitHub] [flink] flinkbot edited a comment on pull request #14838: [FLINK-19503][state] Add StateChangelog API

[GitBox]

flinkbot edited a comment on pull request #14838:
URL: https://github.com/apache/flink/pull/14838#issuecomment-772060058


   
   ## CI report:
   
   * f560722ab6bbf452340228a2c88e025ecc797274 UNKNOWN
   * 42dd6bd7e4e64b3ff0ecf5156cae0f860226d629 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13420)
 
   * 000fa3e28def66c51ebf7b5975d8b6106988 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #14959: [FLINK-16947][Azure] Attempt to fix maven network issues

[GitBox]

flinkbot commented on pull request #14959:
URL: https://github.com/apache/flink/pull/14959#issuecomment-781217170


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 3e3889ee5cc45f7d8dd654b9c5296a80ab6e88e0 (Thu Feb 18 
09:42:44 UTC 2021)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [flink] syucream commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf

[GitBox]

syucream commented on pull request #14376:
URL: https://github.com/apache/flink/pull/14376#issuecomment-781216459


   Any progress?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (FLINK-16947) ArtifactResolutionException: Could not transfer artifact. Entry [...] has not been leased from this pool

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/FLINK-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-16947:
---
Labels: pull-request-available test-stability  (was: test-stability)

> ArtifactResolutionException: Could not transfer artifact.  Entry [...] has 
> not been leased from this pool
> -
>
> Key: FLINK-16947
> URL: https://issues.apache.org/jira/browse/FLINK-16947
> Project: Flink
>  Issue Type: Bug
>  Components: Build System / Azure Pipelines
>Reporter: Piotr Nowojski
>Assignee: Robert Metzger
>Priority: Critical
>  Labels: pull-request-available, test-stability
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6982=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> Build of flink-metrics-availability-test failed with:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (end-to-end-tests) 
> on project flink-metrics-availability-test: Unable to generate classpath: 
> org.apache.maven.artifact.resolver.ArtifactResolutionException: Could not 
> transfer artifact org.apache.maven.surefire:surefire-grouper:jar:2.22.1 
> from/to google-maven-central 
> (https://maven-central-eu.storage-download.googleapis.com/maven2/): Entry 
> [id:13][route:{s}->https://maven-central-eu.storage-download.googleapis.com:443][state:null]
>  has not been leased from this pool
> [ERROR] org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR] 
> [ERROR] from the specified remote repositories:
> [ERROR] google-maven-central 
> (https://maven-central-eu.storage-download.googleapis.com/maven2/, 
> releases=true, snapshots=false),
> [ERROR] apache.snapshots (https://repository.apache.org/snapshots, 
> releases=false, snapshots=true)
> [ERROR] Path to dependency:
> [ERROR] 1) dummy:dummy:jar:1.0
> [ERROR] 2) org.apache.maven.surefire:surefire-junit47:jar:2.22.1
> [ERROR] 3) org.apache.maven.surefire:common-junit48:jar:2.22.1
> [ERROR] 4) org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR] -> [Help 1]
> [ERROR] 
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR] 
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR] 
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :flink-metrics-availability-test
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [flink] rmetzger opened a new pull request #14959: [FLINK-16947][Azure] Attempt to fix maven network issues

[GitBox]

rmetzger opened a new pull request #14959:
URL: https://github.com/apache/flink/pull/14959


   This updates the base docker image to use a modified maven version with a 
fixed http wagon version.
   
   Diff on the Dockerfile:
   
   ```diff
   diff --git a/Dockerfile b/Dockerfile
   index dc8f1b5..68d080d 100644
   --- a/Dockerfile
   +++ b/Dockerfile
   @@ -37,6 +37,17 @@ RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
  && rm -f /tmp/apache-maven.tar.gz \
  && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
   
   +# put custom http-wagon. More details: 
https://issues.apache.org/jira/browse/FLINK-16947?focusedCommentId=17285028
   +RUN cd /usr/share/maven/lib/ \
   +  && rm wagon-http-*-shaded.jar \
   +  && curl -O 
https://jitpack.io/com/github/lhotari/maven-wagon/wagon-http/5ff79d284/wagon-http-5ff79d284-shaded.jar
   +
   +# add commons logging (needed for custom wagon)
   +RUN cd /tmp \
   +  && wget 
https://mirror.synyx.de/apache//commons/logging/binaries/commons-logging-1.2-bin.zip
 \
   +  && unzip commons-logging-1.2-bin.zip \
   +  && cp commons-logging-1.2/commons-logging-1.2.jar /usr/share/maven/lib/
   +
ENV MAVEN_HOME /usr/share/maven
   ```
   
   This has been committed in 
https://github.com/rmetzger/flink-ci/commit/98ad098128398c6b0621a7b17bea05fc0fbf1c07
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org