[GitHub] klion26 commented on a change in pull request #7412: [FLINK-10866][Runtime] 1. Explicitly enable qs server and proxy. 2. QS
klion26 commented on a change in pull request #7412: [FLINK-10866][Runtime] 1.
Explicitly enable qs server and proxy. 2. QS
URL: https://github.com/apache/flink/pull/7412#discussion_r245463212
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/QueryableStateOptions.java
##
@@ -106,6 +106,12 @@
.defaultValue(0)
.withDescription("Number of query Threads for queryable
state server. Uses the number of slots if set to 0.");
+ /** Enable Queryable state server and proxy. */
+ public static final ConfigOption
ENABLE_QUERYABLE_STATE_PROXY_SERVER =
+ key("query.enable")
Review comment:
how about changing the key to `query.server.xxx` as above, and the
description to "Option whether ..." as `CheckpointingOptions.ASYNC_SNAPSHOTS`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] klion26 commented on a change in pull request #7412: [FLINK-10866][Runtime] 1. Explicitly enable qs server and proxy. 2. QS
klion26 commented on a change in pull request #7412: [FLINK-10866][Runtime] 1. Explicitly enable qs server and proxy. 2. QS URL: https://github.com/apache/flink/pull/7412#discussion_r245463422 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServicesConfiguration.java ## @@ -107,7 +107,7 @@ public TaskManagerServicesConfiguration( this.localRecoveryStateRootDirectories = checkNotNull(localRecoveryStateRootDirectories); this.localRecoveryEnabled = checkNotNull(localRecoveryEnabled); this.networkConfig = checkNotNull(networkConfig); - this.queryableStateConfig = checkNotNull(queryableStateConfig); + this.queryableStateConfig = queryableStateConfig; Review comment: Should we add `@Nullable` to the parameter `queryableStateConfig`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua commented on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer
yanghua commented on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer URL: https://github.com/apache/flink/pull/7405#issuecomment-451621417 @pnowojski Can you help me to confirm that this state of migration is correct? After getting your confirmation, I will implement the migrations tests in a separate commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] NicoK commented on issue #7348: [FLINK-11207][core] update commons-compress to 1.18
NicoK commented on issue #7348: [FLINK-11207][core] update commons-compress to 1.18 URL: https://github.com/apache/flink/pull/7348#issuecomment-451603867 Done - except for `flink-shaded-hadoop/flink-shaded-hadoop2/src/main/resources/META-INF/NOTICE` which did not contain any reference to `commons-compress` in the first place. btw: is there an automated way to generate the correct contents of these files? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid concurrent usage of StateSnapshotTransformer
tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid
concurrent usage of StateSnapshotTransformer
URL: https://github.com/apache/flink/pull/7320#discussion_r245370394
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/StateSnapshotTransformer.java
##
@@ -183,4 +207,17 @@ public
MapStateSnapshotTransformer(StateSnapshotTransformer entryValueTransfo
Optional>
createForSerializedState();
}
+
+ abstract class StateSnapshotTransformFactoryWrapAdaptor
implements StateSnapshotTransformFactory {
Review comment:
Maybe `CollectionStateSnapshotTransformerFactory` would be a better name
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid concurrent usage of StateSnapshotTransformer
tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid
concurrent usage of StateSnapshotTransformer
URL: https://github.com/apache/flink/pull/7320#discussion_r245358031
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
##
@@ -229,7 +227,7 @@ public HeapKeyedStateBackend(
private StateTable tryRegisterStateTable(
TypeSerializer namespaceSerializer,
StateDescriptor stateDesc,
- @Nullable StateSnapshotTransformer
snapshotTransformer) throws StateMigrationException {
+ @Nonnull StateSnapshotTransformFactory
snapshotTransformer) throws StateMigrationException {
Review comment:
change parameter name
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid concurrent usage of StateSnapshotTransformer
tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid
concurrent usage of StateSnapshotTransformer
URL: https://github.com/apache/flink/pull/7320#discussion_r245370513
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/StateSnapshotTransformer.java
##
@@ -120,6 +122,17 @@ public
ListStateSnapshotTransformer(StateSnapshotTransformer entryValueTransf
}
}
+ class ListStateSnapshotTransformFactory extends
StateSnapshotTransformFactoryWrapAdaptor> {
Review comment:
Why do all these factory classes live within the `StateSnapshotTransformer`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-11088) Improve Kerberos Authentication using Keytab in YARN proxy user mode
[ https://issues.apache.org/jira/browse/FLINK-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rong Rong updated FLINK-11088: -- Issue Type: Sub-task (was: Improvement) Parent: FLINK-11271 > Improve Kerberos Authentication using Keytab in YARN proxy user mode > > > Key: FLINK-11088 > URL: https://issues.apache.org/jira/browse/FLINK-11088 > Project: Flink > Issue Type: Sub-task > Components: Security, YARN >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > Currently flink-yarn assumes keytab is shipped as application master > environment local resource on client side and will be distributed to all the > TMs. This does not work for YARN proxy user mode [1] since proxy user or > super user might not have access to actual users' keytab, but can request > delegation tokens on users' behalf. > Based on the type of security options for long-living YARN service[2], we > propose to have the keytab file path discovery configurable depending on the > launch mode of the YARN client. > Reference: > [1] > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html > [2] > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Securing_Long-lived_YARN_Services -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid concurrent usage of StateSnapshotTransformer
tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid
concurrent usage of StateSnapshotTransformer
URL: https://github.com/apache/flink/pull/7320#discussion_r245368875
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/CopyOnWriteStateTableSnapshot.java
##
@@ -147,7 +147,8 @@ public StateKeyGroupWriter getKeyGroupWriter() {
localKeySerializer.serialize(element.key, dov);
localStateSerializer.serialize(element.state, dov);
};
- StateSnapshotTransformer stateSnapshotTransformer =
owningStateTable.metaInfo.getSnapshotTransformer();
+ StateSnapshotTransformer stateSnapshotTransformer =
owningStateTable.metaInfo.
+
getStateSnapshotTransformFactory().createForDeserializedState().orElse(null);
Review comment:
The part below this line could be refactored to not check again if
`stateSnapshotTransformer != null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid concurrent usage of StateSnapshotTransformer
tillrohrmann commented on a change in pull request #7320: [FLINK-11171] Avoid
concurrent usage of StateSnapshotTransformer
URL: https://github.com/apache/flink/pull/7320#discussion_r245362372
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/RegisteredKeyValueStateBackendMetaInfo.java
##
@@ -145,13 +146,13 @@ private RegisteredKeyValueStateBackendMetaInfo(
return stateSerializerProvider.previousSchemaSerializer();
}
- @Nullable
- public StateSnapshotTransformer getSnapshotTransformer() {
- return snapshotTransformer;
+ @Nonnull
+ public StateSnapshotTransformFactory
getStateSnapshotTransformFactory() {
Review comment:
Should the factory be called `StateSnapshotTransformerFactory`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-11088) Improve Kerberos Authentication Keytab discovery on YARN
[ https://issues.apache.org/jira/browse/FLINK-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rong Rong updated FLINK-11088: -- Summary: Improve Kerberos Authentication Keytab discovery on YARN (was: Improve Kerberos Authentication using Keytab in YARN proxy user mode) > Improve Kerberos Authentication Keytab discovery on YARN > > > Key: FLINK-11088 > URL: https://issues.apache.org/jira/browse/FLINK-11088 > Project: Flink > Issue Type: Sub-task > Components: Security, YARN >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > > Currently flink-yarn assumes keytab is shipped as application master > environment local resource on client side and will be distributed to all the > TMs. This does not work for YARN proxy user mode [1] since proxy user or > super user might not have access to actual users' keytab, but can request > delegation tokens on users' behalf. > Based on the type of security options for long-living YARN service[2], we > propose to have the keytab file path discovery configurable depending on the > launch mode of the YARN client. > Reference: > [1] > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html > [2] > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Securing_Long-lived_YARN_Services -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11271) Improve Kerberos Credential Distribution
Rong Rong created FLINK-11271: - Summary: Improve Kerberos Credential Distribution Key: FLINK-11271 URL: https://issues.apache.org/jira/browse/FLINK-11271 Project: Flink Issue Type: Improvement Components: Security, YARN Reporter: Rong Rong Assignee: Rong Rong This is the master JIRA for the improvement listed in: https://docs.google.com/document/d/1rBLCpyQKg6Ld2P0DEgv4VIOMTwv4sitd7h7P5r202IE/edit#heading=h.y34f96ctioqk -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10848) Flink's Yarn ResourceManager can allocate too many excess containers
[ https://issues.apache.org/jira/browse/FLINK-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734322#comment-16734322 ] Shuyi Chen commented on FLINK-10848: Fixed in master: e26d90fc86b266978b4bac84fe02ca34b62983fe. I'll patch the change to 1.6 and 1.7 later. > Flink's Yarn ResourceManager can allocate too many excess containers > > > Key: FLINK-10848 > URL: https://issues.apache.org/jira/browse/FLINK-10848 > Project: Flink > Issue Type: Bug > Components: YARN >Affects Versions: 1.3.3, 1.4.2, 1.5.5, 1.6.2 >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently, both the YarnFlinkResourceManager and YarnResourceManager do not > call removeContainerRequest() on container allocation success. Because the > YARN AM-RM protocol is not a delta protocol (please see YARN-1902), > AMRMClient will keep all ContainerRequests that are added and send them to RM. > In production, we observe the following that verifies the theory: 16 > containers are allocated and used upon cluster startup; when a TM is killed, > 17 containers are allocated, 1 container is used, and 16 excess containers > are returned; when another TM is killed, 18 containers are allocated, 1 > container is used, and 17 excess containers are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] suez1224 commented on issue #7078: [FLINK-10848][YARN] properly remove YARN ContainerRequest upon container allocation success
suez1224 commented on issue #7078: [FLINK-10848][YARN] properly remove YARN ContainerRequest upon container allocation success URL: https://github.com/apache/flink/pull/7078#issuecomment-451497384 Thanks a lot for all the reviews. We've tested the branch in production for some time, merging into the master now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #7078: [FLINK-10848][YARN] properly remove YARN ContainerRequest upon container allocation success
asfgit closed pull request #7078: [FLINK-10848][YARN] properly remove YARN
ContainerRequest upon container allocation success
URL: https://github.com/apache/flink/pull/7078
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
index 3763f6592af..f1e6a3a767c 100644
--- a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
+++ b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java
@@ -165,6 +165,8 @@
YARN_CONFIGURATION.setInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 3600);
YARN_CONFIGURATION.setBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED, false);
YARN_CONFIGURATION.setInt(YarnConfiguration.NM_VCORES, 666); //
memory is overwritten in the MiniYARNCluster.
+
YARN_CONFIGURATION.set("yarn.scheduler.capacity.resource-calculator",
+
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator");
// so we have to change the number of cores for testing.
YARN_CONFIGURATION.setInt(YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS, 2);
// 20 seconds expiry (to ensure we properly heartbeat with YARN).
}
diff --git
a/flink-yarn/src/main/java/org/apache/flink/yarn/YarnFlinkResourceManager.java
b/flink-yarn/src/main/java/org/apache/flink/yarn/YarnFlinkResourceManager.java
index 8e686bbbe34..3327505e32d 100644
---
a/flink-yarn/src/main/java/org/apache/flink/yarn/YarnFlinkResourceManager.java
+++
b/flink-yarn/src/main/java/org/apache/flink/yarn/YarnFlinkResourceManager.java
@@ -438,6 +438,8 @@ private void containersAllocated(List
containers) {
numPendingContainerRequests = Math.max(0,
numPendingContainerRequests - 1);
LOG.info("Received new container: {} - Remaining
pending container requests: {}",
container.getId(), numPendingContainerRequests);
+ resourceManagerClient.removeContainerRequest(new
AMRMClient.ContainerRequest(
+ container.getResource(), null, null,
container.getPriority()));
// decide whether to return the container, or whether
to start a TaskManager
if (numRegistered + containersInLaunch.size() <
numRequired) {
diff --git
a/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
b/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
index 6ff5cd66487..6669f16fa40 100644
--- a/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
+++ b/flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
@@ -361,7 +361,8 @@ public void onContainersAllocated(List
containers) {
"Received new container: {} - Remaining
pending container requests: {}",
container.getId(),
numPendingContainerRequests);
-
+
resourceManagerClient.removeContainerRequest(new AMRMClient.ContainerRequest(
+ container.getResource(), null,
null, container.getPriority()));
if (numPendingContainerRequests > 0) {
numPendingContainerRequests--;
diff --git
a/flink-yarn/src/test/java/org/apache/flink/yarn/YarnFlinkResourceManagerTest.java
b/flink-yarn/src/test/java/org/apache/flink/yarn/YarnFlinkResourceManagerTest.java
index 10b2ce97d6f..d665df6bc7c 100644
---
a/flink-yarn/src/test/java/org/apache/flink/yarn/YarnFlinkResourceManagerTest.java
+++
b/flink-yarn/src/test/java/org/apache/flink/yarn/YarnFlinkResourceManagerTest.java
@@ -43,6 +43,8 @@
import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
import org.apache.hadoop.yarn.client.api.AMRMClient;
import org.apache.hadoop.yarn.client.api.NMClient;
import org.apache.hadoop.yarn.client.api.async.AMRMClientAsync;
@@ -69,8 +71,11 @@
import scala.concurrent.duration.FiniteDuration;
import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.any;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/**
@@
[GitHub] tillrohrmann commented on a change in pull request #7277: [FLINK-11022] Add LICENSE & NOTICE files for 1.6
tillrohrmann commented on a change in pull request #7277: [FLINK-11022] Add LICENSE & NOTICE files for 1.6 URL: https://github.com/apache/flink/pull/7277#discussion_r245327116 ## File path: flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE ## @@ -1,82 +1,36 @@ +flink-s3-fs-presto +Copyright 2014-2018 The Apache Software Foundation + This project includes software developed at The Apache Software Foundation (http://www.apache.org/). -- - -This project bundles the following dependencies under -the Apache Software License 2.0 - - - com.facebook.presto : presto-hive version 0.185 - - com.facebook.presto.hadoop : hadoop-apache2 version 2.7.3-1 - - com.amazonaws : aws-java-sdk-core version 1.11.165 - - com.amazonaws : aws-java-sdk-s3 version 1.11.165 - - com.amazonaws : aws-java-sdk-kms version 1.11.165 - - com.amazonaws : jmespath-java version 1.11.165 - - software.amazon.ion : ion-java version 1.0.2 - - io.airlift : stats version 0.148 - - io.airlift : log version 0.148 - - io.airlift : configuration version 0.148 - - io.airlift : slice version 0.31 - - io.airlift : units version 1.0 - - com.google.guava : guava version 21.0 - - com.google.code.findbugs : annotations version 2.0.3 - - org.weakref : jmxutils version 1.19 - - joda-time : joda-time version 2.5 - - commons-logging : commons-logging version 1.1.3 - - org.apache.httpcomponents : httpclient version 4.5.3 - - org.apache.httpcomponents : httpcore version 4.4.6 - - commons-codec : commons-codec version 1.10 - - com.fasterxml.jackson.core : jackson-core version 2.7.4 - - com.fasterxml.jackson.core : jackson-databind version 2.7.4 - - com.fasterxml.jackson.core : jackson-annotations version 2.7.4 - - com.fasterxml.jackson.dataformat : jackson-dataformat-cbor version 2.6.7 - -=== - Notice for HdrHistogram -=== - -This project bundles HdrHistogram (v. 2.1.9) under the BSD 2-Clause License - -Original source repository: https://github.com/HdrHistogram/HdrHistogram - -The code in this repository code was Written by Gil Tene, Michael Barker, -and Matt Warren, and released to the public domain, as explained at -http://creativecommons.org/publicdomain/zero/1.0/ - -For users of this code who wish to consume it under the "BSD" license -rather than under the public domain or CC0 contribution text mentioned -above, the code found under this directory is *also* provided under the -following license (commonly referred to as the BSD 2-Clause License). This -license does not detract from the above stated release of the code into -the public domain, and simply represents an additional license granted by -the Author. - -- -** Beginning of "BSD 2-Clause License" text. ** - - Copyright (c) 2012, 2013, 2014, 2015, 2016 Gil Tene - Copyright (c) 2014 Michael Barker - Copyright (c) 2014 Matt Warren - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are met: - - 1. Redistributions of source code must retain the above copyright notice, -this list of conditions and the following disclaimer. - - 2. Redistributions in binary form must reproduce the above copyright notice, -this list of conditions and the following disclaimer in the documentation -and/or other materials provided with the distribution. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE - LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF - THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file +This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + + com.amazonaws:aws-java-sdk-core:1.11.165 + com.amazonaws:aws-java-sdk-kms:1.11.165 + com.amazonaws:aws-java-sdk-s3:1.11.165 + com.amazonaws:jmespath-java:1.11.165 + com.facebook.presto.hadoop:hadoop-apache2:2.7.3-1 + com.facebook.presto:presto-hive:0.18 Review comment: wrong ordering This is an automated message from the Apache Git Service. To respond to the message, please log on
[GitHub] tillrohrmann commented on a change in pull request #7277: [FLINK-11022] Add LICENSE & NOTICE files for 1.6
tillrohrmann commented on a change in pull request #7277: [FLINK-11022] Add LICENSE & NOTICE files for 1.6 URL: https://github.com/apache/flink/pull/7277#discussion_r245337952 ## File path: NOTICE-binary ## @@ -0,0 +1,4297 @@ +// -- +// NOTICE file corresponding to the section 4d of The Apache License, +// Version 2.0, in this case for Apache Flink +// -- + +Apache Flink +Copyright 2006-2018 The Apache Software Foundation + +This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + +- log4j:log4j:1.2.17 +- org.apache.kafka:kafka-clients:jar:0.10.2.1 + +This project bundles the following dependencies under the MIT/X11 license. +See bundled license files for details. + +- org.slf4j:slf4j-log4j12:1.7.15 Review comment: should be `1.7.7` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-2597) Add a test for Avro-serialized Kafka messages
[ https://issues.apache.org/jira/browse/FLINK-2597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-2597: -- Labels: pull-request-available (was: ) > Add a test for Avro-serialized Kafka messages > -- > > Key: FLINK-2597 > URL: https://issues.apache.org/jira/browse/FLINK-2597 > Project: Flink > Issue Type: Bug > Components: Batch Connectors and Input/Output Formats, Kafka > Connector >Affects Versions: 0.10.0 >Reporter: Robert Metzger >Assignee: Vimal >Priority: Minor > Labels: pull-request-available > > A user has asked for serializing Avro messages from Kafka. > I think its a legitimate use-case that we should cover by a test case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] rmetzger closed pull request #2705: [FLINK-2597][FLINK-4050] Add wrappers for Kafka serializers, test for partitioner and documentation
rmetzger closed pull request #2705: [FLINK-2597][FLINK-4050] Add wrappers for
Kafka serializers, test for partitioner and documentation
URL: https://github.com/apache/flink/pull/2705
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index 9a360d47bfb..32f08d353fe 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -279,7 +279,7 @@ FlinkKafkaProducer010.writeToKafkaWithTimestamps(stream,
"my-topic", new SimpleS
-You can also define a custom Kafka producer configuration for the KafkaSink
with the constructor. Please refer to
+You can also define a custom Kafka producer configuration for the
FlinkKafkaProducer with the constructor. Please refer to
the [Apache Kafka documentation](https://kafka.apache.org/documentation.html)
for details on how to configure
Kafka Producers.
@@ -289,6 +289,11 @@ one producer instance can send data to multiple topics.
The interface of the serialization schema is called `KeyedSerializationSchema`.
+The last argument of the `FlinkKafkaProducer` allows passing a custom
partitioner, extending Flink's `KafkaPartitioner`
+class. The partitioner provides some Flink specific data, like the number of
parallel sources, and the source id.
+
+The producer supports Kafka's `Partitioner` interface as well. To use it, pass
`null` as the partitioner to the
+constructor and set the `partitioner.class` property, pointing to your
implementation.
**Note**: By default, the number of retries is set to "0". This means that the
producer fails immediately on errors,
including leader changes. The value is set to "0" by default to avoid
duplicate messages in the target topic.
@@ -330,3 +335,74 @@ config.setWriteTimestampToKafka(true);
{% endhighlight %}
+### Using Flink's Kafka connector with Kafka serializers
+
+Flink uses its own interfaces for specifying serializers for the Kafka
connector. However, we also provide a wrapper to use
+Kafka serializers with Flink.
+
+The wrappers are called `KafkaSerializerWrapper` and
`KafkaDeserializerWrapper`. The following example shows how to use Confluent's
+`KafkaAvroSerializer` with the schema registry.
+
+For using the Avro serializers in a Maven project, we first need to add
Confluent's repository and the serializer dependency.
+
+{% highlight xml %}
+
+
+
+
+confluent
+http://packages.confluent.io/maven
+
+
+
+
+io.confluent
+kafka-avro-serializer
+3.0.0
+test
+
+
+org.apache.kafka
+*
+
+
+log4j
+log4j
+
+
+org.slf4j
+slf4j-log4j12
+
+
+
+{% endhighlight %}
+
+
+**The serializer for the `FlinkKafkaProducer` is specified like this**:
+
+{% highlight java %}
+Map config = null;
+KeyedSerializationSchema> serSchema = new
KafkaSerializerWrapper<>(KafkaAvroSerializer.class, KafkaAvroSerializer.class,
config);
+FlinkKafkaProducer09> producer = new
FlinkKafkaProducer09<>("topic", serSchema, new Properties());
+{% endhighlight %}
+
+The first two arguments are the Kafka serializer classes for the keys and
values in the topic. The last argument `config` allows
+you to pass configuration values to the serializers. The `KafkaAvroSerializer`
in this example uses these configs for
+the schema registry URL.
+
+The wrapper is internally calling Kafka's `Serializer.serialize(topic,
message)` method, which expects the topic as an argument as well.
+Since Flink's serialize*() methods don't provide the topic, there is a special
configuration key to
+set the topic for the serializer through this wrapper. The configuration key
is stored in the `KafkaSerializerWrapper.SERIALIZER_TOPIC` constant.
+
+**The deserializer for the `FlinkKafkaConsumer` is specified like this**:
+
+{% highlight java %}
+Map config = null;
+KafkaDeserializerWrapper kvDeSer = new
KafkaDeserializerWrapper<>(KafkaAvroDeserializer.class,
KafkaAvroDeserializer.class, Void.class, GenericRecord.class, config);
+FlinkKafkaConsumer09> consumer = new
FlinkKafkaConsumer09<>("topic", kvDeSer, new Properties());
+{% endhighlight %}
+
+Similar to the serializer, the deserializer takes the two Kafka deserializers
for the key and values. The deserializer wrapper also needs to know the target
types
+the deserializer is creating.
+
+
diff --git a/flink-streaming-connectors/flink-connector-kafka-0.10/pom.xml
b/flink-streaming-connectors/flink-connector-kafka-0.10/pom.xml
index 8108afc1a22..644eeef6c13 100644
--- a/flink-streaming-connectors/flink-connector-kafka-0.10/pom.xml
+++ b/flink-streaming-connectors/flink-connector-kafka-0.10/pom.xml
@@ -40,6 +40,13 @@ under the License.
[GitHub] hequn8128 commented on issue #7271: [FLINK-11124][table] Add private[flink] to TemporalTableFunction.create() method
hequn8128 commented on issue #7271: [FLINK-11124][table] Add private[flink] to TemporalTableFunction.create() method URL: https://github.com/apache/flink/pull/7271#issuecomment-451475041 @twalthr Thanks a lot for the review and merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-11124) Add private[flink] to TemporalTableFunction.create()
[
https://issues.apache.org/jira/browse/FLINK-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-11124.
Resolution: Fixed
Fix Version/s: 1.8.0
Fixed in master: b82b24d720bdebd58ba507af98e68ab027790a4f
> Add private[flink] to TemporalTableFunction.create()
>
>
> Key: FLINK-11124
> URL: https://issues.apache.org/jira/browse/FLINK-11124
> Project: Flink
> Issue Type: Improvement
> Components: Table API SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{TemporalTableFunction}} is an user-oriented class. I think it would be
> better to add {{private[flink]}} to the {{TemporalTableFunction.create()}}
> method in order to make it invisible to users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #7271: [FLINK-11124][table] Add private[flink] to TemporalTableFunction.create() method
asfgit closed pull request #7271: [FLINK-11124][table] Add private[flink] to
TemporalTableFunction.create() method
URL: https://github.com/apache/flink/pull/7271
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/TemporalTableFunction.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/TemporalTableFunction.scala
index bbb9911c63f..bf668480a0a 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/TemporalTableFunction.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/TemporalTableFunction.scala
@@ -65,7 +65,7 @@ class TemporalTableFunction private(
}
object TemporalTableFunction {
- def create(
+ private[flink] def create(
table: Table,
timeAttribute: Expression,
primaryKey: String): TemporalTableFunction = {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-11001) Window rowtime attribute can't be renamed in Java
[
https://issues.apache.org/jira/browse/FLINK-11001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734222#comment-16734222
]
Timo Walther commented on FLINK-11001:
--
Updated exception message in master: 038dc0008f7b27c787e3ccba2204f3a2851dff7d
> Window rowtime attribute can't be renamed in Java
> -
>
> Key: FLINK-11001
> URL: https://issues.apache.org/jira/browse/FLINK-11001
> Project: Flink
> Issue Type: Bug
> Environment:
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> Currently, we can rename window rowtime attribute like this in Scala:
> {code:java}
> table
> .window(Tumble over 2.millis on 'rowtime as 'w)
> .groupBy('w)
> .select('w.rowtime as 'rowtime, 'int.count as 'int)
> {code}
> However, an exception will be thrown if we use java(by changing the
> Expressions to String):
> {code:java}
> table
> .window(Tumble over 2.millis on 'rowtime as 'w)
> .groupBy('w)
> .select("w.rowtime as rowtime, int.count as int")
> {code}
> The Exception is:
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 11: `,' expected but `a' found
> w.rowtime as rowtime, int.count as int
> {code}
>
> To solve the problem, we can add rename support in {{ExpressionParser}}.
> However, this may conflict with the design of source which use as before
> rowtime:
> {code:java}
> stream.toTable(
> tEnv,
> ExpressionParser.parseExpressionList("(b as b).rowtime, c as c, a as
> a"): _*)
> {code}
> Personally, I think we should keep the two consistent, so the final api would
> be:
> {code:java}
> // window case
> .select("w.rowtime as rowtime, int.count as int")
> // source case
> stream.toTable(
> tEnv,
> ExpressionParser.parseExpressionList("b.rowtime as b, c as c, a as a"):
> _*)
> {code}
> Any suggestions would be greatly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] twalthr commented on a change in pull request #7271: [FLINK-11124][table] Add private[flink] to TemporalTableFunction.create() method
twalthr commented on a change in pull request #7271: [FLINK-11124][table] Add
private[flink] to TemporalTableFunction.create() method
URL: https://github.com/apache/flink/pull/7271#discussion_r245321721
##
File path:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/TemporalTableFunction.scala
##
@@ -65,7 +65,7 @@ class TemporalTableFunction private(
}
object TemporalTableFunction {
- def create(
+ private[flink] def create(
Review comment:
@hequn8128 we could also remove this method entirely. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] twalthr commented on a change in pull request #7271: [FLINK-11124][table] Add private[flink] to TemporalTableFunction.create() method
twalthr commented on a change in pull request #7271: [FLINK-11124][table] Add
private[flink] to TemporalTableFunction.create() method
URL: https://github.com/apache/flink/pull/7271#discussion_r245321721
##
File path:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/functions/TemporalTableFunction.scala
##
@@ -65,7 +65,7 @@ class TemporalTableFunction private(
}
object TemporalTableFunction {
- def create(
+ private[flink] def create(
Review comment:
@hequn8128 we could also remove this method entirely. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] twalthr closed pull request #7289: [FLINK-11001][table] Fix window rowtime attribute can't be renamed bug in Java
twalthr closed pull request #7289: [FLINK-11001][table] Fix window rowtime
attribute can't be renamed bug in Java
URL: https://github.com/apache/flink/pull/7289
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md
index f308aca6824..e44df24ad5e 100644
--- a/docs/dev/table/tableApi.md
+++ b/docs/dev/table/tableApi.md
@@ -1728,7 +1728,7 @@ This is the EBNF grammar for expressions:
expressionList = expression , { "," , expression } ;
-expression = timeIndicator | overConstant | alias ;
+expression = overConstant | alias ;
alias = logic | ( logic , "as" , fieldReference ) | ( logic , "as" , "(" ,
fieldReference , { "," , fieldReference } , ")" ) ;
@@ -1744,7 +1744,7 @@ unary = [ "!" | "-" | "+" ] , composite ;
composite = over | suffixed | nullLiteral | prefixed | atom ;
-suffixed = interval | suffixAs | suffixCast | suffixIf | suffixDistinct |
suffixFunctionCall ;
+suffixed = interval | suffixAs | suffixCast | suffixIf | suffixDistinct |
suffixFunctionCall | timeIndicator ;
prefixed = prefixAs | prefixCast | prefixIf | prefixDistinct |
prefixFunctionCall ;
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/StreamTableEnvironment.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/StreamTableEnvironment.scala
index 8c6a1e0a04b..4fa501cde4d 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/StreamTableEnvironment.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/StreamTableEnvironment.scala
@@ -669,12 +669,15 @@ abstract class StreamTableEnvironment(
case (RowtimeAttribute(UnresolvedFieldReference(name)), idx) =>
extractRowtime(idx, name, None)
- case (RowtimeAttribute(Alias(UnresolvedFieldReference(origName), name,
_)), idx) =>
+ case (Alias(RowtimeAttribute(UnresolvedFieldReference(origName)), name,
_), idx) =>
extractRowtime(idx, name, Some(origName))
case (ProctimeAttribute(UnresolvedFieldReference(name)), idx) =>
extractProctime(idx, name)
+ case (Alias(ProctimeAttribute(UnresolvedFieldReference(_)), name, _),
idx) =>
+extractProctime(idx, name)
+
case (UnresolvedFieldReference(name), _) => fieldNames = name ::
fieldNames
case (Alias(UnresolvedFieldReference(_), name, _), _) => fieldNames =
name :: fieldNames
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/TableEnvironment.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/TableEnvironment.scala
index e28a471681d..ba789638ca3 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/TableEnvironment.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/TableEnvironment.scala
@@ -1089,7 +1089,7 @@ abstract class TableEnvironment(val config: TableConfig) {
} else {
referenceByName(origName, t).map((_, name))
}
- case (_: TimeAttribute, _) =>
+ case (_: TimeAttribute, _) | (Alias(_: TimeAttribute, _, _), _) =>
None
case _ => throw new TableException(
"Field reference expression or alias on field expression
expected.")
@@ -1101,7 +1101,7 @@ abstract class TableEnvironment(val config: TableConfig) {
referenceByName(name, p).map((_, name))
case Alias(UnresolvedFieldReference(origName), name: String, _) =>
referenceByName(origName, p).map((_, name))
- case _: TimeAttribute =>
+ case _: TimeAttribute | Alias(_: TimeAttribute, _, _) =>
None
case _ => throw new TableException(
"Field reference expression or alias on field expression
expected.")
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
index 7fd9309b5db..d5d64b48d69 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
@@ -355,7 +355,9 @@ object ExpressionParser extends JavaTokenParsers with
PackratParsers {
// expression with distinct suffix modifier
suffixDistinct |
// function call must always be at the end
-suffixFunctionCall | suffixFunctionCallOneArg
+suffixFunctionCall | suffixFunctionCallOneArg |
+// rowtime or proctime
+timeIndicator
// prefix operators
@@ -525,15 +527,13 @@ object ExpressionParser
[GitHub] twalthr commented on issue #7289: [FLINK-11001][table] Fix window rowtime attribute can't be renamed bug in Java
twalthr commented on issue #7289: [FLINK-11001][table] Fix window rowtime attribute can't be renamed bug in Java URL: https://github.com/apache/flink/pull/7289#issuecomment-451459198 @hequn8128 thanks for mentioning the `StreamTableEnvironmentValidationTest.testInvalidAliasWithProctimeAttribute()` test. The test is still correct. It should not be possible to alias a field that does not exist. However, after `.proctime` the field does somehow exist so aliasing is fine in this case. The only thing I will fix in a hotfix is the exception message, because it has not been updated. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-11173) Proctime attribute validation throws an incorrect exception message
[
https://issues.apache.org/jira/browse/FLINK-11173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-11173.
Resolution: Fixed
Fix Version/s: 1.8.0
1.7.2
1.6.4
Fixed in master: c09dc96a460a4947a348639c10d52ed56360fb96
Fixed in 1.7: e2dba59e79cbef6bbf80309beb097310c2fb
Fixed in 1.6: 46820483140946301cc4f13b353a94081b9b34d9
> Proctime attribute validation throws an incorrect exception message
> ---
>
> Key: FLINK-11173
> URL: https://issues.apache.org/jira/browse/FLINK-11173
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: vinoyang
>Assignee: TANG Wen-hui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.4, 1.7.2, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In {{TableSourceUtil#validateTableSource, when uses
> DefinedProctimeAttribute}} (proctime), it throws an incorrect exception
> message :
>
> {code:java}
> tableSource match {
> case p: DefinedProctimeAttribute if p.getProctimeAttribute != null =>
> val proctimeAttribute = p.getProctimeAttribute
> val proctimeIdx = schema.getFieldNames.indexOf(proctimeAttribute)
> // ensure that field exists
> if (proctimeIdx < 0) {
> throw new ValidationException(s"Found a RowtimeAttributeDescriptor for
> field " +
> s"'$proctimeAttribute' but field '$proctimeAttribute' does not exist
> in table.")
> }
> // ensure that field is of type TIMESTAMP
> if (schema.getFieldTypes()(proctimeIdx) != Types.SQL_TIMESTAMP) {
> throw new ValidationException(s"Found a RowtimeAttributeDescriptor for
> field " +
> s"'$proctimeAttribute' but field '$proctimeAttribute' is not of type
> TIMESTAMP.")
> }
> case _ => // nothing to validate
> }
> {code}
> It still contains the keyword "RowtimeAttributeDescriptor".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #7374: [FLINK-11173][table] Fix the exception message of proctime attribute validation in TableSourceUtil#validateTableSource
asfgit closed pull request #7374: [FLINK-11173][table] Fix the exception
message of proctime attribute validation in TableSourceUtil#validateTableSource
URL: https://github.com/apache/flink/pull/7374
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSourceUtil.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSourceUtil.scala
index 15d68900d21..3a0ab503de4 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSourceUtil.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSourceUtil.scala
@@ -138,12 +138,12 @@ object TableSourceUtil {
val proctimeIdx = schema.getFieldNames.indexOf(proctimeAttribute)
// ensure that field exists
if (proctimeIdx < 0) {
- throw new ValidationException(s"Found a RowtimeAttributeDescriptor
for field " +
+ throw new ValidationException(s"Found a processing time attribute:
field " +
s"'$proctimeAttribute' but field '$proctimeAttribute' does not
exist in table.")
}
// ensure that field is of type TIMESTAMP
if (schema.getFieldTypes()(proctimeIdx) != Types.SQL_TIMESTAMP) {
- throw new ValidationException(s"Found a RowtimeAttributeDescriptor
for field " +
+ throw new ValidationException(s"Found a processing time attribute:
field " +
s"'$proctimeAttribute' but field '$proctimeAttribute' is not of
type TIMESTAMP.")
}
case _ => // nothing to validate
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (FLINK-11227) The DescriptorProperties contains some bounds checking errors
[
https://issues.apache.org/jira/browse/FLINK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-11227.
Resolution: Fixed
Fix Version/s: (was: 1.6.4)
Fixed in master: 1c3c663ba7d55fcd33af1502ca426274d51d2ba1
Fixed in 1.7: 8ee0465761d58eb7fd96846c09614410d644ed65
> The DescriptorProperties contains some bounds checking errors
> -
>
> Key: FLINK-11227
> URL: https://issues.apache.org/jira/browse/FLINK-11227
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.7.1
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.2, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In {{DescriptorProperties}}, both the {{validateFixedIndexedProperties()}}
> and {{validateArray()}} use wrong upperbounds for validation, which leads to
> the last element not being validated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-11227) The DescriptorProperties contains some bounds checking errors
[
https://issues.apache.org/jira/browse/FLINK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther updated FLINK-11227:
-
Affects Version/s: (was: 1.6.3)
> The DescriptorProperties contains some bounds checking errors
> -
>
> Key: FLINK-11227
> URL: https://issues.apache.org/jira/browse/FLINK-11227
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Affects Versions: 1.7.1
>Reporter: Xingcan Cui
>Assignee: Xingcan Cui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.4, 1.7.2, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In {{DescriptorProperties}}, both the {{validateFixedIndexedProperties()}}
> and {{validateArray()}} use wrong upperbounds for validation, which leads to
> the last element not being validated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #7373: [FLINK-11227] [table] The DescriptorProperties contains some bounds checking errors
asfgit closed pull request #7373: [FLINK-11227] [table] The
DescriptorProperties contains some bounds checking errors
URL: https://github.com/apache/flink/pull/7373
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
index c4769814728..76289128b5b 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
@@ -938,7 +938,7 @@ public void validateFixedIndexedProperties(String key,
boolean allowEmpty, Map> subKey :
subKeyValidation.entrySet()) {
final String fullKey = key + '.' + i + '.' +
subKey.getKey();
if (properties.containsKey(fullKey)) {
@@ -1134,7 +1134,7 @@ public void validateArray(String key, Consumer
elementValidation, int mi
}
// validate array elements
- for (int i = 0; i < maxIndex; i++) {
+ for (int i = 0; i <= maxIndex; i++) {
final String fullKey = key + '.' + i;
if (properties.containsKey(fullKey)) {
// run validation logic
diff --git
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/descriptors/DescriptorPropertiesTest.scala
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/descriptors/DescriptorPropertiesTest.scala
index b2a8ec9f567..fe7c75df6cf 100644
---
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/descriptors/DescriptorPropertiesTest.scala
+++
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/descriptors/DescriptorPropertiesTest.scala
@@ -20,6 +20,7 @@ package org.apache.flink.table.descriptors
import java.util
import java.util.Collections
+import java.util.function.Consumer
import org.apache.flink.table.api.ValidationException
import org.apache.flink.table.util.JavaScalaConversionUtil.toJava
@@ -32,6 +33,9 @@ import org.junit.Test
class DescriptorPropertiesTest {
private val ARRAY_KEY = "my-array"
+ private val FIXED_INDEXED_PROPERTY_KEY = "my-fixed-indexed-property"
+ private val PROPERTY_1_KEY = "property-1"
+ private val PROPERTY_2_KEY = "property-2"
@Test
def testEquals(): Unit = {
@@ -97,8 +101,8 @@ class DescriptorPropertiesTest {
def testArrayInvalidValues(): Unit = {
val properties = new DescriptorProperties()
properties.putString(s"$ARRAY_KEY.0", "12")
-properties.putString(s"$ARRAY_KEY.1", "INVALID")
-properties.putString(s"$ARRAY_KEY.2", "66")
+properties.putString(s"$ARRAY_KEY.1", "66")
+properties.putString(s"$ARRAY_KEY.2", "INVALID")
testArrayValidation(properties, 1, Integer.MAX_VALUE)
}
@@ -118,6 +122,19 @@ class DescriptorPropertiesTest {
testArrayValidation(properties, 1, Integer.MAX_VALUE)
}
+ @Test(expected = classOf[ValidationException])
+ def testInvalidFixedIndexedProperties(): Unit = {
+val property = new DescriptorProperties()
+val list = new util.ArrayList[util.List[String]]()
+list.add(util.Arrays.asList("1", "string"))
+list.add(util.Arrays.asList("INVALID", "string"))
+property.putIndexedFixedProperties(
+ FIXED_INDEXED_PROPERTY_KEY,
+ util.Arrays.asList(PROPERTY_1_KEY, PROPERTY_2_KEY),
+ list)
+testFixedIndexedPropertiesValidation(property)
+ }
+
@Test
def testRemoveKeys(): Unit = {
val properties = new DescriptorProperties()
@@ -155,7 +172,7 @@ class DescriptorPropertiesTest {
minLength: Int,
maxLength: Int)
: Unit = {
-val validator: (String) => Unit = (key: String) => {
+val validator: String => Unit = (key: String) => {
properties.validateInt(key, false)
}
@@ -165,4 +182,26 @@ class DescriptorPropertiesTest {
minLength,
maxLength)
}
+
+ private def testFixedIndexedPropertiesValidation(properties:
DescriptorProperties): Unit = {
+
+val validatorMap = new util.HashMap[String, Consumer[String]]()
+
+// PROPERTY_1 should be Int
+val validator1: String => Unit = (key: String) => {
+ properties.validateInt(key, false)
+}
+validatorMap.put(PROPERTY_1_KEY, toJava(validator1))
+// PROPERTY_2 should be String
+val validator2: String => Unit = (key: String) => {
+ properties.validateString(key, false)
+}
+validatorMap.put(PROPERTY_2_KEY, toJava(validator2))
+
+properties.validateFixedIndexedProperties(
+
[jira] [Updated] (FLINK-11268) Update deploy_staging_jars.sh to deploy multiple flink-shaded-hadoop2 versions
[
https://issues.apache.org/jira/browse/FLINK-11268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-11268:
---
Labels: pull-request-available (was: )
> Update deploy_staging_jars.sh to deploy multiple flink-shaded-hadoop2 versions
> --
>
> Key: FLINK-11268
> URL: https://issues.apache.org/jira/browse/FLINK-11268
> Project: Flink
> Issue Type: Sub-task
> Components: Release System
>Affects Versions: 1.8.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> We will have to make a few changes to the {{fink-shaded-hadoop2}} module to
> support deployment of multiple versions. Currently neither the module name
> nor version contains the contained hadoop version; as such maven cannot
> differentiate between 2 versions of this artifact that were built with
> different a {{hadoop.version}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol opened a new pull request #7417: [FLINK-11268][elease] Deploy multiple flink-shaded-hadoop2 artifacts
zentol opened a new pull request #7417: [FLINK-11268][elease] Deploy multiple
flink-shaded-hadoop2 artifacts
URL: https://github.com/apache/flink/pull/7417
## What is the purpose of the change
This PR modifies the release scripts to deploy multiple
`flink-shaded-hadoop2` artifacts, one for each supported hadoop version.
With #7416 we only release hadoop-free Flink. Effectively this means that
users have to assemble hadoop-specific distributions themselves (if they need
flink-shaded-hadoop2). For convenience we should deploy various
`flink-shaded-hadoop2` artifacts that can be copied into /lib, one for each
hadoop version.
The current `flink-shaded-hadoop2` modules however do not allow deployment
of multiple artifacts for different hadoop versions, as the hadoop.version
property is not used in neither the artifactId noror version. Attempting to
deploy these modules would lead to them overriding each other, and a random
version winning out.
I have opted for including the hadoop.version property in the version of the
artifact. We _could_ include it in the artifactId, but we'd either end up with
periods in the name (e.g. `flink-shaded-hadoop2-2.4.8`) which is unusual, or
require another "version-like" property to be set that doesn't contain periods.
The downside is that `flink-shaded-hadoop2` modules now have to set
`1.8-SNAPSHOT-${hadoop.version}`, that is they have to
explicitly refer to the SNAPSHOT version. AFAIK there's no way to refer to the
parents version instead (`project.version` refers to the current module).
## Brief change log
* enable deployment for flink-shaded-hadoop2-uber
* add hadoop version to flink-shaded-hadoop2(-uber) version
* modify `deploy_staging_jars.sh` to release multiple versions of hadoop
artifacts
## Verifying this change
The changes to the release scripts can be verified by running the scrip and
checking the uploaded artifacts. For convenience I'd remove the calls that
deploy Flink itself.
The pom changes are covered by existing E2E tests and IT cases (the yarn IT
cases use flink-dist and require flink-shaded-hadoop to be in /lib).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (FLINK-11119) Incorrect Scala example for Table Function
[
https://issues.apache.org/jira/browse/FLINK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-9.
Resolution: Fixed
Fix Version/s: 1.8.0
1.7.2
1.6.4
Fixed in master: d3f0075b041c8fd681a776b3278d12c4483a48ac
Fixed in 1.7: f9c587475abb0c6aeb646b068602a1d7cb2e5cf3
Fixed in 1.6: 3b8126531b8dfa4f6ce592afbd61cd6e06f0f8ed
> Incorrect Scala example for Table Function
> --
>
> Key: FLINK-9
> URL: https://issues.apache.org/jira/browse/FLINK-9
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.6.2, 1.7.0
>Reporter: Denys Fakhritdinov
>Assignee: TANG Wen-hui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.4, 1.7.2, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Issue in Scala example in documentation:
> [https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/udfs.html#table-functions]
> Currently it is:
> {code:java}
> tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN
> TABLE(split(a)) as T(word, length) ON TRUE")
> {code}
> Should be (like in Java version):
> {code:java}
> tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL
> TABLE(split(a)) as T(word, length) ON TRUE"){code}
> {color:#FF}*LATERAL*{color} is missed in Scala version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #7379: [FLINK-11119][docs] Correct Scala example for Table Function in User-…
asfgit closed pull request #7379: [FLINK-9][docs] Correct Scala example for
Table Function in User-…
URL: https://github.com/apache/flink/pull/7379
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/udfs.md b/docs/dev/table/udfs.md
index 44b8d431c8d..d93b5c8c3f7 100644
--- a/docs/dev/table/udfs.md
+++ b/docs/dev/table/udfs.md
@@ -190,7 +190,7 @@ tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT
JOIN LATERAL TABLE(s
class Split(separator: String) extends TableFunction[(String, Int)] {
def eval(str: String): Unit = {
// use collect(...) to emit a row.
-str.split(separator).foreach(x -> collect((x, x.length))
+str.split(separator).foreach(x => collect((x, x.length)))
}
}
@@ -210,7 +210,7 @@ tableEnv.registerFunction("split", new Split("#"))
// CROSS JOIN a table function (equivalent to "join" in Table API)
tableEnv.sqlQuery("SELECT a, word, length FROM MyTable, LATERAL
TABLE(split(a)) as T(word, length)")
// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API)
-tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN
TABLE(split(a)) as T(word, length) ON TRUE")
+tableEnv.sqlQuery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL
TABLE(split(a)) as T(word, length) ON TRUE")
{% endhighlight %}
**IMPORTANT:** Do not implement TableFunction as a Scala object. Scala object
is a singleton and will cause concurrency issues.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-11267) Update create_binary_release.sh to not create hadoop-specific releases
[ https://issues.apache.org/jira/browse/FLINK-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11267: --- Labels: pull-request-available (was: ) > Update create_binary_release.sh to not create hadoop-specific releases > -- > > Key: FLINK-11267 > URL: https://issues.apache.org/jira/browse/FLINK-11267 > Project: Flink > Issue Type: Sub-task > Components: Release System >Affects Versions: 1.8.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zentol opened a new pull request #7416: [FLINK-11267][release] Only create hadoop-free binary by default
zentol opened a new pull request #7416: [FLINK-11267][release] Only create hadoop-free binary by default URL: https://github.com/apache/flink/pull/7416 ## What is the purpose of the change With this PR the `create_binary_release` script only creates hadoop-free Flink by default. This significantly reduces the size of our release and the time it takes to assemble it. The script can still be used to create hadoop/scala specific distributions if explicitly told to do so. ## Verifying this change The change itself is a trivial change without test coverage. However this change implies that users have to copy flink-shaded-hadoop2 themselves into /lib if desired. This action is equivalent to the assembly of an hadoop-specific flink-dist, which really only copies flink-shaded-hadoop2 into /lib during the build, and thus is covered by existing tests. (hadoop free yarn+kerberos on docker; Yarn IT cases) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] leesf opened a new pull request #7415: [hotfix][docs] fix typo in addShutdownHookThread Javadoc
leesf opened a new pull request #7415: [hotfix][docs] fix typo in addShutdownHookThread Javadoc URL: https://github.com/apache/flink/pull/7415 ## What is the purpose of the change Fix typo in ShutdownHookUtil#addShutdownHookThread Javadoc ## Brief change log Fix typo in ShutdownHookUtil#addShutdownHookThread Javadoc ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dawidwys closed pull request #7380: [hotfix][docs] Fix incorrect example in cep doc
dawidwys closed pull request #7380: [hotfix][docs] Fix incorrect example in cep doc URL: https://github.com/apache/flink/pull/7380 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/dev/libs/cep.md b/docs/dev/libs/cep.md index ce9a9c3d72b..8c89974b6b0 100644 --- a/docs/dev/libs/cep.md +++ b/docs/dev/libs/cep.md @@ -1324,7 +1324,7 @@ For example, for a given pattern `b+ c` and a data stream `b1 b2 b3 c`, the diff Have a look also at another example to better see the difference between NO_SKIP and SKIP_TO_FIRST: -Pattern: `(a | c) (b | c) c+.greedy d` and sequence: `a b c1 c2 c3 d` Then the results will be: +Pattern: `(a | b | c) (b | c) c+.greedy d` and sequence: `a b c1 c2 c3 d` Then the results will be: @@ -1339,12 +1339,11 @@ Pattern: `(a | c) (b | c) c+.greedy d` and sequence: `a b c1 c2 c3 d` Then the r a b c1 c2 c3 d b c1 c2 c3 d c1 c2 c3 d -c2 c3 d After found matching a b c1 c2 c3 d, the match process will not discard any result. -SKIP_TO_FIRST[b*] +SKIP_TO_FIRST[c*] a b c1 c2 c3 d c1 c2 c3 d @@ -1373,7 +1372,7 @@ Pattern: `a b+` and sequence: `a b1 b2 b3` Then the results will be: After found matching a b1, the match process will not discard any result. -SKIP_TO_NEXT[b*] +SKIP_TO_NEXT a b1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-11270) Do not include hadoop in flink-dist by default
Chesnay Schepler created FLINK-11270:
Summary: Do not include hadoop in flink-dist by default
Key: FLINK-11270
URL: https://issues.apache.org/jira/browse/FLINK-11270
Project: Flink
Issue Type: Sub-task
Components: Build System
Affects Versions: 1.8.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.8.0
In order to build a hadoop-free Flink it is currently necessary to activate the
{{without-hadoop}} profile.
We should revert this so that flink-dist is hadoop-free by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-11026) Rework creation of sql-client connector/format jars
[
https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-11026.
Resolution: Fixed
Release Note:
The naming scheme for kafka/elasticsearch6 sql-jars has been changed.
In maven terms, they no longer have the "sql-jar" qualifier and the artifactId
is now prefixed with "flink-sql" instead of "flink", e.g.,
"flink-sql-connector-kafka...".
> Rework creation of sql-client connector/format jars
> ---
>
> Key: FLINK-11026
> URL: https://issues.apache.org/jira/browse/FLINK-11026
> Project: Flink
> Issue Type: Improvement
> Components: Build System, SQL Client
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For the SQL client we currently have a separate {{sql-jars}} profile in
> various connectors/formats that create an additional fat jar with a separate
> classifier.
> One of the core maven mantras is "One artifact per module.", and we see the
> importance of this mantra as our current packaging strategy makes it
> impossible to provide different NOTICE files for the created jars (regular
> and sql-jar).
> Currently we would be forced to provide the same file for both jars, which
> will cause problems for any downstream users that wants to handle NOTICE
> files properly. We would cause the same issue we had with netty, which
> categorically claims to be bundling dependencies although it doesn't, forcing
> us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that
> depend on the original module.
> I will use {{flink-connector-elasticsearch6}} as an example, which packages
> both a regular jar without any included dependencies, and a sql jar bundling
> everything.
> * create a separate
> {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}}
> module
> * this module depends on {{flink-connector-elasticsearch6}}, and bundles all
> dependencies
> * move the current shading logic for the sql jar out of the {{sql-jars}}
> profile{{}}
> * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the
> creation of these jars
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-11026) Rework creation of sql-client connector/format jars
[
https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reopened FLINK-11026:
--
> Rework creation of sql-client connector/format jars
> ---
>
> Key: FLINK-11026
> URL: https://issues.apache.org/jira/browse/FLINK-11026
> Project: Flink
> Issue Type: Improvement
> Components: Build System, SQL Client
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For the SQL client we currently have a separate {{sql-jars}} profile in
> various connectors/formats that create an additional fat jar with a separate
> classifier.
> One of the core maven mantras is "One artifact per module.", and we see the
> importance of this mantra as our current packaging strategy makes it
> impossible to provide different NOTICE files for the created jars (regular
> and sql-jar).
> Currently we would be forced to provide the same file for both jars, which
> will cause problems for any downstream users that wants to handle NOTICE
> files properly. We would cause the same issue we had with netty, which
> categorically claims to be bundling dependencies although it doesn't, forcing
> us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that
> depend on the original module.
> I will use {{flink-connector-elasticsearch6}} as an example, which packages
> both a regular jar without any included dependencies, and a sql jar bundling
> everything.
> * create a separate
> {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}}
> module
> * this module depends on {{flink-connector-elasticsearch6}}, and bundles all
> dependencies
> * move the current shading logic for the sql jar out of the {{sql-jars}}
> profile{{}}
> * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the
> creation of these jars
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #7388: [hotfix][javadocs][table] Fix typos in JavaDocs for Table API
asfgit closed pull request #7388: [hotfix][javadocs][table] Fix typos in
JavaDocs for Table API
URL: https://github.com/apache/flink/pull/7388
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptor.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptor.java
index a1d3e783cdc..ed62b46b36b 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptor.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptor.java
@@ -68,7 +68,7 @@ protected final boolean isFormatNeeded() {
/**
* Converts this descriptor into a set of connector properties. Usually
prefixed with
-* {@link FormatDescriptorValidator#FORMAT}.
+* {@link ConnectorDescriptorValidator#CONNECTOR}.
*/
protected abstract Map toConnectorProperties();
}
diff --git
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptorValidator.java
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptorValidator.java
index 88cf34905f4..cf9860ae69e 100644
---
a/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptorValidator.java
+++
b/flink-libraries/flink-table-common/src/main/java/org/apache/flink/table/descriptors/ConnectorDescriptorValidator.java
@@ -26,6 +26,11 @@
@Internal
public abstract class ConnectorDescriptorValidator implements
DescriptorValidator {
+ /**
+* Prefix for connector-related properties.
+*/
+ public static final String CONNECTOR = "connector";
+
/**
* Key for describing the type of the connector. Usually used for
factory discovery.
*/
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
index 45414ee3ff6..ce57070ac98 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
@@ -270,8 +270,8 @@ class ExternalCatalogTableBuilder(private val
connectorDescriptor: ConnectorDesc
* Explicitly declares this external table for supporting only batch
environments.
*/
def supportsBatch(): ExternalCatalogTableBuilder = {
-isBatch = false
-isStreaming = true
+isBatch = true
+isStreaming = false
this
}
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/factories/BatchTableSinkFactory.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/factories/BatchTableSinkFactory.scala
index 6fd1f7afb16..0b685239e05 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/factories/BatchTableSinkFactory.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/factories/BatchTableSinkFactory.scala
@@ -23,7 +23,7 @@ import java.util
import org.apache.flink.table.sinks.BatchTableSink
/**
- * A factory to create configured table sink instances in a streaming
environment based on
+ * A factory to create configured table sink instances in a batch environment
based on
* string-based properties. See also [[TableFactory]] for more information.
*
* @tparam T type of records that the factory consumes
diff --git
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
index 64fcc8ac7c4..1209595837b 100644
---
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
+++
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
@@ -85,7 +85,9 @@ object CommonTestData {
.withSchema(schemaDesc1)
if (isStreaming) {
- externalTableBuilder1.inAppendMode()
+ externalTableBuilder1.supportsStreaming().inAppendMode()
+} else {
+ externalTableBuilder1.supportsBatch()
}
val csvRecord2 = Seq(
@@ -126,7 +128,9 @@ object CommonTestData {
.withSchema(schemaDesc2)
if (isStreaming) {
- externalTableBuilder2.inAppendMode()
+ externalTableBuilder2.supportsStreaming().inAppendMode()
+} else {
+ externalTableBuilder2.supportsBatch()
}
[jira] [Closed] (FLINK-11026) Rework creation of sql-client connector/format jars
[
https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-11026.
Resolution: Fixed
Fix Version/s: 1.8.0
master: b6237688b27ee2ca2cdda49820c5d2d541e12bbc
> Rework creation of sql-client connector/format jars
> ---
>
> Key: FLINK-11026
> URL: https://issues.apache.org/jira/browse/FLINK-11026
> Project: Flink
> Issue Type: Improvement
> Components: Build System, SQL Client
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For the SQL client we currently have a separate {{sql-jars}} profile in
> various connectors/formats that create an additional fat jar with a separate
> classifier.
> One of the core maven mantras is "One artifact per module.", and we see the
> importance of this mantra as our current packaging strategy makes it
> impossible to provide different NOTICE files for the created jars (regular
> and sql-jar).
> Currently we would be forced to provide the same file for both jars, which
> will cause problems for any downstream users that wants to handle NOTICE
> files properly. We would cause the same issue we had with netty, which
> categorically claims to be bundling dependencies although it doesn't, forcing
> us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that
> depend on the original module.
> I will use {{flink-connector-elasticsearch6}} as an example, which packages
> both a regular jar without any included dependencies, and a sql jar bundling
> everything.
> * create a separate
> {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}}
> module
> * this module depends on {{flink-connector-elasticsearch6}}, and bundles all
> dependencies
> * move the current shading logic for the sql jar out of the {{sql-jars}}
> profile{{}}
> * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the
> creation of these jars
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11268) Update deploy_staging_jars.sh to deploy multiple flink-shaded-hadoop2 versions
Chesnay Schepler created FLINK-11268:
Summary: Update deploy_staging_jars.sh to deploy multiple
flink-shaded-hadoop2 versions
Key: FLINK-11268
URL: https://issues.apache.org/jira/browse/FLINK-11268
Project: Flink
Issue Type: Sub-task
Components: Release System
Affects Versions: 1.8.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.8.0
We will have to make a few changes to the {{fink-shaded-hadoop2}} module to
support deployment of multiple versions. Currently neither the module name nor
version contains the contained hadoop version; as such maven cannot
differentiate between 2 versions of this artifact that were built with
different a {{hadoop.version}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (FLINK-11026) Rework creation of sql-client connector/format jars
[
https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reopened FLINK-11026:
--
> Rework creation of sql-client connector/format jars
> ---
>
> Key: FLINK-11026
> URL: https://issues.apache.org/jira/browse/FLINK-11026
> Project: Flink
> Issue Type: Improvement
> Components: Build System, SQL Client
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For the SQL client we currently have a separate {{sql-jars}} profile in
> various connectors/formats that create an additional fat jar with a separate
> classifier.
> One of the core maven mantras is "One artifact per module.", and we see the
> importance of this mantra as our current packaging strategy makes it
> impossible to provide different NOTICE files for the created jars (regular
> and sql-jar).
> Currently we would be forced to provide the same file for both jars, which
> will cause problems for any downstream users that wants to handle NOTICE
> files properly. We would cause the same issue we had with netty, which
> categorically claims to be bundling dependencies although it doesn't, forcing
> us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that
> depend on the original module.
> I will use {{flink-connector-elasticsearch6}} as an example, which packages
> both a regular jar without any included dependencies, and a sql jar bundling
> everything.
> * create a separate
> {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}}
> module
> * this module depends on {{flink-connector-elasticsearch6}}, and bundles all
> dependencies
> * move the current shading logic for the sql jar out of the {{sql-jars}}
> profile{{}}
> * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the
> creation of these jars
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-11026) Rework creation of sql-client connector/format jars
[
https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734042#comment-16734042
]
Chesnay Schepler edited comment on FLINK-11026 at 1/4/19 11:11 AM:
---
master:
kafka: b6237688b27ee2ca2cdda49820c5d2d541e12bbc
ES6: eb7039d48d2dcc39fc5b35ce21b459d2cefd2beb
was (Author: zentol):
master: b6237688b27ee2ca2cdda49820c5d2d541e12bbc
> Rework creation of sql-client connector/format jars
> ---
>
> Key: FLINK-11026
> URL: https://issues.apache.org/jira/browse/FLINK-11026
> Project: Flink
> Issue Type: Improvement
> Components: Build System, SQL Client
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For the SQL client we currently have a separate {{sql-jars}} profile in
> various connectors/formats that create an additional fat jar with a separate
> classifier.
> One of the core maven mantras is "One artifact per module.", and we see the
> importance of this mantra as our current packaging strategy makes it
> impossible to provide different NOTICE files for the created jars (regular
> and sql-jar).
> Currently we would be forced to provide the same file for both jars, which
> will cause problems for any downstream users that wants to handle NOTICE
> files properly. We would cause the same issue we had with netty, which
> categorically claims to be bundling dependencies although it doesn't, forcing
> us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that
> depend on the original module.
> I will use {{flink-connector-elasticsearch6}} as an example, which packages
> both a regular jar without any included dependencies, and a sql jar bundling
> everything.
> * create a separate
> {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}}
> module
> * this module depends on {{flink-connector-elasticsearch6}}, and bundles all
> dependencies
> * move the current shading logic for the sql jar out of the {{sql-jars}}
> profile{{}}
> * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the
> creation of these jars
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-11026) Rework creation of sql-client connector/format jars
[
https://issues.apache.org/jira/browse/FLINK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-11026.
Resolution: Fixed
> Rework creation of sql-client connector/format jars
> ---
>
> Key: FLINK-11026
> URL: https://issues.apache.org/jira/browse/FLINK-11026
> Project: Flink
> Issue Type: Improvement
> Components: Build System, SQL Client
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For the SQL client we currently have a separate {{sql-jars}} profile in
> various connectors/formats that create an additional fat jar with a separate
> classifier.
> One of the core maven mantras is "One artifact per module.", and we see the
> importance of this mantra as our current packaging strategy makes it
> impossible to provide different NOTICE files for the created jars (regular
> and sql-jar).
> Currently we would be forced to provide the same file for both jars, which
> will cause problems for any downstream users that wants to handle NOTICE
> files properly. We would cause the same issue we had with netty, which
> categorically claims to be bundling dependencies although it doesn't, forcing
> us to manually cut out the valid parts.
> My proposal is to move custom packaging strategies into their own module that
> depend on the original module.
> I will use {{flink-connector-elasticsearch6}} as an example, which packages
> both a regular jar without any included dependencies, and a sql jar bundling
> everything.
> * create a separate
> {{flink-sql-connector-elasticsearch6/}}{{flink-connector-elasticsearch6-uber}}{{}}
> module
> * this module depends on {{flink-connector-elasticsearch6}}, and bundles all
> dependencies
> * move the current shading logic for the sql jar out of the {{sql-jars}}
> profile{{}}
> * add a {{sql-jars}} profile to {{flink-connectors}} for skipping the
> creation of these jars
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zentol closed pull request #7341: [FLINK-11026][kafka][SQL] Rework kafka sql-client jar creation
zentol closed pull request #7341: [FLINK-11026][kafka][SQL] Rework kafka
sql-client jar creation
URL: https://github.com/apache/flink/pull/7341
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/connect.md b/docs/dev/table/connect.md
index 2255f1f0dfc..0d67444c184 100644
--- a/docs/dev/table/connect.md
+++ b/docs/dev/table/connect.md
@@ -45,10 +45,10 @@ The following table list all available connectors and
formats. Their mutual comp
| Filesystem| | Built-in |
Built-in |
| Elasticsearch | 6 | `flink-connector-elasticsearch6` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-elasticsearch6{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-elasticsearch6{{site.scala_version_suffix}}-{{site.version}}.jar)
|
| Apache Kafka | 0.8 | `flink-connector-kafka-0.8` | Not
available |
-| Apache Kafka | 0.9 | `flink-connector-kafka-0.9` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka-0.9{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka-0.9{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar)
|
-| Apache Kafka | 0.10| `flink-connector-kafka-0.10` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka-0.10{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka-0.10{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar)
|
-| Apache Kafka | 0.11| `flink-connector-kafka-0.11` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka-0.11{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka-0.11{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar)
|
-| Apache Kafka | 0.11+ (`universal`) | `flink-connector-kafka` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-connector-kafka{{site.scala_version_suffix}}/{{site.version}}/flink-connector-kafka{{site.scala_version_suffix}}-{{site.version}}-sql-jar.jar)
|
+| Apache Kafka | 0.9 | `flink-connector-kafka-0.9` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka-0.9{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka-0.9{{site.scala_version_suffix}}-{{site.version}}.jar)
|
+| Apache Kafka | 0.10| `flink-connector-kafka-0.10` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka-0.10{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka-0.10{{site.scala_version_suffix}}-{{site.version}}.jar)
|
+| Apache Kafka | 0.11| `flink-connector-kafka-0.11` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka-0.11{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka-0.11{{site.scala_version_suffix}}-{{site.version}}.jar)
|
+| Apache Kafka | 0.11+ (`universal`) | `flink-connector-kafka` |
[Download](http://central.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka{{site.scala_version_suffix}}/{{site.version}}/flink-sql-connector-kafka{{site.scala_version_suffix}}-{{site.version}}.jar)
|
### Formats
diff --git a/flink-connectors/flink-connector-kafka-0.10/pom.xml
b/flink-connectors/flink-connector-kafka-0.10/pom.xml
index 75ced033e90..8871d791e4f 100644
--- a/flink-connectors/flink-connector-kafka-0.10/pom.xml
+++ b/flink-connectors/flink-connector-kafka-0.10/pom.xml
@@ -174,59 +174,6 @@ under the License.
-
-
-
- sql-jars
-
-
- !skipSqlJars
-
-
-
-
-
-
org.apache.maven.plugins
-
maven-shade-plugin
-
-
-
package
-
-
shade
-
-
-
true
-
[jira] [Closed] (FLINK-11234) ExternalTableCatalogBuilder unable to build a batch-only table
[
https://issues.apache.org/jira/browse/FLINK-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-11234.
Resolution: Fixed
Fix Version/s: 1.7.2
1.6.4
Fixed in master: 04035ce8b7927136c7f1e43f0514a055ce45e56a
Fixed in 1.7: ae832444320d546275ab1214df7255324d3b8fa9
Fixed in 1.6: eeb1814036243119a60f8133a8591f5b7da0e8f8
> ExternalTableCatalogBuilder unable to build a batch-only table
> --
>
> Key: FLINK-11234
> URL: https://issues.apache.org/jira/browse/FLINK-11234
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL, Tests
>Reporter: Eron Wright
>Assignee: Eron Wright
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.6.4, 1.7.2, 1.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The {{ExternalTableCatalogBuilder::supportsBatch}} method should set
> {{isBatch}} to {{true}} and {{isStreaming}} to {{false}}, but the logic is
> presently inverted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-11269) Extend Download page to list optional components
Chesnay Schepler created FLINK-11269: Summary: Extend Download page to list optional components Key: FLINK-11269 URL: https://issues.apache.org/jira/browse/FLINK-11269 Project: Flink Issue Type: Improvement Components: Project Website Reporter: Chesnay Schepler Assignee: Chesnay Schepler Extend the download page of the Flink Website to support listing additional optional jars to be added to a Flink distribution. These optional jars may include (among others): * flink-shaded-hadoop2 (see FLINK-11266) * SQL format/connector jars * metric reporters Overall this will allow us to slim down flink-dist and make it more convenient for users to download these jars. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11267) Update create_binary_release.sh to not create hadoop-specific releases
Chesnay Schepler created FLINK-11267: Summary: Update create_binary_release.sh to not create hadoop-specific releases Key: FLINK-11267 URL: https://issues.apache.org/jira/browse/FLINK-11267 Project: Flink Issue Type: Sub-task Components: Release System Affects Versions: 1.8.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.8.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-11266) Only release hadoop-free Flink
Chesnay Schepler created FLINK-11266:
Summary: Only release hadoop-free Flink
Key: FLINK-11266
URL: https://issues.apache.org/jira/browse/FLINK-11266
Project: Flink
Issue Type: Improvement
Components: Release System
Affects Versions: 1.8.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Fix For: 1.8.0
Currently we release 10 different binary releases (2 scala versions * (4 hadoop
version + hadoop-free)), which has increased the size of our release to more
than 2 GB.
Naturally, building Flink 10 times also takes a while, slowing down the release
process.
However, the only difference between the hadoop versions is the bundled
{{flink-shaded-hadoop2}} jar; the rest is completely identical.
I propose to stop releasing hadoop-specific distributions, and instead have us
release multiple versions of {{flink-shaded-hadoop2}} that users copy into the
hadoop-free distribution if required.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] KarmaGYZ commented on a change in pull request #7402: [hotfix][docs] Fulfill some empty description in Config Doc
KarmaGYZ commented on a change in pull request #7402: [hotfix][docs] Fulfill
some empty description in Config Doc
URL: https://github.com/apache/flink/pull/7402#discussion_r245267301
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java
##
@@ -89,78 +95,89 @@
public static final ConfigOption UPLOAD_DIR =
key("web.upload.dir")
.noDefaultValue()
- .withDeprecatedKeys("jobmanager.web.upload.dir");
+ .withDeprecatedKeys("jobmanager.web.upload.dir")
+ .withDescription("The config parameter defining the
directory for uploading the job jars. If not" +
+ " specified a dynamic directory will be used
under the directory specified by JOB_MANAGER_WEB_TMPDIR_KEY.");
/**
* The config parameter defining the number of archived jobs for the
jobmanager.
*/
public static final ConfigOption ARCHIVE_COUNT =
key("web.history")
.defaultValue(5)
- .withDeprecatedKeys("jobmanager.web.history");
+ .withDeprecatedKeys("jobmanager.web.history")
+ .withDescription("The config parameter defining the
number of archived jobs for the jobmanager.");
/**
* The log file location (may be in /log for standalone but under log
directory when using YARN).
*/
public static final ConfigOption LOG_PATH =
key("web.log.path")
.noDefaultValue()
- .withDeprecatedKeys("jobmanager.web.log.path");
+ .withDeprecatedKeys("jobmanager.web.log.path")
+ .withDescription("Define the log file location (may be
in /log for standalone but under log directory when using YARN).");
Review comment:
Thanks for the advice. Agree with you. Could you kindly point out which one
is insufficient or just give me an example?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-11253) Incorrect way to stop yarn session described in yarn_setup document
[
https://issues.apache.org/jira/browse/FLINK-11253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-11253:
---
Labels: pull-request-available (was: )
> Incorrect way to stop yarn session described in yarn_setup document
> ---
>
> Key: FLINK-11253
> URL: https://issues.apache.org/jira/browse/FLINK-11253
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: Tao Yang
>Assignee: leesf
>Priority: Minor
> Labels: pull-request-available
>
> There are two ways to stop yarn session described in yarn_setup document:
> {noformat}
> Stop the YARN session by stopping the unix process (using CTRL+C) or by
> entering ‘stop’ into the client.
> {noformat}
> But in fact, yarn session application still can run after stopping the unix
> process (using CTRL+C).
> We can either update the yarn_setup document to remove this incorrect way or
> add ShutdownHook to stop yarn session in FlinkYarnSessionCli to make it
> correct.
> Looking forward to the feedbacks and would like to work on this ticket.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] KarmaGYZ commented on a change in pull request #7402: [hotfix][docs] Fulfill some empty description in Config Doc
KarmaGYZ commented on a change in pull request #7402: [hotfix][docs] Fulfill
some empty description in Config Doc
URL: https://github.com/apache/flink/pull/7402#discussion_r245267301
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java
##
@@ -89,78 +95,89 @@
public static final ConfigOption UPLOAD_DIR =
key("web.upload.dir")
.noDefaultValue()
- .withDeprecatedKeys("jobmanager.web.upload.dir");
+ .withDeprecatedKeys("jobmanager.web.upload.dir")
+ .withDescription("The config parameter defining the
directory for uploading the job jars. If not" +
+ " specified a dynamic directory will be used
under the directory specified by JOB_MANAGER_WEB_TMPDIR_KEY.");
/**
* The config parameter defining the number of archived jobs for the
jobmanager.
*/
public static final ConfigOption ARCHIVE_COUNT =
key("web.history")
.defaultValue(5)
- .withDeprecatedKeys("jobmanager.web.history");
+ .withDeprecatedKeys("jobmanager.web.history")
+ .withDescription("The config parameter defining the
number of archived jobs for the jobmanager.");
/**
* The log file location (may be in /log for standalone but under log
directory when using YARN).
*/
public static final ConfigOption LOG_PATH =
key("web.log.path")
.noDefaultValue()
- .withDeprecatedKeys("jobmanager.web.log.path");
+ .withDeprecatedKeys("jobmanager.web.log.path")
+ .withDescription("Define the log file location (may be
in /log for standalone but under log directory when using YARN).");
Review comment:
Thanks for the advise.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-11252) Add Scala 2.12 download column
[ https://issues.apache.org/jira/browse/FLINK-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-11252: - Summary: Add Scala 2.12 download column (was: Download page contains irrelevant "Scala 2.11" column) > Add Scala 2.12 download column > -- > > Key: FLINK-11252 > URL: https://issues.apache.org/jira/browse/FLINK-11252 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Attachments: downloads.png > > Time Spent: 20m > Remaining Estimate: 0h > > The download page has a "Scala 2.11" column, that was used in the past to > provide distinct download links for different scala versions. > We currently however list releases separately for each scala version. > We should either remove the column title or refactor the download page to > also have a "Scala 2.12" column. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #7386: [FLINK-11234][table][tests] fix ExternalTableCatalogBuilder::supportsBatch
asfgit closed pull request #7386: [FLINK-11234][table][tests] fix
ExternalTableCatalogBuilder::supportsBatch
URL: https://github.com/apache/flink/pull/7386
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
index 45414ee3ff6..ce57070ac98 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/catalog/ExternalCatalogTable.scala
@@ -270,8 +270,8 @@ class ExternalCatalogTableBuilder(private val
connectorDescriptor: ConnectorDesc
* Explicitly declares this external table for supporting only batch
environments.
*/
def supportsBatch(): ExternalCatalogTableBuilder = {
-isBatch = false
-isStreaming = true
+isBatch = true
+isStreaming = false
this
}
diff --git
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
index 64fcc8ac7c4..1209595837b 100644
---
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
+++
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/utils/CommonTestData.scala
@@ -85,7 +85,9 @@ object CommonTestData {
.withSchema(schemaDesc1)
if (isStreaming) {
- externalTableBuilder1.inAppendMode()
+ externalTableBuilder1.supportsStreaming().inAppendMode()
+} else {
+ externalTableBuilder1.supportsBatch()
}
val csvRecord2 = Seq(
@@ -126,7 +128,9 @@ object CommonTestData {
.withSchema(schemaDesc2)
if (isStreaming) {
- externalTableBuilder2.inAppendMode()
+ externalTableBuilder2.supportsStreaming().inAppendMode()
+} else {
+ externalTableBuilder2.supportsBatch()
}
val tempFilePath3 = writeToTempFile("", "csv-test3", "tmp")
@@ -145,7 +149,9 @@ object CommonTestData {
.withSchema(schemaDesc3)
if (isStreaming) {
- externalTableBuilder3.inAppendMode()
+ externalTableBuilder3.supportsStreaming().inAppendMode()
+} else {
+ externalTableBuilder3.supportsBatch()
}
val catalog = new InMemoryExternalCatalog("test")
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-11253) Incorrect way to stop yarn session described in yarn_setup document
[
https://issues.apache.org/jira/browse/FLINK-11253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734024#comment-16734024
]
Tao Yang commented on FLINK-11253:
--
Thanks [~zjffdu] and [~till.rohrmann] for your suggestions.
I have submitted a PR to add shutdown hook for yarn seesion client in attached
mode, could you please help to review in your free time? Thanks.
> Incorrect way to stop yarn session described in yarn_setup document
> ---
>
> Key: FLINK-11253
> URL: https://issues.apache.org/jira/browse/FLINK-11253
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: Tao Yang
>Assignee: leesf
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are two ways to stop yarn session described in yarn_setup document:
> {noformat}
> Stop the YARN session by stopping the unix process (using CTRL+C) or by
> entering ‘stop’ into the client.
> {noformat}
> But in fact, yarn session application still can run after stopping the unix
> process (using CTRL+C).
> We can either update the yarn_setup document to remove this incorrect way or
> add ShutdownHook to stop yarn session in FlinkYarnSessionCli to make it
> correct.
> Looking forward to the feedbacks and would like to work on this ticket.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-11252) Add Scala 2.12 download column
[ https://issues.apache.org/jira/browse/FLINK-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-11252. Resolution: Fixed asf-site: e7ece77db1621c1b993932b53803fa69b3f8629d > Add Scala 2.12 download column > -- > > Key: FLINK-11252 > URL: https://issues.apache.org/jira/browse/FLINK-11252 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Attachments: downloads.png > > Time Spent: 20m > Remaining Estimate: 0h > > The download page has a "Scala 2.11" column, that was used in the past to > provide distinct download links for different scala versions. > We currently however list releases separately for each scala version. > We should either remove the column title or refactor the download page to > also have a "Scala 2.12" column. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yt526 opened a new pull request #7414: [FLINK-11253] Add shutdown hook for yarn session client in attached mode
yt526 opened a new pull request #7414: [FLINK-11253] Add shutdown hook for yarn session client in attached mode URL: https://github.com/apache/flink/pull/7414 ## What is the purpose of the change This pull request makes yarn session cluster shutdown through adding shutdown hook if it's submitted in attached mode. ## Brief change log - *Add shutdown hook for the attached mode in FlinkYarnSessionCli, and move the shutdown logic to a independent method which can be called by shutdown hook and the handling process of stop command* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? docs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] pnowojski commented on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer
pnowojski commented on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer URL: https://github.com/apache/flink/pull/7405#issuecomment-451407399 Good point @zentol. It looks like kafka producer migrations tests are missing and only kafka consumer has it. Such test must be added alongside the fix for [FLINK-11249](https://issues.apache.org/jira/browse/FLINK-11249) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-11265) Invalid reference to AvroSinkWriter in example AvroKeyValueSinkWriter
[ https://issues.apache.org/jira/browse/FLINK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-11265: --- Labels: pull-request-available (was: ) > Invalid reference to AvroSinkWriter in example AvroKeyValueSinkWriter > - > > Key: FLINK-11265 > URL: https://issues.apache.org/jira/browse/FLINK-11265 > Project: Flink > Issue Type: Bug >Affects Versions: 1.7.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 1.7.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko opened a new pull request #7413: [FLINK-11265][docs] AvroSinkWriter → AvroKeyValueSinkWriter
Fokko opened a new pull request #7413: [FLINK-11265][docs] AvroSinkWriter → AvroKeyValueSinkWriter URL: https://github.com/apache/flink/pull/7413 AvroSinkWriter should be AvroKeyValueSinkWriter This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-11265) Invalid reference to AvroSinkWriter in example AvroKeyValueSinkWriter
Fokko Driesprong created FLINK-11265: Summary: Invalid reference to AvroSinkWriter in example AvroKeyValueSinkWriter Key: FLINK-11265 URL: https://issues.apache.org/jira/browse/FLINK-11265 Project: Flink Issue Type: Bug Affects Versions: 1.7.1 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 1.7.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zentol commented on a change in pull request #7402: [hotfix][docs] Fulfill some empty description in Config Doc
zentol commented on a change in pull request #7402: [hotfix][docs] Fulfill some
empty description in Config Doc
URL: https://github.com/apache/flink/pull/7402#discussion_r245254995
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java
##
@@ -89,78 +95,89 @@
public static final ConfigOption UPLOAD_DIR =
key("web.upload.dir")
.noDefaultValue()
- .withDeprecatedKeys("jobmanager.web.upload.dir");
+ .withDeprecatedKeys("jobmanager.web.upload.dir")
+ .withDescription("The config parameter defining the
directory for uploading the job jars. If not" +
+ " specified a dynamic directory will be used
under the directory specified by JOB_MANAGER_WEB_TMPDIR_KEY.");
/**
* The config parameter defining the number of archived jobs for the
jobmanager.
*/
public static final ConfigOption ARCHIVE_COUNT =
key("web.history")
.defaultValue(5)
- .withDeprecatedKeys("jobmanager.web.history");
+ .withDeprecatedKeys("jobmanager.web.history")
+ .withDescription("The config parameter defining the
number of archived jobs for the jobmanager.");
/**
* The log file location (may be in /log for standalone but under log
directory when using YARN).
*/
public static final ConfigOption LOG_PATH =
key("web.log.path")
.noDefaultValue()
- .withDeprecatedKeys("jobmanager.web.log.path");
+ .withDeprecatedKeys("jobmanager.web.log.path")
+ .withDescription("Define the log file location (may be
in /log for standalone but under log directory when using YARN).");
Review comment:
let's get some consistency in here. Some descriptions start with `The config
parameter`, others with `Config parameter defining`, and others with `Define`.
I know you just copied the javadocs, but imo this isn't sufficient for the
docs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tillrohrmann commented on a change in pull request #7346: [FLINK-11134][rest] Do not log stacktrace handled exceptions
tillrohrmann commented on a change in pull request #7346: [FLINK-11134][rest]
Do not log stacktrace handled exceptions
URL: https://github.com/apache/flink/pull/7346#discussion_r245252896
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/AbstractHandler.java
##
@@ -160,28 +160,46 @@ protected void respondAsLeader(ChannelHandlerContext
ctx, RoutedRequest routedRe
final FileUploads finalUploadedFiles = uploadedFiles;
requestProcessingFuture
.whenComplete((Void ignored, Throwable
throwable) -> {
-
inFlightRequestTracker.deregisterRequest();
- cleanupFileUploads(finalUploadedFiles);
+ if (throwable != null) {
+
handleException(ExceptionUtils.stripCompletionException(throwable), ctx,
httpRequest);
+ }
+
finalizeRequestProcessing(finalUploadedFiles);
Review comment:
We are slightly changing semantics here, because before we finalized the
request processing after the error message has been sent. Now we send the error
message and without waiting for the completion we finalize the request right
away. One could change it to
`requestProcessingFuture.exceptionally(handleException).thenCompose(Function.identity).whenComplete()`.
But I think it should also be ok to finalize the request right away since we
only send an error message and whether the error message arrives or the client
connection is terminated via a `ConnectionException` should not matter too much.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zentol commented on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer
zentol commented on issue #7405: [FLINK-11249] FlinkKafkaProducer011 can not be migrated to FlinkKafkaProducer URL: https://github.com/apache/flink/pull/7405#issuecomment-451398602 @pnowojski Do we have a migration test for the kafka producers(i.e. from 1.7 -> 1.8)? I'm curious since travis still passed, even though it should fail for current users. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-11244) flink web.port can not be configured correctly in yarn mode
[ https://issues.apache.org/jira/browse/FLINK-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-11244. Resolution: Not A Problem > flink web.port can not be configured correctly in yarn mode > --- > > Key: FLINK-11244 > URL: https://issues.apache.org/jira/browse/FLINK-11244 > Project: Flink > Issue Type: Bug > Components: Metrics, REST, YARN >Affects Versions: 1.4.2 > Environment: ubuntu 16.04 > openjdk-8 > Hadoop 2.7.1.2.3.6.0-3796 > flink 1.4.0 >Reporter: Kaicheng Sun >Priority: Major > Labels: documentation > Attachments: capture.PNG > > > I want to get flink's metrics information via REST api, my flink is managed > by YARN, but after changing web.port configuration in flink-conf.yaml, the > change has no affect, and the web.port in the flink dashboard is always 0. So > I can not get the flink metrics information via REST api. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (FLINK-11244) flink web.port can not be configured correctly in yarn mode
[ https://issues.apache.org/jira/browse/FLINK-11244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reopened FLINK-11244: -- > flink web.port can not be configured correctly in yarn mode > --- > > Key: FLINK-11244 > URL: https://issues.apache.org/jira/browse/FLINK-11244 > Project: Flink > Issue Type: Bug > Components: Metrics, REST, YARN >Affects Versions: 1.4.2 > Environment: ubuntu 16.04 > openjdk-8 > Hadoop 2.7.1.2.3.6.0-3796 > flink 1.4.0 >Reporter: Kaicheng Sun >Priority: Major > Labels: documentation > Attachments: capture.PNG > > > I want to get flink's metrics information via REST api, my flink is managed > by YARN, but after changing web.port configuration in flink-conf.yaml, the > change has no affect, and the web.port in the flink dashboard is always 0. So > I can not get the flink metrics information via REST api. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zentol closed pull request #7411: [hotfix][docs] Fix typos
zentol closed pull request #7411: [hotfix][docs] Fix typos
URL: https://github.com/apache/flink/pull/7411
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index 351a4dc2d41..b7f65b5cd1c 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -91,7 +91,7 @@ For most users, the `FlinkKafkaConsumer08` (part of
`flink-connector-kafka`) is
The version of the client it uses may change between Flink releases.
Modern Kafka clients are backwards compatible with broker versions
0.10.0 or later.
However for Kafka 0.11.x and 0.10.x versions, we recommend using
dedicated
-flink-connector-kafka-0.11{{ site.scala_version_suffix }} and
link-connector-kafka-0.10{{ site.scala_version_suffix }} respectively.
+flink-connector-kafka-0.11{{ site.scala_version_suffix }} and
flink-connector-kafka-0.10{{ site.scala_version_suffix }} respectively.
Attention: as of Flink 1.7 the universal Kafka
connector is considered to be
in a BETA status and might not be as stable as the
0.11 connector.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-10866) Queryable state can prevent cluster from starting
[
https://issues.apache.org/jira/browse/FLINK-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10866:
---
Labels: pull-request-available (was: )
> Queryable state can prevent cluster from starting
> -
>
> Key: FLINK-10866
> URL: https://issues.apache.org/jira/browse/FLINK-10866
> Project: Flink
> Issue Type: Improvement
> Components: Local Runtime
>Affects Versions: 1.5.5, 1.6.2, 1.7.0
>Reporter: Till Rohrmann
>Assignee: BoWang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> The {{KvStateServerImpl}} can currently prevent the {{TaskExecutor}} from
> starting.
> Currently, the QS server starts per default on port {{9067}}. If this port is
> not free, then it fails and stops the whole initialization of the
> {{TaskExecutor}}. I think the QS server should not stop the {{TaskExecutor}}
> from starting.
> We should at least change the default port to {{0}} to avoid port conflicts.
> However, this will break all setups which don't explicitly set the QS port
> because now it either needs to be setup or extracted from the logs.
> Additionally, we should think about whether a QS server startup failure
> should lead to a {{TaskExecutor}} failure or simply be logged. Both
> approaches have pros and cons. Currently, a failing QS server will also
> affect users which don't want to use QS. If we tolerate failures in the QS
> server, then a user who wants to use QS might run into problems with state
> not being reachable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] eaglewatcherwb opened a new pull request #7412: [FLINK-10866][Runtime] 1. Explicitly enable qs server and proxy. 2. QS
eaglewatcherwb opened a new pull request #7412: [FLINK-10866][Runtime] 1. Explicitly enable qs server and proxy. 2. QS URL: https://github.com/apache/flink/pull/7412 startup failure not block TM startup Change-Id: Id79c8bf97002a387a80be563a43dce3210143dc2 ## What is the purpose of the change 1. Explicitly enable qs server and proxy. 2. QS startup failure not block TM startup. ## Brief change log 1. add query.enable configuration to indicate whether start QS server and proxy, the default value of which is false. 2. QS server and proxy startup failures does not cause TM startup failure any more. ## Verifying this change This change added tests and can be verified as follows: 1. With default configuration, QS server and proxy does not start. 2. Set query.enable to true, QS server and proxy start. 3. QS server or proxy does not cause TM failure when got startup failures. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wangzzu opened a new pull request #7411: [hotfix][docs] Fix typos
wangzzu opened a new pull request #7411: [hotfix][docs] Fix typos URL: https://github.com/apache/flink/pull/7411 ## What is the purpose of the change Fix typos ## Brief change log Fix typos ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shuiqiangchen opened a new pull request #7410: [Flink 11264] Fail to start scala shell
shuiqiangchen opened a new pull request #7410: [Flink 11264] Fail to start scala shell URL: https://github.com/apache/flink/pull/7410 ## What is the purpose of the change fix the problem of Fail to start scala shell when outside the bin directory. ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) no - The serializers: (yes / no / don't know) no - The runtime per-record code paths (performance sensitive): (yes / no / don't know) no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know) no - The S3 file system connector: (yes / no / don't know) no ## Documentation - Does this pull request introduce a new feature? (yes / no) no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
