[jira] [Closed] (FLINK-16189) Remove testing logic from FlinkDistribution
[
https://issues.apache.org/jira/browse/FLINK-16189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-16189.
Resolution: Fixed
master: 1924f82512e17a43f874b8c86492d9ce6c8d7dd1
> Remove testing logic from FlinkDistribution
> ---
>
> Key: FLINK-16189
> URL: https://issues.apache.org/jira/browse/FLINK-16189
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, the {{FlinkDistribution}} is responsible for locating and copying
> the distribution for usage in tests, and contains hooks for backing up log
> files.
> This makes the class a bit inflexible to use, as for example a
> {{FlinkResource}} could not create 2 distributions in separate places.
> I suggest to move these responsibilities into the {{FlinkResource}}
> implementations, and "demote" the distribution to a simple wrapper, only
> providing programmatic access for mutating/interacting with the distribution.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16172) Add baseline set of allowed unused dependencies
[ https://issues.apache.org/jira/browse/FLINK-16172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052232#comment-17052232 ] Chesnay Schepler edited comment on FLINK-16172 at 3/5/20, 3:11 PM: --- master: 8587ce8dacb2b6c9fae1d3d3fd923ee0a32b7ee4 was (Author: zentol): master: 8587ce8dacb2b6c9fae1d3d3fd923ee0a32b7ee4 a762e76c0fa270367a6cf6cebca807ba80e124ae > Add baseline set of allowed unused dependencies > --- > > Key: FLINK-16172 > URL: https://issues.apache.org/jira/browse/FLINK-16172 > Project: Flink > Issue Type: Sub-task > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We define several dependencies in our root pom, usually for logging (e.g., > log4j), testing (e.g., junit) or build purposes (e.g., force-shading), to > reduce noise in downstream modules. > These commonly show up as unused, because they may actually not be used (like > junit in uber modules), are used indirectly (log4j implementation) or don't > contain any classes (force-shading). > We should define a baseline for unused dependencies that we allow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16172) Add baseline set of allowed unused dependencies
[ https://issues.apache.org/jira/browse/FLINK-16172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-16172. Resolution: Fixed master: 8587ce8dacb2b6c9fae1d3d3fd923ee0a32b7ee4 a762e76c0fa270367a6cf6cebca807ba80e124ae > Add baseline set of allowed unused dependencies > --- > > Key: FLINK-16172 > URL: https://issues.apache.org/jira/browse/FLINK-16172 > Project: Flink > Issue Type: Sub-task > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We define several dependencies in our root pom, usually for logging (e.g., > log4j), testing (e.g., junit) or build purposes (e.g., force-shading), to > reduce noise in downstream modules. > These commonly show up as unused, because they may actually not be used (like > junit in uber modules), are used indirectly (log4j implementation) or don't > contain any classes (force-shading). > We should define a baseline for unused dependencies that we allow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16173) Reduce noise for used undeclared dependencies
[ https://issues.apache.org/jira/browse/FLINK-16173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-16173. Resolution: Fixed master: a762e76c0fa270367a6cf6cebca807ba80e124ae > Reduce noise for used undeclared dependencies > - > > Key: FLINK-16173 > URL: https://issues.apache.org/jira/browse/FLINK-16173 > Project: Flink > Issue Type: Sub-task > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.11.0 > > > When analyzing dependency usages there are a bunch of noisy entries, > particularly around transitive Flink dependencies, hamcrest and powermock, > that would create more work than benefit. > These should be allowed to be used without having an explicit dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #11140: [FLINK-16172][build] Add baseline set of allowed unused dependencies
zentol merged pull request #11140: [FLINK-16172][build] Add baseline set of allowed unused dependencies URL: https://github.com/apache/flink/pull/11140 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-6537) Umbrella issue for fixes to incremental snapshots
[ https://issues.apache.org/jira/browse/FLINK-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-6537. --- > Umbrella issue for fixes to incremental snapshots > - > > Key: FLINK-6537 > URL: https://issues.apache.org/jira/browse/FLINK-6537 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.3.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This issue tracks ongoing fixes in the incremental checkpointing feature for > the 1.3 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-6537) Umbrella issue for fixes to incremental snapshots
[ https://issues.apache.org/jira/browse/FLINK-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-6537. - Resolution: Fixed All subtasks are closed. > Umbrella issue for fixes to incremental snapshots > - > > Key: FLINK-6537 > URL: https://issues.apache.org/jira/browse/FLINK-6537 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.3.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This issue tracks ongoing fixes in the incremental checkpointing feature for > the 1.3 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-6540) Add more thorough tests for RocksDB and incremental checkpointing
[ https://issues.apache.org/jira/browse/FLINK-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-6540. --- Resolution: Abandoned > Add more thorough tests for RocksDB and incremental checkpointing > - > > Key: FLINK-6540 > URL: https://issues.apache.org/jira/browse/FLINK-6540 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Affects Versions: 1.3.0 >Reporter: Stefan Richter >Priority: Major > > We should have tests that cover all the sub-issues from the umbrella issue, > to prevent that similar problems can be reintroduced in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-6599) Communicate last committed checkpoint through trigger checkpoint message
[
https://issues.apache.org/jira/browse/FLINK-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen closed FLINK-6599.
---
Resolution: Abandoned
> Communicate last committed checkpoint through trigger checkpoint message
>
>
> Key: FLINK-6599
> URL: https://issues.apache.org/jira/browse/FLINK-6599
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Affects Versions: 1.3.0
>Reporter: Stefan Richter
>Priority: Minor
>
> Small potential improvement:
> We could communicate the last completed checkpoint, which will serve as basis
> for future incremental checkpoints, through the trigger checkpoint message
> for that checkpoint, i.e. the trigger determines the base checkpoint for the
> next increment. This would be slightly more reliable than going through the
> {{CheckpointListener}} mechanism, which is not guaranteed the deliver its
> message.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol edited a comment on issue #11305: [FLINK-16410][e2e][build] Add explicit flink-runtime dependency
zentol edited a comment on issue #11305: [FLINK-16410][e2e][build] Add explicit flink-runtime dependency URL: https://github.com/apache/flink/pull/11305#issuecomment-595277009 > Would every test now need to know about this mechanism and need to manually promote the transitive runtime dependency? No, because we are fixing this at the source. Note that this PR only modifies `flink-end-to-end-tests`, not the module of the failing test `flink-metrics-reporter-prometheus-test`. This is all just due to a combination of using transitive undeclared dependencies + provided scope + dependency reduction. Could we make flink-dist a `compile` dependency? Sure, but it's a) highly unusual for us to do so and b) we don't actually want all transitive dependencies to bleed into downstream modules. Could we ban used undeclared dependencies? Sure, but no one (but me) wants that noise in the pom. Could we selectively disable dependency reduction? Sure, but that introduces additional special cases of which we have more than enough. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
flinkbot edited a comment on issue #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager URL: https://github.com/apache/flink/pull/11323#issuecomment-595267888 ## CI report: * 2e0fb83e0784cd3b3315856e51e0fece5eedd0ed Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151974539) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3.
flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3. URL: https://github.com/apache/flink/pull/11123#issuecomment-587468233 ## CI report: * 3d28892370884e69d1dfa550c69fd664d0053fdd Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151954486) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-16410) PrometheusReporterEndToEndITCase fails with ClassNotFoundException
[
https://issues.apache.org/jira/browse/FLINK-16410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-16410.
Fix Version/s: 1.11.0
Resolution: Fixed
master: b6f07ddfe779698f929959289c317dbafefc4eb5
> PrometheusReporterEndToEndITCase fails with ClassNotFoundException
> --
>
> Key: FLINK-16410
> URL: https://issues.apache.org/jira/browse/FLINK-16410
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics, Tests
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Logs:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5883=logs=b1623ac9-0979-5b0d-2e5e-1377d695c991=e7804547-1789-5225-2bcf-269eeaa37447
> {code}
> [INFO] Running
> org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.005
> s <<< FAILURE! - in
> org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase
> [ERROR]
> testReporter(org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase)
> Time elapsed: 0.005 s <<< ERROR!
> java.lang.NoClassDefFoundError:
> org/apache/flink/runtime/rest/messages/RequestBody
> at
> org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase.(PrometheusReporterEndToEndITCase.java:119)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.runtime.rest.messages.RequestBody
> at
> org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase.(PrometheusReporterEndToEndITCase.java:119)
> [INFO]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #11305: [FLINK-16410][e2e][build] Add explicit flink-runtime dependency
zentol merged pull request #11305: [FLINK-16410][e2e][build] Add explicit flink-runtime dependency URL: https://github.com/apache/flink/pull/11305 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on issue #11305: [FLINK-16410][e2e][build] Add explicit flink-runtime dependency
zentol commented on issue #11305: [FLINK-16410][e2e][build] Add explicit flink-runtime dependency URL: https://github.com/apache/flink/pull/11305#issuecomment-595277009 > Would every test now need to know about this mechanism and need to manually promote the transitive runtime dependency? No, because we are fixing this at the source. Note that this PR only modifies `flink-end-to-end-tests`, not the module of the failing test `flink-metrics-reporter-prometheus-test`. This is all just due to a combination of using transitive undeclared dependencies + provided scope + dependency reduction. Could we make flink-dist a `compile` dependency? Sure, but it's a) highly unusual for us to do so and b) we don't actually want all transitive dependencies to bleed into downstream modules. Could we ban used undeclared dependencies? Sure, but no one wants that noise in the pom. Could we selectively disable dependency reduction? Sure, but that introduces additional special cases of which we have more than enough. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388340028
##
File path:
flink-core/src/main/java/org/apache/flink/core/plugin/PluginLoader.java
##
@@ -69,7 +68,7 @@ public static PluginLoader create(PluginDescriptor
pluginDescriptor, ClassLoader
* @param Type of the requested plugin service.
* @return An iterator of all implementations of the given service
interface that could be loaded from the plugin.
*/
- public Iterator load(Class service) {
+ public Iterator load(Class service) {
Review comment:
That was actually my proposal, since there is no benefit from implementing
Plugin and its `#configure` method does not work well with existing metric
factories.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388346850
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/ReporterSetup.java
##
@@ -179,28 +164,82 @@ private static ReporterSetup createReporterSetup(String
reporterName, MetricConf
metricReporterOptional.ifPresent(reporter -> {
MetricConfig metricConfig = new
MetricConfig();
reporterConfig.addAllToProperties(metricConfig);
-
-
reporterArguments.add(createReporterSetup(reporterName, metricConfig,
reporter));
+
reporterSetups.add(createReporterSetup(reporterName, metricConfig, reporter));
});
- }
- catch (Throwable t) {
+ } catch (Throwable t) {
LOG.error("Could not instantiate metrics
reporter {}. Metrics might not be exposed/reported.", reporterName, t);
}
}
- return reporterArguments;
+ return reporterSetups;
}
- private static Map
loadReporterFactories() {
- final ServiceLoader serviceLoader =
ServiceLoader.load(MetricReporterFactory.class);
+ private static List>
loadReporterConfigurations(Configuration configuration, Set
namedReporters) {
+ final List>
reporterConfigurations = new ArrayList<>(namedReporters.size());
+ for (String namedReporter: namedReporters) {
+ DelegatingConfiguration delegatingConfiguration = new
DelegatingConfiguration(
+ configuration,
+ ConfigConstants.METRICS_REPORTER_PREFIX +
namedReporter + '.');
+
+ reporterConfigurations.add(Tuple2.of(namedReporter,
delegatingConfiguration));
+ }
+ return reporterConfigurations;
+ }
+
+ private static Set
findEnabledReportersInConfiguration(Configuration configuration, String
includedReportersString) {
+ Set includedReporters =
reporterListPattern.splitAsStream(includedReportersString)
+ .filter(r -> !r.isEmpty()) // splitting an empty string
results in an empty string on jdk9+
+ .collect(Collectors.toSet());
+
+ // use a TreeSet to make the reporter order deterministic,
which is useful for testing
+ Set namedOrderedReporters = new
TreeSet<>(String::compareTo);
+
+ // scan entire configuration for keys starting with
METRICS_REPORTER_PREFIX and determine the set of enabled reporters
+ for (String key : configuration.keySet()) {
+ if
(key.startsWith(ConfigConstants.METRICS_REPORTER_PREFIX)) {
+ Matcher matcher =
reporterClassPattern.matcher(key);
+ if (matcher.matches()) {
Review comment:
warn if it doesn't match?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388310058
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java
##
@@ -160,13 +161,14 @@ public void startCluster() throws
ClusterEntrypointException {
LOG.info("Starting {}.", getClass().getSimpleName());
try {
-
- configureFileSystems(configuration);
+ //TODO: push down filesystem initialization into
runCluster - initializeServices (?)
Review comment:
Just tagging TODO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388347812
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/ReporterSetup.java
##
@@ -178,36 +164,96 @@ private static ReporterSetup createReporterSetup(String
reporterName, MetricConf
metricReporterOptional.ifPresent(reporter -> {
MetricConfig metricConfig = new
MetricConfig();
reporterConfig.addAllToProperties(metricConfig);
-
-
reporterArguments.add(createReporterSetup(reporterName, metricConfig,
reporter));
+
reporterSetups.add(createReporterSetup(reporterName, metricConfig, reporter));
});
- }
- catch (Throwable t) {
+ } catch (Throwable t) {
LOG.error("Could not instantiate metrics
reporter {}. Metrics might not be exposed/reported.", reporterName, t);
}
}
- return reporterArguments;
+ return reporterSetups;
}
- private static Map
loadReporterFactories() {
- final ServiceLoader serviceLoader =
ServiceLoader.load(MetricReporterFactory.class);
+ private static List>
loadReporterConfigurations(Configuration configuration, Set
namedReporters) {
+ final List>
reporterConfigurations = new ArrayList<>(namedReporters.size());
+
+ for (String namedReporter: namedReporters) {
+ DelegatingConfiguration delegatingConfiguration = new
DelegatingConfiguration(
+ configuration,
+ ConfigConstants.METRICS_REPORTER_PREFIX +
namedReporter + '.');
+ reporterConfigurations.add(Tuple2.of(namedReporter,
delegatingConfiguration));
+ }
+ return reporterConfigurations;
+ }
+
+ private static Set
findEnabledReportersInConfiguration(Configuration configuration, String
includedReportersString) {
+ Set includedReporters =
reporterListPattern.splitAsStream(includedReportersString)
+ .filter(r -> !r.isEmpty()) // splitting an empty string
results in an empty string on jdk9+
+ .collect(Collectors.toSet());
+
+ // use a TreeSet to make the reporter order deterministic,
which is useful for testing
+ Set namedOrderedReporters = new
TreeSet<>(String::compareTo);
+
+ // scan entire configuration for keys starting with
METRICS_REPORTER_PREFIX and determine the set of enabled reporters
+ for (String key : configuration.keySet()) {
+ if
(key.startsWith(ConfigConstants.METRICS_REPORTER_PREFIX)) {
+ Matcher matcher =
reporterClassPattern.matcher(key);
+ if (matcher.matches()) {
+ String reporterName = matcher.group(1);
+ if (includedReporters.isEmpty() ||
includedReporters.contains(reporterName)) {
+ if
(namedOrderedReporters.contains(reporterName)) {
+ LOG.warn("Duplicate
class configuration detected for reporter {}.", reporterName);
+ } else {
+
namedOrderedReporters.add(reporterName);
+ }
+ } else {
+ LOG.info("Excluding reporter
{}, not configured in reporter list ({}).", reporterName,
includedReportersString);
+ }
+ }
+ }
+ }
+ return namedOrderedReporters;
+ }
+
+ private static Map
loadAvailableReporterFactories(PluginManager pluginManager) {
final Map reporterFactories =
new HashMap<>(2);
- final Iterator factoryIterator =
serviceLoader.iterator();
+ final Iterator factoryIterator =
getAllReporterFactories(pluginManager);
+ LOG.debug("All available factories (from both SPIs and
Plugins):");
+ getAllReporterFactories(pluginManager).forEachRemaining(i ->
LOG.debug(i.toString()));
// do not use streams or for-each loops here because they do
not allow catching individual ServiceConfigurationErrors
// such an error might be caused if the META-INF/services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388346247
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/ReporterSetup.java
##
@@ -179,28 +164,82 @@ private static ReporterSetup createReporterSetup(String
reporterName, MetricConf
metricReporterOptional.ifPresent(reporter -> {
MetricConfig metricConfig = new
MetricConfig();
reporterConfig.addAllToProperties(metricConfig);
-
-
reporterArguments.add(createReporterSetup(reporterName, metricConfig,
reporter));
+
reporterSetups.add(createReporterSetup(reporterName, metricConfig, reporter));
});
- }
- catch (Throwable t) {
+ } catch (Throwable t) {
LOG.error("Could not instantiate metrics
reporter {}. Metrics might not be exposed/reported.", reporterName, t);
}
}
- return reporterArguments;
+ return reporterSetups;
}
- private static Map
loadReporterFactories() {
- final ServiceLoader serviceLoader =
ServiceLoader.load(MetricReporterFactory.class);
+ private static List>
loadReporterConfigurations(Configuration configuration, Set
namedReporters) {
+ final List>
reporterConfigurations = new ArrayList<>(namedReporters.size());
+ for (String namedReporter: namedReporters) {
+ DelegatingConfiguration delegatingConfiguration = new
DelegatingConfiguration(
+ configuration,
+ ConfigConstants.METRICS_REPORTER_PREFIX +
namedReporter + '.');
+
+ reporterConfigurations.add(Tuple2.of(namedReporter,
delegatingConfiguration));
+ }
+ return reporterConfigurations;
+ }
+
+ private static Set
findEnabledReportersInConfiguration(Configuration configuration, String
includedReportersString) {
+ Set includedReporters =
reporterListPattern.splitAsStream(includedReportersString)
+ .filter(r -> !r.isEmpty()) // splitting an empty string
results in an empty string on jdk9+
+ .collect(Collectors.toSet());
Review comment:
Collectors.toCollection(TreeSet::new) to get rid of the next few lines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388344724
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/ReporterSetup.java
##
@@ -120,55 +124,37 @@ public static ReporterSetup forReporter(String
reporterName, MetricConfig metric
}
private static ReporterSetup createReporterSetup(String reporterName,
MetricConfig metricConfig, MetricReporter reporter) {
- LOG.info("Configuring {} with {}.", reporterName, metricConfig);
+ LOG.debug("Configuring {} with {}.", reporterName,
metricConfig);
reporter.open(metricConfig);
return new ReporterSetup(reporterName, metricConfig, reporter);
}
- public static List fromConfiguration(final Configuration
configuration) {
+ public static List fromConfiguration(final Configuration
configuration, final PluginManager pluginManager) {
+ LOG.debug("Initializing Reporters from Configuration: {}",
configuration);
String includedReportersString =
configuration.getString(MetricOptions.REPORTERS_LIST, "");
- Set includedReporters =
reporterListPattern.splitAsStream(includedReportersString)
Review comment:
to split up refactoring from actual commit. But in general also to
refactorings.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388312499
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java
##
@@ -119,7 +120,7 @@
private boolean shutdown;
- public TaskManagerRunner(Configuration configuration, ResourceID
resourceId) throws Exception {
+ public TaskManagerRunner(Configuration configuration, ResourceID
resourceId, PluginManager pluginManager) throws Exception {
Review comment:
`FileSystem` can also take a null for plugin manager. We should probably
extract an interface and have a no-op implementation instead. That's out of
scope for this PR though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388310406
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java
##
@@ -202,9 +204,11 @@ private SecurityContext
installSecurityContext(Configuration configuration) thro
return SecurityUtils.getInstalledContext();
}
- private void runCluster(Configuration configuration) throws Exception {
+ private void runCluster(Configuration configuration, PluginManager
pluginManager) throws Exception {
synchronized (lock) {
- initializeServices(configuration);
+
+ //TODO: Ask why FileSystem is not initialized here too.
Review comment:
Just tagging TODO.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters URL: https://github.com/apache/flink/pull/11195#discussion_r388313102 ## File path: flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/FlinkDistribution.java ## @@ -77,6 +77,7 @@ private Path conf; Review comment: Commit message should explain what's actually happening. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388344253
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/ReporterSetup.java
##
@@ -178,36 +164,96 @@ private static ReporterSetup createReporterSetup(String
reporterName, MetricConf
metricReporterOptional.ifPresent(reporter -> {
MetricConfig metricConfig = new
MetricConfig();
reporterConfig.addAllToProperties(metricConfig);
-
-
reporterArguments.add(createReporterSetup(reporterName, metricConfig,
reporter));
+
reporterSetups.add(createReporterSetup(reporterName, metricConfig, reporter));
});
- }
- catch (Throwable t) {
+ } catch (Throwable t) {
Review comment:
In general, the best option is to leave old code as is to not blow up the
PR. You could make a separate hotfix to address code style fixes though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use plugins mechanism for initializing MetricReporters
AHeise commented on a change in pull request #11195: [FLINK-16222][runtime] Use
plugins mechanism for initializing MetricReporters
URL: https://github.com/apache/flink/pull/11195#discussion_r388348348
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/ReporterSetup.java
##
@@ -178,36 +164,96 @@ private static ReporterSetup createReporterSetup(String
reporterName, MetricConf
metricReporterOptional.ifPresent(reporter -> {
MetricConfig metricConfig = new
MetricConfig();
reporterConfig.addAllToProperties(metricConfig);
-
-
reporterArguments.add(createReporterSetup(reporterName, metricConfig,
reporter));
+
reporterSetups.add(createReporterSetup(reporterName, metricConfig, reporter));
});
- }
- catch (Throwable t) {
+ } catch (Throwable t) {
LOG.error("Could not instantiate metrics
reporter {}. Metrics might not be exposed/reported.", reporterName, t);
}
}
- return reporterArguments;
+ return reporterSetups;
}
- private static Map
loadReporterFactories() {
- final ServiceLoader serviceLoader =
ServiceLoader.load(MetricReporterFactory.class);
+ private static List>
loadReporterConfigurations(Configuration configuration, Set
namedReporters) {
+ final List>
reporterConfigurations = new ArrayList<>(namedReporters.size());
+
+ for (String namedReporter: namedReporters) {
+ DelegatingConfiguration delegatingConfiguration = new
DelegatingConfiguration(
+ configuration,
+ ConfigConstants.METRICS_REPORTER_PREFIX +
namedReporter + '.');
+ reporterConfigurations.add(Tuple2.of(namedReporter,
delegatingConfiguration));
+ }
+ return reporterConfigurations;
+ }
+
+ private static Set
findEnabledReportersInConfiguration(Configuration configuration, String
includedReportersString) {
+ Set includedReporters =
reporterListPattern.splitAsStream(includedReportersString)
+ .filter(r -> !r.isEmpty()) // splitting an empty string
results in an empty string on jdk9+
+ .collect(Collectors.toSet());
+
+ // use a TreeSet to make the reporter order deterministic,
which is useful for testing
+ Set namedOrderedReporters = new
TreeSet<>(String::compareTo);
+
+ // scan entire configuration for keys starting with
METRICS_REPORTER_PREFIX and determine the set of enabled reporters
+ for (String key : configuration.keySet()) {
+ if
(key.startsWith(ConfigConstants.METRICS_REPORTER_PREFIX)) {
+ Matcher matcher =
reporterClassPattern.matcher(key);
+ if (matcher.matches()) {
+ String reporterName = matcher.group(1);
+ if (includedReporters.isEmpty() ||
includedReporters.contains(reporterName)) {
+ if
(namedOrderedReporters.contains(reporterName)) {
+ LOG.warn("Duplicate
class configuration detected for reporter {}.", reporterName);
+ } else {
+
namedOrderedReporters.add(reporterName);
+ }
+ } else {
+ LOG.info("Excluding reporter
{}, not configured in reporter list ({}).", reporterName,
includedReportersString);
+ }
+ }
+ }
+ }
+ return namedOrderedReporters;
+ }
+
+ private static Map
loadAvailableReporterFactories(PluginManager pluginManager) {
final Map reporterFactories =
new HashMap<>(2);
- final Iterator factoryIterator =
serviceLoader.iterator();
+ final Iterator factoryIterator =
getAllReporterFactories(pluginManager);
+ LOG.debug("All available factories (from both SPIs and
Plugins):");
+ getAllReporterFactories(pluginManager).forEachRemaining(i ->
LOG.debug(i.toString()));
// do not use streams or for-each loops here because they do
not allow catching individual ServiceConfigurationErrors
// such an error might be caused if the META-INF/services
[jira] [Resolved] (FLINK-16177) Integrate Operator Coordinators with Checkpoints
[ https://issues.apache.org/jira/browse/FLINK-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-16177. -- Resolution: Fixed Implemented in 1.11.0 via 963974f99d18d6a9f36fa78b792dcc2bc9e53de5 > Integrate Operator Coordinators with Checkpoints > > > Key: FLINK-16177 > URL: https://issues.apache.org/jira/browse/FLINK-16177 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The operator coordinators are stateful and hence need to store state in > checkpoints. > See FLIP-27 for details on operator coordinators: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > The initial implementation approach is to trigger coordinator checkpoints > first, and when all coordinator checkpoints are done, then the source > checkpoint barriers will be injected. > _Note:_ This functionality will eventually replace the checkpoint master > hooks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-16177) Integrate Operator Coordinators with Checkpoints
[ https://issues.apache.org/jira/browse/FLINK-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-16177. > Integrate Operator Coordinators with Checkpoints > > > Key: FLINK-16177 > URL: https://issues.apache.org/jira/browse/FLINK-16177 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The operator coordinators are stateful and hence need to store state in > checkpoints. > See FLIP-27 for details on operator coordinators: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > The initial implementation approach is to trigger coordinator checkpoints > first, and when all coordinator checkpoints are done, then the source > checkpoint barriers will be injected. > _Note:_ This functionality will eventually replace the checkpoint master > hooks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] asfgit closed pull request #11274: [FLINK-16177][checkpointing] Integrate OperatorCoordinator checkpoint triggering and committing
asfgit closed pull request #11274: [FLINK-16177][checkpointing] Integrate OperatorCoordinator checkpoint triggering and committing URL: https://github.com/apache/flink/pull/11274 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner
flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner URL: https://github.com/apache/flink/pull/11252#issuecomment-592511050 ## CI report: * 43041585ccc2dd381baf70240aca178fef9c7b93 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151949187) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5968) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
flinkbot commented on issue #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager URL: https://github.com/apache/flink/pull/11323#issuecomment-595267888 ## CI report: * 2e0fb83e0784cd3b3315856e51e0fece5eedd0ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3.
flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3. URL: https://github.com/apache/flink/pull/11123#issuecomment-587468233 ## CI report: * 3d28892370884e69d1dfa550c69fd664d0053fdd Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151954486) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
flinkbot commented on issue #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager URL: https://github.com/apache/flink/pull/11323#issuecomment-595263117 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 2e0fb83e0784cd3b3315856e51e0fece5eedd0ed (Thu Mar 05 14:40:54 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-16439).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16439) Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
[ https://issues.apache.org/jira/browse/FLINK-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16439: --- Labels: pull-request-available (was: ) > Make KubernetesResourceManager starts workers using WorkerResourceSpec > requested by SlotManager > --- > > Key: FLINK-16439 > URL: https://issues.apache.org/jira/browse/FLINK-16439 > Project: Flink > Issue Type: Sub-task > Components: Deployment / Kubernetes >Reporter: Xintong Song >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > This means KubernetesResourceManager no longer: > - be aware of the default task executor resources > - assumes all workers are identical -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong opened a new pull request #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager
xintongsong opened a new pull request #11323: [FLINK-16439][k8s] Make KubernetesResourceManager starts workers using WorkerResourceSpec requested by SlotManager URL: https://github.com/apache/flink/pull/11323 ## What is the purpose of the change This PR is one step of FLINK-14106, making `KubernetesResourceManager` starts workers using `WorkerResourceSpec` requested by `SlotManager`. This also means `KubernetesResourceManager` no longer: - be aware of the default task executor resources - assumes all workers are identical This PR is based on #11320. ## Brief change log - 632fd000bcf0552ccc7fe58a7d33e3dbffc766d7..b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0: Commits of previous PR. - f9f39d10deea936899a72a8433c6dbdc76c559e9: Minor code clean-ups. - 2e0fb83e0784cd3b3315856e51e0fece5eedd0ed: KubernetesResourceManager starts workers with resources requested by SlotManager. ## Verifying this change - Add `KubernetesResourceManagerTest#testStartAndRecoverVariousResourceSpec` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner
flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner URL: https://github.com/apache/flink/pull/11252#issuecomment-592511050 ## CI report: * 43041585ccc2dd381baf70240aca178fef9c7b93 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151949187) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5968) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 2a005b9c7d287b338a5a4c0795816d61e8f09312 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151951525) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5970) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16444) Count the read/write/seek/next latency of RocksDB as metrics
[ https://issues.apache.org/jira/browse/FLINK-16444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052173#comment-17052173 ] Stephan Ewen commented on FLINK-16444: -- +1 for this feature, users have asked about more insights into RocksDB's performance > Count the read/write/seek/next latency of RocksDB as metrics > > > Key: FLINK-16444 > URL: https://issues.apache.org/jira/browse/FLINK-16444 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yun Tang >Priority: Major > Fix For: 1.11.0 > > > Currently, user cannot know the read/write/seek/next latency of RocksDB, we > could add these helpful metrics to know the overall state performance. To not > affect the action performance much, we could introduce counter to only record > the latency at interval of some actions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16313) flink-state-processor-api: surefire execution unstable on Azure
[
https://issues.apache.org/jira/browse/FLINK-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052168#comment-17052168
]
Robert Metzger commented on FLINK-16313:
The jvm is writing a {{.dumpstream}} file with the following contents:
{code}
# Created at 2020-03-05T12:51:24.963
pure virtual method called
# Created at 2020-03-05T12:51:24.964
terminate called without an active exception
# Created at 2020-03-05T12:51:26.982
Aborted (core dumped)
{code}
... investigating further
> flink-state-processor-api: surefire execution unstable on Azure
> ---
>
> Key: FLINK-16313
> URL: https://issues.apache.org/jira/browse/FLINK-16313
> Project: Flink
> Issue Type: Bug
> Components: API / State Processor, Tests
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
>
> Log file:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5686=logs=41cba0bb-1271-5adb-01cc-4768f26a8311=44574c85-1cd0-5978-cccf-f0cf7e87a36a
> {code}
> 2020-02-27T12:36:35.2860111Z [INFO] flink-table-planner
> SUCCESS [01:47 min]
> 2020-02-27T12:36:35.2860966Z [INFO] flink-cep-scala
> SUCCESS [ 5.041 s]
> 2020-02-27T12:36:35.2861740Z [INFO] flink-sql-client
> ... SUCCESS [03:00 min]
> 2020-02-27T12:36:35.2862503Z [INFO] flink-state-processor-api
> .. FAILURE [ 15.394 s]
> 2020-02-27T12:36:35.2863237Z [INFO]
>
> 2020-02-27T12:36:35.2863587Z [INFO] BUILD FAILURE
> 2020-02-27T12:36:35.2864071Z [INFO]
>
> 2020-02-27T12:36:35.2864428Z [INFO] Total time: 05:38 min
> 2020-02-27T12:36:35.2866349Z [INFO] Finished at: 2020-02-27T12:36:35+00:00
> 2020-02-27T12:36:35.9345815Z [INFO] Final Memory: 147M/2914M
> 2020-02-27T12:36:35.9347238Z [INFO]
>
> 2020-02-27T12:36:35.9355362Z [WARNING] The requested profile
> "skip-webui-build" could not be activated because it does not exist.
> 2020-02-27T12:36:35.9367919Z [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test
> (integration-tests) on project flink-state-processor-api_2.11: There are test
> failures.
> 2020-02-27T12:36:35.9368804Z [ERROR]
> 2020-02-27T12:36:35.9369489Z [ERROR] Please refer to
> /__w/2/s/flink-libraries/flink-state-processing-api/target/surefire-reports
> for the individual test results.
> 2020-02-27T12:36:35.9370249Z [ERROR] Please refer to dump files (if any
> exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> 2020-02-27T12:36:35.9370713Z [ERROR] ExecutionException Error occurred in
> starting fork, check output in log
> 2020-02-27T12:36:35.9371279Z [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException Error occurred in starting fork, check output in log
> 2020-02-27T12:36:35.9372275Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> 2020-02-27T12:36:35.9372917Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> 2020-02-27T12:36:35.9373498Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
> 2020-02-27T12:36:35.9374064Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> 2020-02-27T12:36:35.9374636Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> 2020-02-27T12:36:35.9375344Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> 2020-02-27T12:36:35.9376194Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> 2020-02-27T12:36:35.9376791Z [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> 2020-02-27T12:36:35.9377375Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> 2020-02-27T12:36:35.9377898Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> 2020-02-27T12:36:35.9378435Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> 2020-02-27T12:36:35.9379063Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> 2020-02-27T12:36:35.9379709Z [ERROR] at
>
[jira] [Commented] (FLINK-16422) Cannot use [catalog].[db].table with Hive catalog
[ https://issues.apache.org/jira/browse/FLINK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052163#comment-17052163 ] Gyula Fora commented on FLINK-16422: Thanks [~Leonard Xu] for the nicer writeup :) > Cannot use [catalog].[db].table with Hive catalog > - > > Key: FLINK-16422 > URL: https://issues.apache.org/jira/browse/FLINK-16422 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Gyula Fora >Priority: Major > > When trying to select a table from the Hive , the SQL CLI automcompletes to > the full table name: > select * from hive.default.itemtransactions ; > but then we get the following error: > [ERROR] Could not execute SQL statement. Reason: > org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered > ". default" at line 1, column 19. > Was expecting one of:... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16444) Count the read/write/seek/next latency of RocksDB as metrics
Yun Tang created FLINK-16444: Summary: Count the read/write/seek/next latency of RocksDB as metrics Key: FLINK-16444 URL: https://issues.apache.org/jira/browse/FLINK-16444 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Reporter: Yun Tang Fix For: 1.11.0 Currently, user cannot know the read/write/seek/next latency of RocksDB, we could add these helpful metrics to know the overall state performance. To not affect the action performance much, we could introduce counter to only record the latency at interval of some actions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir…
flinkbot edited a comment on issue #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir… URL: https://github.com/apache/flink/pull/11322#issuecomment-595233010 ## CI report: * 497c65724ea2499cd2a5b82a7ce4ffd2f8807502 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151959839) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.
flinkbot edited a comment on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity. URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225 ## CI report: * b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151939995) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16443) Fix wrong fix for user-code CheckpointExceptions
Stephan Ewen created FLINK-16443:
Summary: Fix wrong fix for user-code CheckpointExceptions
Key: FLINK-16443
URL: https://issues.apache.org/jira/browse/FLINK-16443
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Reporter: Stephan Ewen
Fix For: 1.11.0
The problem of having exceptions that are only in the user code classloader was
fixed by proactively serializing them inside the {{CheckpointException}}. That
means all consumers of {{CheckpointException}} now need to be aware of that
and unwrap the serializable exception.
I believe the right way to fix this would have been to use a
SerializedException in the {{DeclineCheckpoint}} message instead, which would
have localized the change to the actual problem: RPC transport.
I would suggest to revert https://github.com/apache/flink/pull/9742 and instead
apply the above described change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise commented on a change in pull request #11303: [FLINK-16245] Decoupling user classloader from context classloader.
AHeise commented on a change in pull request #11303: [FLINK-16245] Decoupling
user classloader from context classloader.
URL: https://github.com/apache/flink/pull/11303#discussion_r388301548
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders.java
##
@@ -82,4 +91,78 @@ public static ResolveOrder fromString(String resolveOrder) {
super(urls, parent);
}
}
+
+ /**
+* Ensures that holding a reference on the context class loader
outliving the scope of user code does not prevent
+* the user classloader to be garbage collected (FLINK-16245).
+*
+* This classloader delegates to the actual user classloader. Upon
{@link #close()}, the delegate is nulled
+* and can be garbage collected. Additional class resolution will be
resolved solely through the bootstrap
+* classloader and most likely result in ClassNotFound exceptions.
+*
+* @param the classloader type that also needs to be closeable.
+*/
+ private static class SafetyNetWrapperClassLoader extends CloseableClassLoader
+ implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(SafetyNetWrapperClassLoader.class);
+
+ private T inner;
+
+ SafetyNetWrapperClassLoader(T inner) {
+ super(null);
+ this.inner = inner;
+ }
+
+ @Override
+ public void close() {
+ if (inner != null) {
+ try {
+ inner.close();
+ } catch (IOException e) {
+ LOG.warn("Could not close user
classloader", e);
+ }
+ }
+ inner = null;
+ }
+
+ @Override
+ protected Class loadClass(String name, boolean resolve)
throws ClassNotFoundException {
+ if (inner == null) {
+ return super.loadClass(name, resolve);
+ }
+
+ synchronized (getClassLoadingLock(name)) {
Review comment:
We would need to introduce a common ancestor to
`Flink[Parent|Child]FirstClassLoader` that does nothing but exposes
`loadClass`.
I'm really not convinced that this is better than do the resolution as is.
It would certainly not be more concise and imho it would increase the
complexity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11303: [FLINK-16245] Decoupling user classloader from context classloader.
AHeise commented on a change in pull request #11303: [FLINK-16245] Decoupling
user classloader from context classloader.
URL: https://github.com/apache/flink/pull/11303#discussion_r388299382
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders.java
##
@@ -18,35 +18,44 @@
package org.apache.flink.runtime.execution.librarycache;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.util.ChildFirstClassLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
+import java.util.Enumeration;
/**
* Gives the URLClassLoader a nicer name for debugging purposes.
*/
public class FlinkUserCodeClassLoaders {
- public static URLClassLoader parentFirst(URL[] urls, ClassLoader
parent) {
+ @VisibleForTesting
+ static URLClassLoader parentFirst(URL[] urls, ClassLoader parent) {
Review comment:
Hm it's not public API and they could achieve the same thing through
`#create` with one extra parameter.
Unless I change the signature of this method, this method cannot use the
safety net (or safety net would also need to be URLClassLoader which I wanted
to avoid), which may also be fine, but I wanted to point that out explicitly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16434) Add document to explain how to pack hive with their own hive dependencies
[ https://issues.apache.org/jira/browse/FLINK-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052143#comment-17052143 ] Leonard Xu commented on FLINK-16434: +1 for this document, hive has many versions and each version has its own dependencies, hope this improvement can help hive user to more easily integrate with Flink. > Add document to explain how to pack hive with their own hive dependencies > - > > Key: FLINK-16434 > URL: https://issues.apache.org/jira/browse/FLINK-16434 > Project: Flink > Issue Type: Task > Components: Connectors / Hive, Documentation >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.11.0 > > > > For HDP and CDH users, I think they probably should use the HDP and CDH > >version Hive jars. For example, HDP Hive-1.2.1 has cherry picked lots of > >patches from newer version and therefore is different from Apache > >Hive-1.2.1. So using the HDP version usually means users can have the extra > >bug fixes than the Apache version. > Users need add dependencies carefully, and choose a related hive version to > configure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir…
flinkbot commented on issue #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir… URL: https://github.com/apache/flink/pull/11322#issuecomment-595233010 ## CI report: * 497c65724ea2499cd2a5b82a7ce4ffd2f8807502 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11318: [FLINK-16435][python] Fix ide static check
flinkbot edited a comment on issue #11318: [FLINK-16435][python] Fix ide static check URL: https://github.com/apache/flink/pull/11318#issuecomment-595145846 ## CI report: * 380588ed7bcda11a5a6358b9c45a50d1af2caa28 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151929647) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5961) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
flinkbot edited a comment on issue #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader URL: https://github.com/apache/flink/pull/10922#issuecomment-577180180 ## CI report: * 37cb28b6b3238516df476318cb28d30114689442 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151674029) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5893) * 908bfd539ad84be428f8dd3c908680d87dfb4ec8 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151957099) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5972) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11296: [FLINK-16363] [table] Correct the execution behavior of TableEnvironment and StreamTableEnvironment
flinkbot edited a comment on issue #11296: [FLINK-16363] [table] Correct the execution behavior of TableEnvironment and StreamTableEnvironment URL: https://github.com/apache/flink/pull/11296#issuecomment-59383 ## CI report: * 42befb8bcb61fcb1d8c2551affe9da273f978295 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151690287) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5899) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #11238: [FLINK-16304][python] Remove python packages bundled in the flink-python jar.
WeiZhong94 commented on a change in pull request #11238: [FLINK-16304][python] Remove python packages bundled in the flink-python jar. URL: https://github.com/apache/flink/pull/11238#discussion_r388291429 ## File path: flink-python/src/main/resources/META-INF/licenses/LICENSE.cloudpickle ## @@ -1,32 +0,0 @@ -This module was extracted from the `cloud` package, developed by Review comment: @zentol As the `cloudpickle-1.2.2-src.zip` is still bundled to the binary release, its license file need to copy to the binary release. But `collect_license_files.sh` only collects the license files in jar, which means we need to add some code to ensure that the `LICENSE.cloudpickle` can be collected by the script. There are 2 solutions in my mind: 1. bundle the license file to `flink-python.jar` 2. let the `collect_license_files.sh` also collect the license files in the root `license` directory. What do you think about those solutions? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 2a005b9c7d287b338a5a4c0795816d61e8f09312 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151951525) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5970) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3.
flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3. URL: https://github.com/apache/flink/pull/11123#issuecomment-587468233 ## CI report: * 95e457e1d10f8c7b0de7e52fef42f6ac029912a6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151711705) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5912) * 3d28892370884e69d1dfa550c69fd664d0053fdd Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151954486) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11296: [FLINK-16363] [table] Correct the execution behavior of TableEnvironment and StreamTableEnvironment
flinkbot edited a comment on issue #11296: [FLINK-16363] [table] Correct the execution behavior of TableEnvironment and StreamTableEnvironment URL: https://github.com/apache/flink/pull/11296#issuecomment-59383 ## CI report: * 42befb8bcb61fcb1d8c2551affe9da273f978295 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151690287) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5899) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
flinkbot edited a comment on issue #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader URL: https://github.com/apache/flink/pull/10922#issuecomment-577180180 ## CI report: * 37cb28b6b3238516df476318cb28d30114689442 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151674029) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5893) * 908bfd539ad84be428f8dd3c908680d87dfb4ec8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir…
flinkbot commented on issue #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir… URL: https://github.com/apache/flink/pull/11322#issuecomment-595223763 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 497c65724ea2499cd2a5b82a7ce4ffd2f8807502 (Thu Mar 05 13:17:15 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] jiasheng55 opened a new pull request #11322: [FLINK-16376][yarn] Use consistent method to get Yarn application dir…
jiasheng55 opened a new pull request #11322: [FLINK-16376][yarn] Use consistent
method to get Yarn application dir…
URL: https://github.com/apache/flink/pull/11322
…ectory.
## What is the purpose of the change
Currently, the Yarn application directory of Flink is
"/user/{user.name}/.flink", but this logic is separated in different places.
This PR aims to improve this.
## Brief change log
- Move `getYarnFilesDir` method from `YarnClusterDescriptor` to `Utils`.
- Use `getYarnFilesDir` to get target directory of shipped files.
## Verifying this change
This change is already covered by existing tests, such as
*org.apache.flink.yarn.YarnFileStageTest*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (*yes, Yarn
deployment*)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (FLINK-16427) Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment
[
https://issues.apache.org/jira/browse/FLINK-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052111#comment-17052111
]
Zili Chen edited comment on FLINK-16427 at 3/5/20, 1:00 PM:
Because ProgramInvocationExceptions is in flink-client while
RemoteStreamEnvironment is in flink-streaming-java. Another approach might be
we move one or both of these class to another module for resolving the issue
but since it happens we use {{ProgramInvocationExceptions}} instead of a strong
requirement I tend to choose the way listed here.
was (Author: tison):
Because ProgramInvocationExceptions is in flink-client while
RemoteStreamEnvironment is in flink-streaming-java. Another perspective might
me we move one or both of these class to another module for resolving the issue
but since it happens we use {{ProgramInvocationExceptions}} instead of a strong
requirement I tend to choose the way listed here.
> Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment
>
>
> Key: FLINK-16427
> URL: https://issues.apache.org/jira/browse/FLINK-16427
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This issue is a child issue of FLINK-15090.
> With a previous discussion with [~aljoscha] we tend to reverse the dependency
> from flink-streaming-java to flink-client so that we have a semantic correct
> module dependency. That says, flink-streaming-java, just like
> flink-java(batch api), is an api module which should not know about the
> runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16427) Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment
[
https://issues.apache.org/jira/browse/FLINK-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052111#comment-17052111
]
Zili Chen commented on FLINK-16427:
---
Because ProgramInvocationExceptions is in flink-client while
RemoteStreamEnvironment is in flink-streaming-java. Another perspective might
me we move one or both of these class to another module for resolving the issue
but since it happens we use {{ProgramInvocationExceptions}} instead of a strong
requirement I tend to choose the way listed here.
> Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment
>
>
> Key: FLINK-16427
> URL: https://issues.apache.org/jira/browse/FLINK-16427
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This issue is a child issue of FLINK-15090.
> With a previous discussion with [~aljoscha] we tend to reverse the dependency
> from flink-streaming-java to flink-client so that we have a semantic correct
> module dependency. That says, flink-streaming-java, just like
> flink-java(batch api), is an api module which should not know about the
> runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
JingsongLi commented on a change in pull request #10922: [FLINK-11899][parquet]
Introduce parquet ColumnarRow split reader
URL: https://github.com/apache/flink/pull/10922#discussion_r388277823
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetSplitReaderUtil.java
##
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.parquet.vector;
+
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.parquet.vector.reader.BooleanColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ByteColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.BytesColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.DoubleColumnReader;
+import
org.apache.flink.formats.parquet.vector.reader.FixedLenBytesColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.FloatColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.IntColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.LongColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ShortColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.TimestampColumnReader;
+import org.apache.flink.table.dataformat.Decimal;
+import org.apache.flink.table.dataformat.SqlTimestamp;
+import org.apache.flink.table.dataformat.vector.ColumnVector;
+import org.apache.flink.table.dataformat.vector.VectorizedColumnBatch;
+import org.apache.flink.table.dataformat.vector.heap.HeapBooleanVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapByteVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapBytesVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapDoubleVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapFloatVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapIntVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapLongVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapShortVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapTimestampVector;
+import org.apache.flink.table.dataformat.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.PrimitiveType;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.sql.Date;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.runtime.functions.SqlDateTimeUtils.dateToInternal;
+import static org.apache.parquet.Preconditions.checkArgument;
+
+/**
+ * Util for generating {@link ParquetColumnarRowSplitReader}.
+ */
+public class ParquetSplitReaderUtil {
+
+ /**
+* Util for generating partitioned {@link
ParquetColumnarRowSplitReader}.
+*/
+ public static ParquetColumnarRowSplitReader genPartColumnarRowReader(
+ boolean utcTimestamp,
+ Configuration conf,
+ String[] fullFieldNames,
+ DataType[] fullFieldTypes,
+ Map partitionSpec,
+ int[] selectedFields,
+ int batchSize,
+ Path path,
+ long splitStart,
+ long splitLength) throws IOException {
+ List nonPartNames = Arrays.stream(fullFieldNames)
+
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 2a005b9c7d287b338a5a4c0795816d61e8f09312 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151951525) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5970) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11274: [FLINK-16177][checkpointing] Integrate OperatorCoordinator checkpoint triggering and committing
flinkbot edited a comment on issue #11274: [FLINK-16177][checkpointing] Integrate OperatorCoordinator checkpoint triggering and committing URL: https://github.com/apache/flink/pull/11274#issuecomment-593164504 ## CI report: * 94b8db822437a30b7342f83cea116023af131c53 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151277738) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5788) * 6a772a3f164ed5913cd6a311c5ffd968e6819401 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151951461) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11318: [FLINK-16435][python] Fix ide static check
flinkbot edited a comment on issue #11318: [FLINK-16435][python] Fix ide static check URL: https://github.com/apache/flink/pull/11318#issuecomment-595145846 ## CI report: * 380588ed7bcda11a5a6358b9c45a50d1af2caa28 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151929647) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5961) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3.
flinkbot edited a comment on issue #11123: [FLINK-16014][s3] Force usage of SAXParserFactory over XMLReaderFactory in aws-java-sdk-s3. URL: https://github.com/apache/flink/pull/11123#issuecomment-587468233 ## CI report: * 95e457e1d10f8c7b0de7e52fef42f6ac029912a6 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151711705) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5912) * 3d28892370884e69d1dfa550c69fd664d0053fdd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16427) Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment
[ https://issues.apache.org/jira/browse/FLINK-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052107#comment-17052107 ] Aljoscha Krettek commented on FLINK-16427: -- Why is this a prerequisite of FLINK-15090? > Remove directly throw ProgramInvocationExceptions in RemoteStreamEnvironment > > > Key: FLINK-16427 > URL: https://issues.apache.org/jira/browse/FLINK-16427 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Zili Chen >Assignee: Zili Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > This issue is a child issue of FLINK-15090. > With a previous discussion with [~aljoscha] we tend to reverse the dependency > from flink-streaming-java to flink-client so that we have a semantic correct > module dependency. That says, flink-streaming-java, just like > flink-java(batch api), is an api module which should not know about the > runtime. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] jiasheng55 closed pull request #11319: [FLINK-16376][yarn] Use consistent method to get Yarn application dir…
jiasheng55 closed pull request #11319: [FLINK-16376][yarn] Use consistent method to get Yarn application dir… URL: https://github.com/apache/flink/pull/11319 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16425) Add rate limiting feature for kafka table source
[ https://issues.apache.org/jira/browse/FLINK-16425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052104#comment-17052104 ] Zou commented on FLINK-16425: - [~jark] I find that there is also no rate limiting in FlinkKafkaConsumer, shall we add it first? And shall we open another issue to track this? > Add rate limiting feature for kafka table source > > > Key: FLINK-16425 > URL: https://issues.apache.org/jira/browse/FLINK-16425 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Table SQL / Ecosystem >Reporter: Zou >Priority: Major > > There is a rate limiting feature in kafka source, but kafka table source dose > not support this. We could add this feature in kafka table source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16422) Cannot use [catalog].[db].table with Hive catalog
[
https://issues.apache.org/jira/browse/FLINK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052103#comment-17052103
]
Leonard Xu edited comment on FLINK-16422 at 3/5/20, 12:54 PM:
--
[~gyfora]
I got it, SQL CLI should return :
{code:java}
Flink SQL> select * FROM myhive.^
myhive.`default`.user_info myhive.`default`.user_info
{code}
instead of :
{code:java}
Flink SQL> select * FROM myhive.^
myhive.default.user_info myhive.default.user_info
{code}
when user press tab for autocomplete.
The reason should be planner missed escaping keywords when get completion
hints.
refers: `FlinkPlannerImpl.getCompletionHints(sql: String, cursor: Int)`.
CC: [~danny0405]
was (Author: leonard xu):
[~gyfora]
I got it, SQL CLI should return :
{code:java}
Flink SQL> select * FROM myhive.^
myhive.`default`.user_info myhive.`default`.user_info
{code}
instead of :
{code:java}
Flink SQL> select * FROM myhive.^
myhive.default.user_info myhive.default.user_info
{code}
when user press tab for autocomplete.
The reason should be planner missed escaping keywords when get completion
hints.
refers: `FlinkPlannerImpl.getCompletionHints(sql: String, cursor: Int)`.
CC: [~danny0405]
> Cannot use [catalog].[db].table with Hive catalog
> -
>
> Key: FLINK-16422
> URL: https://issues.apache.org/jira/browse/FLINK-16422
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Major
>
> When trying to select a table from the Hive , the SQL CLI automcompletes to
> the full table name:
> select * from hive.default.itemtransactions ;
> but then we get the following error:
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered
> ". default" at line 1, column 19.
> Was expecting one of:...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16422) Cannot use [catalog].[db].table with Hive catalog
[
https://issues.apache.org/jira/browse/FLINK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052103#comment-17052103
]
Leonard Xu commented on FLINK-16422:
[~gyfora]
I got it, SQL CLI should return :
{code:java}
Flink SQL> select * FROM myhive.^
myhive.`default`.user_info myhive.`default`.user_info
{code}
instead of :
{code:java}
Flink SQL> select * FROM myhive.^
myhive.default.user_info myhive.default.user_info
{code}
when user press tab for autocomplete.
The reason should be planner missed escaping keywords when get completion
hints.
refers: `FlinkPlannerImpl.getCompletionHints(sql: String, cursor: Int)`.
CC: [~danny0405]
> Cannot use [catalog].[db].table with Hive catalog
> -
>
> Key: FLINK-16422
> URL: https://issues.apache.org/jira/browse/FLINK-16422
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Major
>
> When trying to select a table from the Hive , the SQL CLI automcompletes to
> the full table name:
> select * from hive.default.itemtransactions ;
> but then we get the following error:
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.SqlParserException: SQL parse failed. Encountered
> ". default" at line 1, column 19.
> Was expecting one of:...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14807) Add Table#collect api for fetching data to client
[
https://issues.apache.org/jira/browse/FLINK-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052101#comment-17052101
]
Stephan Ewen commented on FLINK-14807:
--
Nice to see this lively discussion. I agree that we should come up with a
proper solution and works backwards from there.
It would also be great to find a way that works across batch and streaming. Not
sure how easy that is possible, but maybe we can think along those lines.
About the ideas proposed so far:
* Concerning exactly-once semantics: That would be nice, but it would
inherently introduce a "checkpoint interval delay", because we can only feed
back completed results. Maybe that is okay in the streaming world, for batch
there is anyways only the "end of job" state.
* Concerning master-failover: I think we can get away with not supporting that
in the first version, as long as we point out that limitation.
* Concerning task-failover: That sounds like an expected functionality to me
* Server sockets as the basic data exchange plane are tricky. Lost connections,
retries, etc. There needs to be a more sophisticated protocol on top of that in
any case.
* The GlobalAggregateManager is itself actually a pretty bad hack at the
moment, which I would hope to replace by the {{OperatorCoordinators}} in the
future.
One thing we could try and do is enhance the accumulators, with blob-server
offload.
* This would be similar as the RPC service works. Small accumulator results are
directly in an RPC message, larger (aggregated results) may be offloaded to the
blob storage.
* We would send new accumulators with the checkpoint (or checkpoint commit),
giving exactly once semantics.
* Clients would pull the accumulators, the rest server would need to pull them
from blob storage if necessary.
* That might work, but it is a lot of indirection / out of band transfer as
soon as results get larger. Which might be expected to some extend, to keep the
memory footprint of the processes small.
> Add Table#collect api for fetching data to client
> -
>
> Key: FLINK-14807
> URL: https://issues.apache.org/jira/browse/FLINK-14807
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.9.1
>Reporter: Jeff Zhang
>Priority: Major
> Labels: usability
> Fix For: 1.11.0
>
> Attachments: table-collect-draft.patch, table-collect.png
>
>
> Currently, it is very unconvinient for user to fetch data of flink job unless
> specify sink expclitly and then fetch data from this sink via its api (e.g.
> write to hdfs sink, then read data from hdfs). However, most of time user
> just want to get the data and do whatever processing he want. So it is very
> necessary for flink to provide api Table#collect for this purpose.
>
> Other apis such as Table#head, Table#print is also helpful.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lirui-apache commented on a change in pull request #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
lirui-apache commented on a change in pull request #10922:
[FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
URL: https://github.com/apache/flink/pull/10922#discussion_r388246126
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetColumnarRowSplitReader.java
##
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.parquet.vector;
+
+import org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.table.dataformat.ColumnarRow;
+import org.apache.flink.table.dataformat.vector.ColumnVector;
+import org.apache.flink.table.dataformat.vector.VectorizedColumnBatch;
+import org.apache.flink.table.dataformat.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.filter2.compat.FilterCompat;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+
+import static
org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createColumnReader;
+import static
org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector;
+import static org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups;
+import static
org.apache.parquet.format.converter.ParquetMetadataConverter.range;
+import static org.apache.parquet.hadoop.ParquetFileReader.readFooter;
+import static org.apache.parquet.hadoop.ParquetInputFormat.getFilter;
+
+/**
+ * This reader is used to read a {@link VectorizedColumnBatch} from input
split.
+ */
+public class ParquetColumnarRowSplitReader implements Closeable {
+
+ private final boolean utcTimestamp;
+
+ private final MessageType fileSchema;
+
+ private final MessageType requestedSchema;
+
+ /**
+* The total number of rows this RecordReader will eventually read. The
sum of the rows of all
+* the row groups.
+*/
+ private final long totalRowCount;
+
+ private final WritableColumnVector[] writableVectors;
+
+ private final VectorizedColumnBatch columnarBatch;
+
+ private final ColumnarRow row;
+
+ private final LogicalType[] selectedTypes;
+
+ private final int batchSize;
+
+ private ParquetFileReader reader;
+
+ /**
+* For each request column, the reader to read this column. This is
NULL if this column is
+* missing from the file, in which case we populate the attribute with
NULL.
+*/
+ private ColumnReader[] columnReaders;
+
+ /**
+* The number of rows that have been returned.
+*/
+ private long rowsReturned;
+
+ /**
+* The number of rows that have been reading, including the current in
flight row group.
+*/
+ private long totalCountLoadedSoFar;
+
+ // the index of the next row to return
+ private int nextRow;
+
+ // the number of rows in the current batch
+ private int rowsInBatch;
+
+ public ParquetColumnarRowSplitReader(
+ boolean utcTimestamp,
+ Configuration conf,
+ LogicalType[] selectedTypes,
+ String[] selectedFieldNames,
+ ColumnBatchGenerator generator,
+ int batchSize,
+ Path path,
+ long splitStart,
+ long splitLength) throws IOException {
+
[GitHub] [flink] lirui-apache commented on a change in pull request #10922: [FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
lirui-apache commented on a change in pull request #10922:
[FLINK-11899][parquet] Introduce parquet ColumnarRow split reader
URL: https://github.com/apache/flink/pull/10922#discussion_r388249255
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/vector/ParquetSplitReaderUtil.java
##
@@ -0,0 +1,384 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.parquet.vector;
+
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.parquet.vector.reader.BooleanColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ByteColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.BytesColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.DoubleColumnReader;
+import
org.apache.flink.formats.parquet.vector.reader.FixedLenBytesColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.FloatColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.IntColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.LongColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.ShortColumnReader;
+import org.apache.flink.formats.parquet.vector.reader.TimestampColumnReader;
+import org.apache.flink.table.dataformat.Decimal;
+import org.apache.flink.table.dataformat.SqlTimestamp;
+import org.apache.flink.table.dataformat.vector.ColumnVector;
+import org.apache.flink.table.dataformat.vector.VectorizedColumnBatch;
+import org.apache.flink.table.dataformat.vector.heap.HeapBooleanVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapByteVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapBytesVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapDoubleVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapFloatVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapIntVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapLongVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapShortVector;
+import org.apache.flink.table.dataformat.vector.heap.HeapTimestampVector;
+import org.apache.flink.table.dataformat.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.PrimitiveType;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.charset.StandardCharsets;
+import java.sql.Date;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.runtime.functions.SqlDateTimeUtils.dateToInternal;
+import static org.apache.parquet.Preconditions.checkArgument;
+
+/**
+ * Util for generating {@link ParquetColumnarRowSplitReader}.
+ */
+public class ParquetSplitReaderUtil {
+
+ /**
+* Util for generating partitioned {@link
ParquetColumnarRowSplitReader}.
+*/
+ public static ParquetColumnarRowSplitReader genPartColumnarRowReader(
+ boolean utcTimestamp,
+ Configuration conf,
+ String[] fullFieldNames,
+ DataType[] fullFieldTypes,
+ Map partitionSpec,
+ int[] selectedFields,
+ int batchSize,
+ Path path,
+ long splitStart,
+ long splitLength) throws IOException {
+ List nonPartNames = Arrays.stream(fullFieldNames)
+
[jira] [Commented] (FLINK-16357) Extend Checkpoint Coordinator to differentiate between "regional restore" and "full restore".
[
https://issues.apache.org/jira/browse/FLINK-16357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052090#comment-17052090
]
Zhu Zhu commented on FLINK-16357:
-
I see. Yes, I think we can achieve it by adding a param like "restoreGlobally"
to {{CheckpointCoordinator#restoreLatestCheckpointedState(...)}} which only
invokes {{OperatorCoordinator#resetToCheckpoint(...)}} if the param is true.
This may need some small adjustment in {{DefaultScheduler}} and
{{SchedulerBase}}.
> Extend Checkpoint Coordinator to differentiate between "regional restore" and
> "full restore".
> -
>
> Key: FLINK-16357
> URL: https://issues.apache.org/jira/browse/FLINK-16357
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 1.11.0
>
>
> The {{ExecutionGraph}} has the notion of "global failure" (failing the entire
> execution graph) and "regional failure" (recover a region with transient
> pipelined data exchanges).
> The latter one is for common failover, the former one is a safety net to
> handle unexpected failures or inconsistencies (full reset of ExecutionGraph
> recovers most inconsistencies).
> The OperatorCoordinators should only be reset to a checkpoint in the "global
> failover" case. In the "regional failover" case, they are only notified of
> the tasks that are reset and keep their internal state and adjust it for the
> failed tasks.
> To implement that, the ExecutionGraph needs to forward the information about
> whether we are restoring from a "regional failure" or from a "global failure".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot commented on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 2a005b9c7d287b338a5a4c0795816d61e8f09312 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11316: [FLINK-16432] Fix dependencies in Hive Connector build
flinkbot edited a comment on issue #11316: [FLINK-16432] Fix dependencies in Hive Connector build URL: https://github.com/apache/flink/pull/11316#issuecomment-595136134 ## CI report: * 0e09cc2806e813a52f53f58b9ec4228465639668 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151926822) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5959) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner
flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner URL: https://github.com/apache/flink/pull/11252#issuecomment-592511050 ## CI report: * 542c6f80de4976c204a68968f6d5af383dad1ac5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151659712) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5890) * 43041585ccc2dd381baf70240aca178fef9c7b93 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151949187) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5968) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11274: [FLINK-16177][checkpointing] Integrate OperatorCoordinator checkpoint triggering and committing
flinkbot edited a comment on issue #11274: [FLINK-16177][checkpointing] Integrate OperatorCoordinator checkpoint triggering and committing URL: https://github.com/apache/flink/pull/11274#issuecomment-593164504 ## CI report: * 94b8db822437a30b7342f83cea116023af131c53 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151277738) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5788) * 6a772a3f164ed5913cd6a311c5ffd968e6819401 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu commented on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner
dianfu commented on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner URL: https://github.com/apache/flink/pull/11252#issuecomment-595204458 @hequn8128 Thanks a lot for your great review and suggestions. That makes much sense to me and have updated the PR accordingly. Regarding to the support of pandas udf for batch mode in old planner, I'd like to add it in a separate PR as the operator for this case is still not added. What's your thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot commented on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595203896 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 2a005b9c7d287b338a5a4c0795816d61e8f09312 (Thu Mar 05 12:26:57 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15585) Improve function identifier string in plan digest
[
https://issues.apache.org/jira/browse/FLINK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-15585:
---
Labels: pull-request-available (was: )
> Improve function identifier string in plan digest
> -
>
> Key: FLINK-15585
> URL: https://issues.apache.org/jira/browse/FLINK-15585
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Currently, we are using {{UserDefinedFunction#functionIdentifier}} as the
> identifier string of UDFs in plan digest, for example:
> {code:java}
> LogicalTableFunctionScan(invocation=[org$apache$flink$table$planner$utils$TableFunc1$8050927803993624f40152a838c98018($2)],
> rowType=...)
> {code}
> However, the result of {{UserDefinedFunction#functionIdentifier}} will change
> if we just add a method in UserDefinedFunction, because it uses Java
> serialization. Then we have to update 60 plan tests which is very annoying.
> In the other hand, displaying the function identifier string in operator name
> in Web UI is verbose to users.
> In order to improve this situation, there are something we can do:
> 1) If the UDF has a catalog function name, we can just use the catalog name
> as the digest. Otherwise, fallback to (2).
> 2) If the UDF doesn't contain fields, we just use the full calss name as the
> digest. Otherwise, fallback to (3).
> 3) Use identifier string which will do the full serialization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] godfreyhe opened a new pull request #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
godfreyhe opened a new pull request #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321 ## What is the purpose of the change *Currently, we use UserDefinedFunction#functionIdentifier as the identifier string of UDFs in plan digest. The identifier string looks like org$apache$flink$table$planner$utils$TableFunc1$8050927803993624f40152a838c98018. It's verbose to users and will be change if the UserDefinedFunction is updated. This pr aim to improve this, here is the principles: 1) If the UDF has a catalog function name, we can just use the catalog name as the digest. Otherwise, fallback to (2). 2) If the UDF doesn't contain fields, we just use the full calss name as the digest. Otherwise, fallback to (3). 3) Use identifier string which will do the full serialization.* ## Brief change log - *UserDefinedFunction#functionIdentifier returns the class name if the function has no constructor or only one constructor without any parameter, else returns the class name and instance's md5. - *unify function identifier and and display name * ## Verifying this change This change added tests and can be verified as follows: - *Many existing tests' plan should be updated* - *Added UserDefinedFunctionTest to verify the plan of different kinds of UDF* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11316: [FLINK-16432] Fix dependencies in Hive Connector build
flinkbot edited a comment on issue #11316: [FLINK-16432] Fix dependencies in Hive Connector build URL: https://github.com/apache/flink/pull/11316#issuecomment-595136134 ## CI report: * 0e09cc2806e813a52f53f58b9ec4228465639668 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151926822) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5959) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11317: [FLINK-16433][table-api] TableEnvironmentImpl doesn't clear…
flinkbot edited a comment on issue #11317: [FLINK-16433][table-api] TableEnvironmentImpl doesn't clear… URL: https://github.com/apache/flink/pull/11317#issuecomment-595145709 ## CI report: * 9c6d3fcae177e05bba552a967d0cf75587412693 Travis: [CANCELED](https://travis-ci.com/flink-ci/flink/builds/151929604) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5960) * 6cb26ab23d18cd85dda0501861873fa9382332ff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11319: [FLINK-16376][yarn] Use consistent method to get Yarn application dir…
flinkbot edited a comment on issue #11319: [FLINK-16376][yarn] Use consistent method to get Yarn application dir… URL: https://github.com/apache/flink/pull/11319#issuecomment-595166037 ## CI report: * c2f13248b87d59271f7fd14b2c5a1f9dc6c72a89 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151937563) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5964) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner
flinkbot edited a comment on issue #11252: [FLINK-16337][python][table-planner][table-planner-blink] Add support of vectorized Python UDF in blink planner and old planner URL: https://github.com/apache/flink/pull/11252#issuecomment-592511050 ## CI report: * 542c6f80de4976c204a68968f6d5af383dad1ac5 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151659712) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5890) * 43041585ccc2dd381baf70240aca178fef9c7b93 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11220: [FLINK-16249][python][ml] Add interfaces for Params, ParamInfo and WithParams
flinkbot edited a comment on issue #11220: [FLINK-16249][python][ml] Add interfaces for Params, ParamInfo and WithParams URL: https://github.com/apache/flink/pull/11220#issuecomment-591333787 ## CI report: * bc6799fbd1e26ba48a902527d870b95d796aab10 UNKNOWN * faf604e0b64fca0c8862e4fbacc5dad9acebb46d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151911284) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5957) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11245: [FLINK-15794][Kubernetes] Generate the Kubernetes default image version
flinkbot edited a comment on issue #11245: [FLINK-15794][Kubernetes] Generate the Kubernetes default image version URL: https://github.com/apache/flink/pull/11245#issuecomment-592002512 ## CI report: * f1d4f9163a2cd60daa02fe6b7af5a459cee9660c UNKNOWN * 607a75b896ddd933027d52e2b994ed8f0a6c5e69 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151914534) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5958) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052067#comment-17052067
]
Zili Chen commented on FLINK-15791:
---
[~fly_in_gis] Yes it is the case.
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052057#comment-17052057
]
Yang Wang edited comment on FLINK-15791 at 3/5/20, 12:01 PM:
-
[~trohrmann], since the FLINK-16194(Refactor the Kubernetes decorator design)
has remove the action watcher by design. So we do not have asynchronous
operations in the {{Fabric8FlinkKubeClient}} now. I will close this PR and
moving the explicit I/O Executor implementation to FLINK-15790(Make
FlinkKubeClient and its implementations asynchronous).
was (Author: fly_in_gis):
[~trohrmann], since the FLINK-16194(Refactor the Kubernetes decorator design)
has remove the action watcher by design. I will close this PR and moving the
explicit I/O Executor implementation to FLINK-15790(Make FlinkKubeClient and
its implementations asynchronous).
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 closed pull request #10971: [FLINK-15791][k8s] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient
wangyang0918 closed pull request #10971: [FLINK-15791][k8s] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient URL: https://github.com/apache/flink/pull/10971 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052057#comment-17052057
]
Yang Wang edited comment on FLINK-15791 at 3/5/20, 11:58 AM:
-
[~trohrmann], since the FLINK-16194(Refactor the Kubernetes decorator design)
has remove the action watcher by design. I will close this PR and moving the
explicit I/O Executor implementation to FLINK-15790(Make FlinkKubeClient and
its implementations asynchronous).
was (Author: fly_in_gis):
[~trohrmann], since the #FLINK-16194(Refactor the Kubernetes decorator design)
has remove the action watcher by design. I will close this PR and moving the
explicit I/O Executor implementation to #FLINK-15790(Make FlinkKubeClient and
its implementations asynchronous).
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15791) Don't use ForkJoinPool#commonPool() for executing asynchronous operations in Fabric8FlinkKubeClient
[
https://issues.apache.org/jira/browse/FLINK-15791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17052057#comment-17052057
]
Yang Wang commented on FLINK-15791:
---
[~trohrmann], since the #FLINK-16194(Refactor the Kubernetes decorator design)
has remove the action watcher by design. I will close this PR and moving the
explicit I/O Executor implementation to #FLINK-15790(Make FlinkKubeClient and
its implementations asynchronous).
> Don't use ForkJoinPool#commonPool() for executing asynchronous operations in
> Fabric8FlinkKubeClient
> ---
>
> Key: FLINK-15791
> URL: https://issues.apache.org/jira/browse/FLINK-15791
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should not use the {{ForkJoinPool#commonPool()}} in order to run
> asynchronous operations in the {{Fabric8FlinkKubeClient}} as it is done
> [here|https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java#L315].
> Since we don't know which other component is using this pool, it can be
> quite dangerous to use it as there might be congestion.
> Instead, I propose to provide an explicit I/O {{Executor}} which is used for
> running asynchronous operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on issue #10971: [FLINK-15791][k8s] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient
wangyang0918 commented on issue #10971: [FLINK-15791][k8s] Use explicit I/O Executor for asynchronous operations in Fabric8FlinkKubeClient URL: https://github.com/apache/flink/pull/10971#issuecomment-595192579 @tillrohrmann @TisonKun Thanks a lot for your suggestion. Since the #11233(Refactor the Kubernetes decorator design) has remove the action watcher. I will close this PR and moving the explicit I/O Executor to [FLINK-15790](https://issues.apache.org/jira/browse/FLINK-15790)(Make FlinkKubeClient and its implementations asynchronous). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe commented on issue #11296: [FLINK-16363] [table] Correct the execution behavior of TableEnvironment and StreamTableEnvironment
godfreyhe commented on issue #11296: [FLINK-16363] [table] Correct the execution behavior of TableEnvironment and StreamTableEnvironment URL: https://github.com/apache/flink/pull/11296#issuecomment-595191381 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-16108) StreamSQLExample is failed if running in blink planner
[
https://issues.apache.org/jira/browse/FLINK-16108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu resolved FLINK-16108.
-
Resolution: Fixed
Fixed in
- mater(1.11.0): 377024ba85058e3ba5fa092aaf5c92be33d7ae09
- 1.10.1: 9685642da56554eb5d0292f3ffe193c48329d423
> StreamSQLExample is failed if running in blink planner
> --
>
> Key: FLINK-16108
> URL: https://issues.apache.org/jira/browse/FLINK-16108
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Critical
> Fix For: 1.10.1, 1.11.0
>
>
> {{StreamSQLExample}} in flink-example will fail if the specified planner is
> blink planner. Exception is as following:
> {code}
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Field types of query result and registered TableSink do not match.
> Query schema: [user: BIGINT, product: STRING, amount: INT]
> Sink schema: [amount: INT, product: STRING, user: BIGINT]
> at
> org.apache.flink.table.planner.sinks.TableSinkUtils$.validateSchemaAndApplyImplicitCast(TableSinkUtils.scala:96)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translateToRel(PlannerBase.scala:229)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:150)
> at
> org.apache.flink.table.planner.delegation.PlannerBase$$anonfun$1.apply(PlannerBase.scala:150)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:150)
> at
> org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.toDataStream(StreamTableEnvironmentImpl.java:361)
> at
> org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.toAppendStream(StreamTableEnvironmentImpl.java:269)
> at
> org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.toAppendStream(StreamTableEnvironmentImpl.java:260)
> at
> org.apache.flink.table.examples.java.StreamSQLExample.main(StreamSQLExample.java:90)
> Process finished with exit code 1
> {code}
> That's because blink planner will also validate the sink schema even if it is
> come from {{toAppendStream()}}. However, the
> {{TableSinkUtils#inferSinkPhysicalDataType}} should derive sink schema from
> query schema when the requested type is POJO [1], because fields order of
> POJO is not deterministic.
> [1]:
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/sinks/TableSinkUtils.scala#L237
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-16269) Generic type can not be matched when convert table to stream.
[
https://issues.apache.org/jira/browse/FLINK-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu resolved FLINK-16269.
-
Resolution: Fixed
Fixed in
- mater(1.11.0): 377024ba85058e3ba5fa092aaf5c92be33d7ae09
- 1.10.1: 9685642da56554eb5d0292f3ffe193c48329d423
> Generic type can not be matched when convert table to stream.
> -
>
> Key: FLINK-16269
> URL: https://issues.apache.org/jira/browse/FLINK-16269
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: xiemeilong
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.1, 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The query result schema printed by table.printSchema():
> {noformat}
> |-- deviceId: BIGINT
> |-- channel: STRING
> |-- schemaId: BIGINT
> |-- productId: BIGINT
> |-- schema: LEGACY('RAW', 'ANY')
> {noformat}
> then excuting table.toRetractStream[DeviceSchema].print(), exception throwed:
> {noformat}
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Field types of query result and registered TableSink do not match.
> Query schema: [deviceId: BIGINT, channel: STRING, schemaId: BIGINT,
> productId: BIGINT, schema: RAW('com.yunmo.iot.schema.Schema', ?)]
> Sink schema: [deviceId: BIGINT, channel: STRING, schemaId: BIGINT,
> productId: BIGINT, schema: LEGACY('RAW',
> 'ANY')]{noformat}
> The com.yunmo.iot.schema.Schema is a generic type.
> The schema field of Query schema change from LEGACY('RAW' to RAW, but the
> Sink schema still a LEGACY('RAW'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #11236: [FLINK-16269][FLINK-16108][table-planner-blink] Fix schema of query and sink do not match when generic or POJO type is requested
wuchong merged pull request #11236: [FLINK-16269][FLINK-16108][table-planner-blink] Fix schema of query and sink do not match when generic or POJO type is requested URL: https://github.com/apache/flink/pull/11236 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16400) HdfsKindTest.testS3Kind fails in Hadoop 2.4.1 nightly test
[
https://issues.apache.org/jira/browse/FLINK-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-16400:
---
Release Note: `org.apache.flink.core.fs.FileSystem#getKind` method has been
formally deprecated, as it was not used by Flink. (was:
`org.apache.flink.core.fs.FileSystem#getKind` method has been officially
deprecated, as it was not used by Flink.)
> HdfsKindTest.testS3Kind fails in Hadoop 2.4.1 nightly test
> --
>
> Key: FLINK-16400
> URL: https://issues.apache.org/jira/browse/FLINK-16400
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Reporter: Robert Metzger
>Assignee: Arvid Heise
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Log:
> [https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5843=logs=f8cdcc9b-111a-5332-0026-209cb3eb5d15=57d35dc9-027e-5d4a-fbeb-1c24315e6ffb]
> and: [https://travis-ci.org/apache/flink/jobs/657296261]
> {code:java}
> 15:57:21.539 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 0.291 s <<< FAILURE! - in
> org.apache.flink.runtime.fs.hdfs.HdfsKindTest
> 15:57:21.552 [ERROR]
> testS3Kind(org.apache.flink.runtime.fs.hdfs.HdfsKindTest) Time elapsed:
> 0.032 s <<< ERROR!
> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find
> a file system implementation for scheme 's3'. The scheme is directly
> supported by Flink through the following plugins: flink-s3-fs-hadoop,
> flink-s3-fs-presto. Please ensure that each plugin resides within its own
> subfolder within the plugins directory. See
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for
> more information. If you want to use a Hadoop file system for that scheme,
> please add the scheme to the configuration fs.allowed-fallback-filesystems.
> For a full list of supported file systems, please see
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
> at
> org.apache.flink.runtime.fs.hdfs.HdfsKindTest.testS3Kind(HdfsKindTest.java:57)
> 15:57:21.574 [INFO] Running
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriterOldHadoopWithNoTruncateSupportTest
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-16400) HdfsKindTest.testS3Kind fails in Hadoop 2.4.1 nightly test
[
https://issues.apache.org/jira/browse/FLINK-16400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-16400.
--
Fix Version/s: 1.11.0
Release Note: `org.apache.flink.core.fs.FileSystem#getKind` method has
been officially deprecated, as it was not used by Flink.
Resolution: Fixed
Merged as be0eb54777 3e66fcfd25
> HdfsKindTest.testS3Kind fails in Hadoop 2.4.1 nightly test
> --
>
> Key: FLINK-16400
> URL: https://issues.apache.org/jira/browse/FLINK-16400
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Reporter: Robert Metzger
>Assignee: Arvid Heise
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Log:
> [https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5843=logs=f8cdcc9b-111a-5332-0026-209cb3eb5d15=57d35dc9-027e-5d4a-fbeb-1c24315e6ffb]
> and: [https://travis-ci.org/apache/flink/jobs/657296261]
> {code:java}
> 15:57:21.539 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 0.291 s <<< FAILURE! - in
> org.apache.flink.runtime.fs.hdfs.HdfsKindTest
> 15:57:21.552 [ERROR]
> testS3Kind(org.apache.flink.runtime.fs.hdfs.HdfsKindTest) Time elapsed:
> 0.032 s <<< ERROR!
> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find
> a file system implementation for scheme 's3'. The scheme is directly
> supported by Flink through the following plugins: flink-s3-fs-hadoop,
> flink-s3-fs-presto. Please ensure that each plugin resides within its own
> subfolder within the plugins directory. See
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for
> more information. If you want to use a Hadoop file system for that scheme,
> please add the scheme to the configuration fs.allowed-fallback-filesystems.
> For a full list of supported file systems, please see
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
> at
> org.apache.flink.runtime.fs.hdfs.HdfsKindTest.testS3Kind(HdfsKindTest.java:57)
> 15:57:21.574 [INFO] Running
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriterOldHadoopWithNoTruncateSupportTest
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
