[GitHub] [flink] flinkbot commented on issue #11405: [FLINK-16413]Reduce hive source parallelism when limit push down
flinkbot commented on issue #11405: [FLINK-16413]Reduce hive source parallelism when limit push down URL: https://github.com/apache/flink/pull/11405#issuecomment-599020246 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b196447b93a3c1bdb64fce172c2aea305e16093b (Sat Mar 14 06:54:16 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhangjun888 opened a new pull request #11405: [FLINK-16413]Reduce hive source parallelism when limit push down
zhangjun888 opened a new pull request #11405: [FLINK-16413]Reduce hive source parallelism when limit push down URL: https://github.com/apache/flink/pull/11405 ## What is the purpose of the change Reduce hive source parallelism when limit push down ## Brief change log when limit push down ,set the parallelism to min(parallelism,limit) ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector:no ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16254) Support -p/--parallelism option for StatefulFunctionsClusterEntryPoint
[
https://issues.apache.org/jira/browse/FLINK-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16254:
---
Labels: pull-request-available (was: )
> Support -p/--parallelism option for StatefulFunctionsClusterEntryPoint
> --
>
> Key: FLINK-16254
> URL: https://issues.apache.org/jira/browse/FLINK-16254
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
>Affects Versions: statefun-1.1
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Labels: pull-request-available
>
> Currently the only way for users to specify parallelism > 1 for Stateful
> Functions applications is to provide a value for {{parallelism.default}} via
> {{flink-conf.yaml}}.
> That is not so nice to use, as users would essentially need to rebuild the
> Stateful Functions application image just to change the parallelism.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai opened a new pull request #58: [FLINK-16254] Support -p / --parallelism command option
tzulitai opened a new pull request #58: [FLINK-16254] Support -p / --parallelism command option URL: https://github.com/apache/flink-statefun/pull/58 This PR adds support for specifying parallelism for Stateful Function applications using the `-p / --parallelism` command. The new behaviour as as follows: - If `-p` is defined in the command line, then that value is always used as the parallelism. - Otherwise, the `parallelism.default` value in `flink-conf.yaml` is used, which by default is 1 if not present. --- ### Changelog - 27e7abc Adds parsing logic of the parallelism option from the command line, and adds the property to `StatefulFunctionsClusterConfiguration`. - 07c8dd2 Resolves the parallelism to use in `StatefulFunctionsJobGraphRetriever`, and uses the resolved value to create the JobGraph. - ee61047 Adapt the E2E tests, so that they use the `-p` command to specify parallelism. --- ### Verfying The changes to the E2E tests in ee61047 verifies this change: `mvn clean verify -Prun-e2e-tests` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16591) Flink-zh Doc show a wrong Email address
[ https://issues.apache.org/jira/browse/FLINK-16591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059181#comment-17059181 ] Zili Chen commented on FLINK-16591: --- Yes of course! Please mark the issue as "In Progress" by clicking "Start Progress" button above when you start progress. > Flink-zh Doc show a wrong Email address > --- > > Key: FLINK-16591 > URL: https://issues.apache.org/jira/browse/FLINK-16591 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: forideal >Assignee: forideal >Priority: Minor > > The Link is https://flink.apache.org/zh/community.html > 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, > user, user-zh 等等） > The right Email pattern is listname-subscr...@flink.apache.org. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16591) Flink-zh Doc show a wrong Email address
[ https://issues.apache.org/jira/browse/FLINK-16591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-16591: - Assignee: forideal > Flink-zh Doc show a wrong Email address > --- > > Key: FLINK-16591 > URL: https://issues.apache.org/jira/browse/FLINK-16591 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: forideal >Assignee: forideal >Priority: Minor > > The Link is https://flink.apache.org/zh/community.html > 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, > user, user-zh 等等） > The right Email pattern is listname-subscr...@flink.apache.org. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16592) The doc of Streaming File Sink has a mistake of grammar
Chen created FLINK-16592:
Summary: The doc of Streaming File Sink has a mistake of grammar
Key: FLINK-16592
URL: https://issues.apache.org/jira/browse/FLINK-16592
Project: Flink
Issue Type: Task
Components: Documentation
Reporter: Chen
Attachments: image-2020-03-14-12-04-27-065.png
In the following link, there is a mistake in grammar.
[https://ci.apache.org/projects/flink/flink-docs-release-1.10/zh/dev/connectors/streamfile_sink.html]
!image-2020-03-14-12-04-27-065.png|width=567,height=369!
it should be
{{}}
{code:java}
final StreamingFileSink sink = StreamingFileSink
.forRowFormat(new Path(outputPath), new SimpleStringEncoder("UTF-8"))
.withRollingPolicy(
DefaultRollingPolicy.builder()
.withRolloverInterval(TimeUnit.MINUTES.toMillis(15))
.withInactivityInterval(TimeUnit.MINUTES.toMillis(5))
.withMaxPartSize(1024 * 1024 * 1024)
.build())
.build();
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16591) Flink-zh Doc show a wrong Email address
[ https://issues.apache.org/jira/browse/FLINK-16591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] forideal updated FLINK-16591: - Description: The Link is https://flink.apache.org/zh/community.html 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, user, user-zh 等等） The right Email pattern is listname-subscr...@flink.apache.org. was: 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, user, user-zh 等等） The right Email pattern is listname-subscr...@flink.apache.org. > Flink-zh Doc show a wrong Email address > --- > > Key: FLINK-16591 > URL: https://issues.apache.org/jira/browse/FLINK-16591 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: forideal >Priority: Minor > > The Link is https://flink.apache.org/zh/community.html > 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, > user, user-zh 等等） > The right Email pattern is listname-subscr...@flink.apache.org. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16591) Flink-zh Doc show a wrong Email address
[ https://issues.apache.org/jira/browse/FLINK-16591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059175#comment-17059175 ] forideal commented on FLINK-16591: -- Hi [~tison] Can you assign this issue to me ? Thank you！ > Flink-zh Doc show a wrong Email address > --- > > Key: FLINK-16591 > URL: https://issues.apache.org/jira/browse/FLINK-16591 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: forideal >Priority: Minor > > 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, > user, user-zh 等等） > The right Email pattern is listname-subscr...@flink.apache.org. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16591) Flink-zh Doc‘s show a wrong Email address
forideal created FLINK-16591: Summary: Flink-zh Doc‘s show a wrong Email address Key: FLINK-16591 URL: https://issues.apache.org/jira/browse/FLINK-16591 Project: Flink Issue Type: Bug Components: Documentation Reporter: forideal 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, user, user-zh 等等） The right Email pattern is listname-subscr...@flink.apache.org. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16591) Flink-zh Doc show a wrong Email address
[ https://issues.apache.org/jira/browse/FLINK-16591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] forideal updated FLINK-16591: - Summary: Flink-zh Doc show a wrong Email address (was: Flink-zh Doc‘s show a wrong Email address) > Flink-zh Doc show a wrong Email address > --- > > Key: FLINK-16591 > URL: https://issues.apache.org/jira/browse/FLINK-16591 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: forideal >Priority: Minor > > 发送一封不包含任何内容或主题的邮件到 subscribe-listn...@flink.apache.org（替换 listname 为 dev, > user, user-zh 等等） > The right Email pattern is listname-subscr...@flink.apache.org. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16379) Introduce fromValues in TableEnvironment
[
https://issues.apache.org/jira/browse/FLINK-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059150#comment-17059150
]
Zhenghua Gao commented on FLINK-16379:
--
+1 to merge fromElements to fromValues.
Construction empty table with Values may be useless because plan with empty
LogicalValues could be optimized.
> Introduce fromValues in TableEnvironment
>
>
> Key: FLINK-16379
> URL: https://issues.apache.org/jira/browse/FLINK-16379
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Introduce a fromValues method to TableEnvironment similar to {{VALUES}}
> clause in SQL
> The suggested API could look like:
> {code}
> /**
>* Creates a Table from a given row constructing expressions.
>*
>* Examples:
>*
>* You can use {@link Expressions#row(Object, Object...)} to create
> a composite rows:
>* {@code
>* tEnv.fromValues(
>* row(1, "ABC"),
>* row(2L, "ABCDE")
>* )
>* }
>* will produce a Table with a schema as follows:
>* {@code
>* root
>* |-- f0: BIGINT NOT NULL
>* |-- f1: VARCHAR(5) NOT NULL
>* }
>*
>* ROWs that are a result of e.g. a function call are not flattened
>* {@code
>* public class RowFunction extends ScalarFunction {
>* @DataTypeHint("ROW")
>* Row eval();
>* }
>*
>* tEnv.fromValues(
>* call(new RowFunction()),
>* call(new RowFunction())
>* )
>* }
>* will produce a Table with a schema as follows:
>* {@code
>* root
>* |-- f0: ROW<`f0` BIGINT, `f1` VARCHAR(5)>
>* }
>*
>* The row constructor can be dropped to create a table with a
> single row:
>*
>* ROWs that are a result of e.g. a function call are not flattened
>* {@code
>* tEnv.fromValues(
>* 1,
>* 2L,
>* 3
>* )
>* }
>* will produce a Table with a schema as follows:
>* {@code
>* root
>* |-- f0: BIGINT NOT NULL
>* }
>*
>* @param expressions Expressions for constructing rows of the VALUES
> table.
>*/
> Table fromValues(Expression... expressions);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16583) SQLClientKafkaITCase.testKafka failed with SqlClientException
[
https://issues.apache.org/jira/browse/FLINK-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059148#comment-17059148
]
Hequn Cheng commented on FLINK-16583:
-
Another instance: https://api.travis-ci.org/v3/job/661966609/log.txt
> SQLClientKafkaITCase.testKafka failed with SqlClientException
> -
>
> Key: FLINK-16583
> URL: https://issues.apache.org/jira/browse/FLINK-16583
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Client, Tests
>Reporter: Zhijiang
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> The end-to-end test {{SQLClientKafkaITCase.testKafka}} failed with
> {code:java}
> 18:13:02.425 [ERROR] testKafka[0: kafka-version:0.10
> kafka-sql-version:.*kafka-0.10.jar](org.apache.flink.tests.util.kafka.SQLClientKafkaITCase)
> Time elapsed: 32.246 s <<< ERROR!
> java.io.IOException:
> Process execution failed due error. Error output:Mar 12, 2020 6:11:46 PM
> org.jline.utils.Log logr
> WARNING: Unable to create a system terminal, creating a dumb terminal (enable
> debug logging for more information)
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Could not submit given SQL update statement to cluster.
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:131)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
> at
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.insertIntoAvroTable(SQLClientKafkaITCase.java:178)
> at
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.testKafka(SQLClientKafkaITCase.java:151)
> 18:13:02.425 [ERROR] testKafka[1: kafka-version:0.11
> kafka-sql-version:.*kafka-0.11.jar](org.apache.flink.tests.util.kafka.SQLClientKafkaITCase)
> Time elapsed: 34.539 s <<< ERROR!
> java.io.IOException:
> Process execution failed due error. Error output:Mar 12, 2020 6:12:21 PM
> org.jline.utils.Log logr
> WARNING: Unable to create a system terminal, creating a dumb terminal (enable
> debug logging for more information)
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Could not submit given SQL update statement to cluster.
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:131)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
> at
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.insertIntoAvroTable(SQLClientKafkaITCase.java:178)
> at
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.testKafka(SQLClientKafkaITCase.java:151)
> {code}
> [https://api.travis-ci.org/v3/job/661535183/log.txt]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhangjun888 closed pull request #11398: [FLINK-16413]Reduce hive source parallelism when limit push down
zhangjun888 closed pull request #11398: [FLINK-16413]Reduce hive source parallelism when limit push down URL: https://github.com/apache/flink/pull/11398 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-16392) oneside sorted cache in intervaljoin
[ https://issues.apache.org/jira/browse/FLINK-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Qin closed FLINK-16392. Resolution: Feedback Received > oneside sorted cache in intervaljoin > > > Key: FLINK-16392 > URL: https://issues.apache.org/jira/browse/FLINK-16392 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.10.0 >Reporter: Chen Qin >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Native intervaljoin rely on statebackend(e.g rocksdb) to insert/fetch left > and right buffer. This design choice reduce minimize heap memory footprint > while bounded process throughput of single taskmanager iops to rocksdb access > speed. Here at Pinterest, we have some large use cases where developers join > large and slow evolving data stream (e.g post updates in last 28 days) with > web traffic datastream (e.g post views up to 28 days after given update). > This post some challenge to current implementation of intervaljoin > * partitioned rocksdb needs to keep both updates and views for 28 days, > large buffer(especially view stream side) cause rocksdb slow down and lead to > overall interval join performance degregate quickly as state build up. > * view stream is web scale, even after setting large parallelism it can put > lot of pressure on each subtask and backpressure entire job > In proposed implementation, we plan to introduce two changes > * support ProcessJoinFunction settings to opt-in earlier cleanup time of > right stream(e.g view stream don't have to stay in buffer for 28 days and > wait for update stream to join, related post views happens after update in > event time semantic) This optimization can reduce state size to improve > rocksdb throughput. If extreme case, user can opt-in in flight join and skip > write into right view stream buffer to save iops budget on each subtask > * support ProcessJoinFunction settings to expedite keyed lookup of slow > changing stream. Instead of every post view pull post updates from rocksdb. > user can opt-in and having one side buffer cache available in memory. If a > given post update, cache load recent views from right buffer and use > sortedMap to find buckets. If a given post view, cache load recent updates > from left buffer to memory. When another view for that post arrives, flink > save cost of rocksdb access. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16575) develop HBaseCatalog to integrate HBase metadata into Flink
[ https://issues.apache.org/jira/browse/FLINK-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-16575: - Description: develop HBaseCatalog to integrate HBase metadata into Flink The ticket includes necessary initial investigation to see if it's possible and brings practical value, since hbase/elasticsearch are schemaless. If it is valuable, then partition/function/stats/views probably shouldn't be implemented, which would be very similar to PostgresCatalog ([https://github.com/apache/flink/pull/11336]). HiveCatalog can also be a good reference. was:develop HBaseCatalog to integrate HBase metadata into Flink > develop HBaseCatalog to integrate HBase metadata into Flink > --- > > Key: FLINK-16575 > URL: https://issues.apache.org/jira/browse/FLINK-16575 > Project: Flink > Issue Type: New Feature > Components: Connectors / HBase >Reporter: Bowen Li >Priority: Major > > develop HBaseCatalog to integrate HBase metadata into Flink > The ticket includes necessary initial investigation to see if it's possible > and brings practical value, since hbase/elasticsearch are schemaless. > > If it is valuable, then partition/function/stats/views probably shouldn't be > implemented, which would be very similar to PostgresCatalog > ([https://github.com/apache/flink/pull/11336]). HiveCatalog can also be a > good reference. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16575) develop HBaseCatalog to integrate HBase metadata into Flink
[ https://issues.apache.org/jira/browse/FLINK-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058974#comment-17058974 ] Bowen Li commented on FLINK-16575: -- this ticket would involve initial research to see if it is feasible and has practical value > develop HBaseCatalog to integrate HBase metadata into Flink > --- > > Key: FLINK-16575 > URL: https://issues.apache.org/jira/browse/FLINK-16575 > Project: Flink > Issue Type: New Feature > Components: Connectors / HBase >Reporter: Bowen Li >Priority: Major > > develop HBaseCatalog to integrate HBase metadata into Flink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve
error reporting on AZP
URL: https://github.com/apache/flink/pull/11372#discussion_r392371252
##
File path: tools/travis/common-logging.sh
##
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Maximum times to retry uploading artifacts file to transfer.sh
+TRANSFER_UPLOAD_MAX_RETRIES=2
+
+# The delay between two retries to upload artifacts file to transfer.sh. The
default exponential
+# backoff algorithm should be too long for the last several retries.
+TRANSFER_UPLOAD_RETRY_DELAY=5
+
+# E.g. travis-artifacts/apache/flink/1595/1595.1
+UPLOAD_TARGET_PATH="travis-artifacts/${TRAVIS_REPO_SLUG}/${TRAVIS_BUILD_NUMBER}/"
+# These variables are stored as secure variables in '.travis.yml', which are
generated per repo via
+# the travis command line tool.
+UPLOAD_BUCKET=$ARTIFACTS_AWS_BUCKET
+UPLOAD_ACCESS_KEY=$ARTIFACTS_AWS_ACCESS_KEY
+UPLOAD_SECRET_KEY=$ARTIFACTS_AWS_SECRET_KEY
+
+
+SCRIPT_DIR="`dirname \"$0\"`"
+SCRIPT_DIR="`( cd \"${SCRIPT_DIR}\" && pwd -P)`"
+export FLINK_ROOT="`( cd \"${SCRIPT_DIR}/..\" && pwd -P)`"
Review comment:
this seems quite brittle
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve
error reporting on AZP
URL: https://github.com/apache/flink/pull/11372#discussion_r392367738
##
File path: tools/azure_controller.sh
##
@@ -53,6 +51,8 @@ print_system_info() {
print_system_info
+# enable core dumps
+sudo ulimit -c unlimited
Review comment:
what does this have to do with core dumps?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP URL: https://github.com/apache/flink/pull/11372#discussion_r392367973 ## File path: tools/azure_controller.sh ## @@ -24,8 +24,6 @@ echo $MAVEN_OPTS mvn -version echo "Commit: $(git rev-parse HEAD)" - - Review comment: unrelated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve
error reporting on AZP
URL: https://github.com/apache/flink/pull/11372#discussion_r392367035
##
File path: tools/travis_watchdog.sh
##
@@ -177,6 +177,14 @@ print_stacktraces () {
done
}
+collect_coredumps() {
+ echo "Searching for .dump, .dumpstream and related files in
$($HERE/../)"
+ for file in `find . -type f -regextype posix-extended -iregex
'.*\.dump|.*\.dumpstream|.*hs.*\.log|.*/core(.[0-9]+)?$'`; do
Review comment:
a simpler solution may be to setup the JVM to directly write the dumps into
`ARTIFACTS_DIR`.
https://stackoverflow.com/questions/1880166/is-it-possible-to-specify-where-jvms-crash-dumps-go
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve
error reporting on AZP
URL: https://github.com/apache/flink/pull/11372#discussion_r392364911
##
File path: flink-end-to-end-tests/test-scripts/test-runner-common.sh
##
@@ -46,7 +46,7 @@ function run_test {
function test_error() {
echo "[FAIL] Test script contains errors."
- post_test_validation 1 "$description" "$skip_check_exceptions"
+ post_test_validation 1 "$description"
Review comment:
My impressions is that this will actually obfuscate errors. People pretty
much only look at the last ~20 lines of the CI output, and don't properly
investigate the case, resulting in several superfluous JIRAs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598702521 ## CI report: * c1facefebee52a1ac84f188f46b114deae0aad89 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153207095) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve
error reporting on AZP
URL: https://github.com/apache/flink/pull/11372#discussion_r392338175
##
File path: tools/travis/common-logging.sh
##
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Maximum times to retry uploading artifacts file to transfer.sh
+TRANSFER_UPLOAD_MAX_RETRIES=2
+
+# The delay between two retries to upload artifacts file to transfer.sh. The
default exponential
+# backoff algorithm should be too long for the last several retries.
+TRANSFER_UPLOAD_RETRY_DELAY=5
+
+# E.g. travis-artifacts/apache/flink/1595/1595.1
+UPLOAD_TARGET_PATH="travis-artifacts/${TRAVIS_REPO_SLUG}/${TRAVIS_BUILD_NUMBER}/"
+# These variables are stored as secure variables in '.travis.yml', which are
generated per repo via
+# the travis command line tool.
+UPLOAD_BUCKET=$ARTIFACTS_AWS_BUCKET
+UPLOAD_ACCESS_KEY=$ARTIFACTS_AWS_ACCESS_KEY
+UPLOAD_SECRET_KEY=$ARTIFACTS_AWS_SECRET_KEY
+
+
+SCRIPT_DIR="`dirname \"$0\"`"
+SCRIPT_DIR="`( cd \"${SCRIPT_DIR}\" && pwd -P)`"
+export FLINK_ROOT="`( cd \"${SCRIPT_DIR}/..\" && pwd -P)`"
+if [ -z "${FLINK_ROOT}" ] ; then
+ # error; for some reason, the path is not accessible
+ # to the script (e.g. permissions re-evaled after suid)
+ exit 1 # fail
+fi
+
+prepare_artifacts() {
+ export ARTIFACTS_DIR="${SCRIPT_DIR}/artifacts"
+
+ mkdir -p $ARTIFACTS_DIR || { echo "FAILURE: cannot create log directory
'${ARTIFACTS_DIR}'." ; exit 1; }
+}
+
+upload_artifacts() {
+ ARTIFACTS_FILE=${TRAVIS_JOB_NUMBER}.tar.gz
+ if [ ! -z "$TF_BUILD" ] ; then
+ # set proper artifacts file name on Azure Pipelines
+ ARTIFACTS_FILE=${BUILD_BUILDNUMBER}.tar.gz
+ if [ ! -z "$MODULE" ] ; then
+ ARTIFACTS_FILE=${BUILD_BUILDNUMBER}-$(echo $MODULE | tr
-dc '[:alnum:]\n\r').tar.gz
+ fi
+ fi
+
+ echo "PRODUCED build artifacts."
Review comment:
as usual this would be easier to review if the moving of code were put into
a separate commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11372: [FLINK-16480] Improve error reporting on AZP
zentol commented on a change in pull request #11372: [FLINK-16480] Improve
error reporting on AZP
URL: https://github.com/apache/flink/pull/11372#discussion_r392337003
##
File path: flink-end-to-end-tests/test-scripts/test-runner-common.sh
##
@@ -46,7 +46,7 @@ function run_test {
function test_error() {
echo "[FAIL] Test script contains errors."
- post_test_validation 1 "$description" "$skip_check_exceptions"
+ post_test_validation 1 "$description"
Review comment:
do you know whether the bash tests upload the logs for failed tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (FLINK-14121) Upgrade commons-compress to 1.20 due to CVE
[ https://issues.apache.org/jira/browse/FLINK-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao closed FLINK-14121. Resolution: Fixed master: 0477368813fba33f0f9aa5ff82304516c499ed41 > Upgrade commons-compress to 1.20 due to CVE > --- > > Key: FLINK-14121 > URL: https://issues.apache.org/jira/browse/FLINK-14121 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Build System, Release System >Affects Versions: 1.9.0 >Reporter: John Lonergan >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > See > https://commons.apache.org/proper/commons-compress/security-reports.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14121) Upgrade commons-compress to 1.20 due to CVE
[ https://issues.apache.org/jira/browse/FLINK-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-14121: - Summary: Upgrade commons-compress to 1.20 due to CVE (was: Upgrade commons-compress to 1.19 due to CVE) > Upgrade commons-compress to 1.20 due to CVE > --- > > Key: FLINK-14121 > URL: https://issues.apache.org/jira/browse/FLINK-14121 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Build System, Release System >Affects Versions: 1.9.0 >Reporter: John Lonergan >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > See > https://commons.apache.org/proper/commons-compress/security-reports.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions
flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions URL: https://github.com/apache/flink/pull/11253#issuecomment-592517598 ## CI report: * 4e1c4f61f4e477262eb3d1d4fa67c97bcec1b85c Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153201488) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14121) Upgrade commons-compress to 1.19 due to CVE
[ https://issues.apache.org/jira/browse/FLINK-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Yao updated FLINK-14121: - Fix Version/s: 1.11.0 > Upgrade commons-compress to 1.19 due to CVE > --- > > Key: FLINK-14121 > URL: https://issues.apache.org/jira/browse/FLINK-14121 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Build System, Release System >Affects Versions: 1.9.0 >Reporter: John Lonergan >Assignee: Niels Basjes >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > See > https://commons.apache.org/proper/commons-compress/security-reports.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] GJL closed pull request #11333: [FLINK-14121] Update commons-compress because of CVE-2019-12402
GJL closed pull request #11333: [FLINK-14121] Update commons-compress because of CVE-2019-12402 URL: https://github.com/apache/flink/pull/11333 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-16553) Missing KafkaFetcher topic/partition metrics
[ https://issues.apache.org/jira/browse/FLINK-16553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058893#comment-17058893 ] Tarush Grover edited comment on FLINK-16553 at 3/13/20, 4:24 PM: - [~aljoscha] I will come up with implementation document and share it with all. was (Author: app-tarush): [~aljoscha] I will come up with implementation document and share with all. > Missing KafkaFetcher topic/partition metrics > > > Key: FLINK-16553 > URL: https://issues.apache.org/jira/browse/FLINK-16553 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Runtime / Metrics >Reporter: Fabian Paul >Assignee: Tarush Grover >Priority: Major > > When using the Kafka universal connector, currently not all KafkaFetcher > metrics > ([link|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java]) > which are exposed through the KafkaConsumer are accessible within the Flink > metrics system. > Especially, all metrics which are related to topics and partitions are not > available. The KafkaConsumer internally only registers those metrics after it > has fetched some records. > Unfortunately, at the moment Flink only checks the available metrics right > after the initialization of the KafkaConsumer when no records are polled, yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16553) Missing KafkaFetcher topic/partition metrics
[ https://issues.apache.org/jira/browse/FLINK-16553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058893#comment-17058893 ] Tarush Grover commented on FLINK-16553: --- [~aljoscha] I will come up with implementation document and share with all. > Missing KafkaFetcher topic/partition metrics > > > Key: FLINK-16553 > URL: https://issues.apache.org/jira/browse/FLINK-16553 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka, Runtime / Metrics >Reporter: Fabian Paul >Assignee: Tarush Grover >Priority: Major > > When using the Kafka universal connector, currently not all KafkaFetcher > metrics > ([link|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java]) > which are exposed through the KafkaConsumer are accessible within the Flink > metrics system. > Especially, all metrics which are related to topics and partitions are not > available. The KafkaConsumer internally only registers those metrics after it > has fetched some records. > Unfortunately, at the moment Flink only checks the available metrics right > after the initialization of the KafkaConsumer when no records are polled, yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters
flinkbot edited a comment on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters URL: https://github.com/apache/flink/pull/11404#issuecomment-598746532 ## CI report: * a7f81066eb1d939e9d6c492895bd5e0eb021360b Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153198834) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16590) flink-oss-fs-hadoop: Not all dependencies in NOTICE file are bundled
[
https://issues.apache.org/jira/browse/FLINK-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao updated FLINK-16590:
-
Description: NOTICE file in flink-oss-fs-hadoop lists
{{org.apache.commons:commons-compress}} as a bundled dependency which is not
correct. There are likely other dependencies that are wrongly listed in the
NOTICE file. (was: NOTICE file in flink-oss-fs-hadoop lists
{{code}}
org.apache.commons:commons-compress
{{code}}
as a bundled dependency which is not correct. There are likely other
dependencies that are wrongly listed in the NOTICE file.)
> flink-oss-fs-hadoop: Not all dependencies in NOTICE file are bundled
>
>
> Key: FLINK-16590
> URL: https://issues.apache.org/jira/browse/FLINK-16590
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Release System
>Affects Versions: 1.11.0
>Reporter: Gary Yao
>Priority: Critical
> Fix For: 1.11.0
>
>
> NOTICE file in flink-oss-fs-hadoop lists
> {{org.apache.commons:commons-compress}} as a bundled dependency which is not
> correct. There are likely other dependencies that are wrongly listed in the
> NOTICE file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-16590) flink-oss-fs-hadoop: Not all dependencies in NOTICE file are bundled
Gary Yao created FLINK-16590:
Summary: flink-oss-fs-hadoop: Not all dependencies in NOTICE file
are bundled
Key: FLINK-16590
URL: https://issues.apache.org/jira/browse/FLINK-16590
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem, Release System
Affects Versions: 1.11.0
Reporter: Gary Yao
Fix For: 1.11.0
NOTICE file in flink-oss-fs-hadoop lists
{{code}}
org.apache.commons:commons-compress
{{code}}
as a bundled dependency which is not correct. There are likely other
dependencies that are wrongly listed in the NOTICE file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16408) Bind user code class loader to lifetime of a slot
[ https://issues.apache.org/jira/browse/FLINK-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058862#comment-17058862 ] Stephan Ewen commented on FLINK-16408: -- I think it should be okay to fix in a 1.10.x release, but want to double check. It may have some implications, for example statics not being re-initialized in restart, because the same classloader is being used. I would assume this to be okay, in fact, really helpful in most cases, but wanted to check if someone would see a concern with changing this behavior. > Bind user code class loader to lifetime of a slot > - > > Key: FLINK-16408 > URL: https://issues.apache.org/jira/browse/FLINK-16408 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.2, 1.10.0 >Reporter: Till Rohrmann >Priority: Critical > Fix For: 1.10.1, 1.11.0 > > > In order to avoid class leaks due to creating multiple user code class > loaders and loading class multiple times in a recovery case, I would suggest > to bind the lifetime of a user code class loader to the lifetime of a slot. > More precisely, the user code class loader should live at most as long as the > slot which is using it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16589) Flink Table SQL fails/crashes with big queries with lots of fields
[
https://issues.apache.org/jira/browse/FLINK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-16589:
Priority: Critical (was: Major)
> Flink Table SQL fails/crashes with big queries with lots of fields
> --
>
> Key: FLINK-16589
> URL: https://issues.apache.org/jira/browse/FLINK-16589
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Viet Pham
>Assignee: Benchao Li
>Priority: Critical
> Fix For: 1.10.1, 1.11.0
>
>
> Hi,
> My use case is a streaming application with a few streaming tables.
> I was trying to build a SELECT query (and registering it as a temporary view)
> with about 200 fields/expressions out of another streaming table. The
> application is successfully submitted to Flink cluster. However the worker
> processes keep crashing, with the exception as quoted below.
> It clearly mentioned in the log that this is a bug, so I fire this ticket. By
> the way, if I lower the number of fields down to 100 then it works nicely.
> Please advice.
> Thanks a lot for all the efforts bring Flink up. It is really amazing!
> {code:java}
> java.lang.RuntimeException: Could not instantiate generated class
> 'GroupAggsHandler$9687'at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)
> at
> org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)
> at
> org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 10 moreCaused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 12 moreCaused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException:
> Compiling "GroupAggsHandler$9687": Code of method
> "retract(Lorg/apache/flink/table/dataformat/BaseRow;)V" of class
> "GroupAggsHandler$9687" grows beyond 64 KBat
> org.codehaus.janino
[jira] [Updated] (FLINK-16589) Flink Table SQL fails/crashes with big queries with lots of fields
[
https://issues.apache.org/jira/browse/FLINK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-16589:
Fix Version/s: 1.11.0
1.10.1
> Flink Table SQL fails/crashes with big queries with lots of fields
> --
>
> Key: FLINK-16589
> URL: https://issues.apache.org/jira/browse/FLINK-16589
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Viet Pham
>Assignee: Benchao Li
>Priority: Major
> Fix For: 1.10.1, 1.11.0
>
>
> Hi,
> My use case is a streaming application with a few streaming tables.
> I was trying to build a SELECT query (and registering it as a temporary view)
> with about 200 fields/expressions out of another streaming table. The
> application is successfully submitted to Flink cluster. However the worker
> processes keep crashing, with the exception as quoted below.
> It clearly mentioned in the log that this is a bug, so I fire this ticket. By
> the way, if I lower the number of fields down to 100 then it works nicely.
> Please advice.
> Thanks a lot for all the efforts bring Flink up. It is really amazing!
> {code:java}
> java.lang.RuntimeException: Could not instantiate generated class
> 'GroupAggsHandler$9687'at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)
> at
> org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)
> at
> org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 10 moreCaused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 12 moreCaused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException:
> Compiling "GroupAggsHandler$9687": Code of method
> "retract(Lorg/apache/flink/table/dataformat/BaseRow;)V" of class
> "GroupAggsHandler$9687" grows beyond 64 KBat
> org.cod
[jira] [Assigned] (FLINK-16589) Flink Table SQL fails/crashes with big queries with lots of fields
[
https://issues.apache.org/jira/browse/FLINK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-16589:
---
Assignee: Benchao Li
> Flink Table SQL fails/crashes with big queries with lots of fields
> --
>
> Key: FLINK-16589
> URL: https://issues.apache.org/jira/browse/FLINK-16589
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Viet Pham
>Assignee: Benchao Li
>Priority: Major
>
> Hi,
> My use case is a streaming application with a few streaming tables.
> I was trying to build a SELECT query (and registering it as a temporary view)
> with about 200 fields/expressions out of another streaming table. The
> application is successfully submitted to Flink cluster. However the worker
> processes keep crashing, with the exception as quoted below.
> It clearly mentioned in the log that this is a bug, so I fire this ticket. By
> the way, if I lower the number of fields down to 100 then it works nicely.
> Please advice.
> Thanks a lot for all the efforts bring Flink up. It is really amazing!
> {code:java}
> java.lang.RuntimeException: Could not instantiate generated class
> 'GroupAggsHandler$9687'at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)
> at
> org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)
> at
> org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 10 moreCaused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 12 moreCaused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException:
> Compiling "GroupAggsHandler$9687": Code of method
> "retract(Lorg/apache/flink/table/dataformat/BaseRow;)V" of class
> "GroupAggsHandler$9687" grows beyond 64 KBat
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
[jira] [Commented] (FLINK-16589) Flink Table SQL fails/crashes with big queries with lots of fields
[
https://issues.apache.org/jira/browse/FLINK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058859#comment-17058859
]
Jark Wu commented on FLINK-16589:
-
Yes. We didn't split code for GroupAggsHandler, but that can be done as
[~libenchao] said, even though it is a complete solution...
Assigned this issue to you [~libenchao].
> Flink Table SQL fails/crashes with big queries with lots of fields
> --
>
> Key: FLINK-16589
> URL: https://issues.apache.org/jira/browse/FLINK-16589
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Viet Pham
>Assignee: Benchao Li
>Priority: Major
> Fix For: 1.10.1, 1.11.0
>
>
> Hi,
> My use case is a streaming application with a few streaming tables.
> I was trying to build a SELECT query (and registering it as a temporary view)
> with about 200 fields/expressions out of another streaming table. The
> application is successfully submitted to Flink cluster. However the worker
> processes keep crashing, with the exception as quoted below.
> It clearly mentioned in the log that this is a bug, so I fire this ticket. By
> the way, if I lower the number of fields down to 100 then it works nicely.
> Please advice.
> Thanks a lot for all the efforts bring Flink up. It is really amazing!
> {code:java}
> java.lang.RuntimeException: Could not instantiate generated class
> 'GroupAggsHandler$9687'at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)
> at
> org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)
> at
> org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 10 moreCaused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 12 moreCaused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException:
[GitHub] [flink] AHeise commented on issue #11303: [FLINK-16245] Decoupling user classloader from context classloader.
AHeise commented on issue #11303: [FLINK-16245] Decoupling user classloader from context classloader. URL: https://github.com/apache/flink/pull/11303#issuecomment-598786153 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-16047) Blink planner produces wrong aggregate results with state clean up
[
https://issues.apache.org/jira/browse/FLINK-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-16047:
---
Assignee: Jark Wu
> Blink planner produces wrong aggregate results with state clean up
> --
>
> Key: FLINK-16047
> URL: https://issues.apache.org/jira/browse/FLINK-16047
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>Priority: Critical
> Fix For: 1.9.3, 1.10.1, 1.11.0
>
>
> It seems that FLINK-10674 has not been ported to the Blink planner.
> Because state clean up happens in processing time, it might be the case that
> retractions are arriving after the state has been cleaned up. Before these
> changes, a new accumulator was created and invalid retraction messages were
> emitted. This change drops retraction messages for which no accumulator
> exists.
> These lines are missing in
> {{org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction}}:
> {code}
> if (null == accumulators) {
> // Don't create a new accumulator for a retraction message. This
> // might happen if the retraction message is the first message for the
> // key or after a state clean up.
> if (!inputC.change) {
> return
> }
> // first accumulate message
> firstRow = true
> accumulators = function.createAccumulators()
> } else {
> firstRow = false
> }
> {code}
> The bug has not been verified. I spotted it only by looking at the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16047) Blink planner produces wrong aggregate results with state clean up
[
https://issues.apache.org/jira/browse/FLINK-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058858#comment-17058858
]
Jark Wu commented on FLINK-16047:
-
I will pick up this issue and submit a PR soon.
> Blink planner produces wrong aggregate results with state clean up
> --
>
> Key: FLINK-16047
> URL: https://issues.apache.org/jira/browse/FLINK-16047
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0
>Reporter: Timo Walther
>Priority: Critical
> Fix For: 1.9.3, 1.10.1, 1.11.0
>
>
> It seems that FLINK-10674 has not been ported to the Blink planner.
> Because state clean up happens in processing time, it might be the case that
> retractions are arriving after the state has been cleaned up. Before these
> changes, a new accumulator was created and invalid retraction messages were
> emitted. This change drops retraction messages for which no accumulator
> exists.
> These lines are missing in
> {{org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction}}:
> {code}
> if (null == accumulators) {
> // Don't create a new accumulator for a retraction message. This
> // might happen if the retraction message is the first message for the
> // key or after a state clean up.
> if (!inputC.change) {
> return
> }
> // first accumulate message
> firstRow = true
> accumulators = function.createAccumulators()
> } else {
> firstRow = false
> }
> {code}
> The bug has not been verified. I spotted it only by looking at the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors URL: https://github.com/apache/flink/pull/11400#issuecomment-598611898 ## CI report: * a86f1e2e6785237893201dac302bc50041610d73 UNKNOWN * 5216b6412ed32669643e576b1879af2122f43e4b Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153193702) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6290) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598702521 ## CI report: * c7595ee5eb7ac60dae52bcd1368878d70d372533 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153184710) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6287) * c1facefebee52a1ac84f188f46b114deae0aad89 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153207095) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598702521 ## CI report: * c7595ee5eb7ac60dae52bcd1368878d70d372533 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153184710) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6287) * c1facefebee52a1ac84f188f46b114deae0aad89 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14348) YarnFileStageTestS3ITCase. testRecursiveUploadForYarnS3a fails to delete files
[
https://issues.apache.org/jira/browse/FLINK-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058832#comment-17058832
]
Aljoscha Krettek commented on FLINK-14348:
--
Same test, but different failure:
https://dev.azure.com/aljoschakrettek/Flink/_build/results?buildId=59&view=logs&j=764762df-f65b-572b-3d5c-65518c777be4&t=da3c2718-4b76-56bf-ef25-cd33ea381f78
> YarnFileStageTestS3ITCase. testRecursiveUploadForYarnS3a fails to delete files
> --
>
> Key: FLINK-14348
> URL: https://issues.apache.org/jira/browse/FLINK-14348
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.9.0
>Reporter: Caizhi Weng
>Priority: Major
>
> YarnFileStageTestS3ITCase. testRecursiveUploadForYarnS3a fails with the
> following exceptions:
> {code:java}
> 15:25:07.359 [ERROR]
> testRecursiveUploadForYarnS3a(org.apache.flink.yarn.YarnFileStageTestS3ITCase)
> Time elapsed: 10.808 s <<<
> ERROR!24649org.apache.hadoop.fs.s3a.AWSS3IOException: delete on
> s3a://[secure]/temp/tests-3565b11f-e9be-4213-a98d-0f0ecd123783/testYarn-s3a:
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 2D1AE3D999528C34; S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=),
> S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=:
> One or more objects could not be deleted (Service: null; Status Code: 200;
> Error Code: null; Request ID: 2D1AE3D999528C34; S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=)24650
>at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:159)24651
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3a(YarnFileStageTestS3ITCase.java:190)24652Caused
> by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 2D1AE3D999528C34; S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=)24653
>at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:159)24654
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3a(YarnFileStageTestS3ITCase.java:190){code}
> Travis log: [https://travis-ci.org/apache/flink/jobs/595082651]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14348) YarnFileStageTestS3ITCase. testRecursiveUploadForYarnS3a fails to delete files
[
https://issues.apache.org/jira/browse/FLINK-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-14348:
-
Component/s: Build System / Azure Pipelines
> YarnFileStageTestS3ITCase. testRecursiveUploadForYarnS3a fails to delete files
> --
>
> Key: FLINK-14348
> URL: https://issues.apache.org/jira/browse/FLINK-14348
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.9.0
>Reporter: Caizhi Weng
>Priority: Major
>
> YarnFileStageTestS3ITCase. testRecursiveUploadForYarnS3a fails with the
> following exceptions:
> {code:java}
> 15:25:07.359 [ERROR]
> testRecursiveUploadForYarnS3a(org.apache.flink.yarn.YarnFileStageTestS3ITCase)
> Time elapsed: 10.808 s <<<
> ERROR!24649org.apache.hadoop.fs.s3a.AWSS3IOException: delete on
> s3a://[secure]/temp/tests-3565b11f-e9be-4213-a98d-0f0ecd123783/testYarn-s3a:
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 2D1AE3D999528C34; S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=),
> S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=:
> One or more objects could not be deleted (Service: null; Status Code: 200;
> Error Code: null; Request ID: 2D1AE3D999528C34; S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=)24650
>at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:159)24651
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3a(YarnFileStageTestS3ITCase.java:190)24652Caused
> by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 2D1AE3D999528C34; S3 Extended Request ID:
> zIX1QsAcsY1ZYSDOeCaYsGJ4bz0NJTy2kw0EYmlJr8Kb7pM8OPmhAKO5XHI26xiOi2tIkTIoBwg=)24653
>at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:159)24654
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3a(YarnFileStageTestS3ITCase.java:190){code}
> Travis log: [https://travis-ci.org/apache/flink/jobs/595082651]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-11088) Allow pre-install Kerberos authentication keytab discovery on YARN
[ https://issues.apache.org/jira/browse/FLINK-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-11088. Resolution: Implemented master: 0920e6447283579551d14a0fed726f8b8d73bd3f > Allow pre-install Kerberos authentication keytab discovery on YARN > -- > > Key: FLINK-11088 > URL: https://issues.apache.org/jira/browse/FLINK-11088 > Project: Flink > Issue Type: Sub-task > Components: Deployment / YARN >Reporter: Rong Rong >Assignee: Rong Rong >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently flink-yarn assumes keytab is shipped as application master > environment local resource on client side and will be distributed to all the > TMs. This does not work for YARN proxy user mode [1] since proxy user or > super user might not have access to actual users' keytab, but can request > delegation tokens on users' behalf. > Based on the type of security options for long-living YARN service[2], we > propose to have the keytab file path discovery configurable depending on the > launch mode of the YARN client. > Reference: > [1] > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html > [2] > https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html#Securing_Long-lived_YARN_Services -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] aljoscha commented on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files
aljoscha commented on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files URL: https://github.com/apache/flink/pull/7702#issuecomment-598772667 I merged this. Thanks a lot for the good collaboration on this. 😃 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha closed pull request #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files
aljoscha closed pull request #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files URL: https://github.com/apache/flink/pull/7702 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] rkhachatryan commented on a change in pull request #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
rkhachatryan commented on a change in pull request #11400:
[FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in
respective constructors
URL: https://github.com/apache/flink/pull/11400#discussion_r392291325
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironment.java
##
@@ -217,6 +218,7 @@ public ShuffleIOOwnerContext createShuffleIOOwnerContext(
for (InputGateDeploymentDescriptor igdd :
inputGateDeploymentDescriptors) {
SingleInputGate inputGate =
singleInputGateFactory.create(
ownerContext.getOwnerName(),
+ counter,
Review comment:
Not sure if I got you right.
I meant to change the type from `Collection` to `List` for:
1. `inputGateDeploymentDescriptors` parameter in
`NettyShuffleEnvironment#createInputGates`
1. `TaskDeploymentDescriptor#inputGates` field
1. `inputGateDeploymentDescriptors` parameter in `Task` constructor
1. `inputGateDeploymentDescriptors` parameter in `TaskDeploymentDescriptor`
constructor
Without it, it's easy to break the ordering, at least for me :)
I think it should be the same PR because requirement for ordering was added
here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions
flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions URL: https://github.com/apache/flink/pull/11253#issuecomment-592517598 ## CI report: * d1368ad6d7262321e92a1e8c70211bc16a61dd05 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153170857) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6284) * 4e1c4f61f4e477262eb3d1d4fa67c97bcec1b85c Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153201488) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11402: [FLINK-16166][build] Append additional javadoc options
flinkbot edited a comment on issue #11402: [FLINK-16166][build] Append additional javadoc options URL: https://github.com/apache/flink/pull/11402#issuecomment-598690738 ## CI report: * 3f91303e7b54b0790f556d22da4fedc4872ac137 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153178906) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11398: [FLINK-16413]Reduce hive source parallelism when limit push down
flinkbot edited a comment on issue #11398: [FLINK-16413]Reduce hive source parallelism when limit push down URL: https://github.com/apache/flink/pull/11398#issuecomment-598579739 ## CI report: * 726c57d3f58bcf3adba172273a8b207e5f0281fb Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153201637) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters
flinkbot edited a comment on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters URL: https://github.com/apache/flink/pull/11404#issuecomment-598746532 ## CI report: * a7f81066eb1d939e9d6c492895bd5e0eb021360b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153198834) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11398: [FLINK-16413]Reduce hive source parallelism when limit push down
flinkbot edited a comment on issue #11398: [FLINK-16413]Reduce hive source parallelism when limit push down URL: https://github.com/apache/flink/pull/11398#issuecomment-598579739 ## CI report: * 4fe5672c65d62cb5e1854db4bf3404d1f4a5fdec Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153129571) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6272) * 726c57d3f58bcf3adba172273a8b207e5f0281fb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors URL: https://github.com/apache/flink/pull/11400#issuecomment-598611898 ## CI report: * a86f1e2e6785237893201dac302bc50041610d73 UNKNOWN * 5216b6412ed32669643e576b1879af2122f43e4b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153193702) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6290) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions
flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions URL: https://github.com/apache/flink/pull/11253#issuecomment-592517598 ## CI report: * d1368ad6d7262321e92a1e8c70211bc16a61dd05 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153170857) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6284) * 4e1c4f61f4e477262eb3d1d4fa67c97bcec1b85c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] twalthr commented on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions
twalthr commented on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions URL: https://github.com/apache/flink/pull/11253#issuecomment-598753216 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha commented on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions
aljoscha commented on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions URL: https://github.com/apache/flink/pull/11253#issuecomment-598750105 Lools good to merge! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10742) Let Netty use Flink's buffers directly in credit-based mode
[ https://issues.apache.org/jira/browse/FLINK-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijiang updated FLINK-10742: - Release Note: After FLINK-10742, while reading data stream from network on receiver side, it would reuse the Flink's buffers directly to avoid extra copy from Netty's ByteBuf to Flink's buffer. Therefore the Netty internal memory overhead is greatly reduced, which can solve the potential direct OutOfMemoryError specially in large-scale jobs. > Let Netty use Flink's buffers directly in credit-based mode > --- > > Key: FLINK-10742 > URL: https://issues.apache.org/jira/browse/FLINK-10742 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.7.0 >Reporter: Nico Kruber >Assignee: Yun Gao >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > For credit-based flow control, we always have buffers available for data that > is sent to use. We could thus use them directly and not copy the network > stream into Netty buffers first and then into our buffers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters
flinkbot commented on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters URL: https://github.com/apache/flink/pull/11404#issuecomment-598746532 ## CI report: * a7f81066eb1d939e9d6c492895bd5e0eb021360b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files
flinkbot edited a comment on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files URL: https://github.com/apache/flink/pull/7702#issuecomment-572195960 ## CI report: * d4a17beb028586a4b8b06a2eebfe7aa21ccf4fbb UNKNOWN * 0e955bd5b9f490a8ed66013ccd8ce3f512955513 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153193280) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6289) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors URL: https://github.com/apache/flink/pull/11400#issuecomment-598611898 ## CI report: * a86f1e2e6785237893201dac302bc50041610d73 UNKNOWN * 3c660599680944e8c6f2b3b0ceaf6975b03fc5c4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153164755) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6282) * 5216b6412ed32669643e576b1879af2122f43e4b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153193702) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6290) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters
flinkbot commented on issue #11404: [FLINK-13000][tests] Remove unused JobID parameters URL: https://github.com/apache/flink/pull/11404#issuecomment-598737637 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a7f81066eb1d939e9d6c492895bd5e0eb021360b (Fri Mar 13 14:08:39 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files
flinkbot edited a comment on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files URL: https://github.com/apache/flink/pull/7702#issuecomment-572195960 ## CI report: * d4a17beb028586a4b8b06a2eebfe7aa21ccf4fbb UNKNOWN * ae58c4fc1dcbe3ea0fa70326c09803c3e5792ede Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153089193) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6253) * 0e955bd5b9f490a8ed66013ccd8ce3f512955513 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153193280) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6289) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13000) Remove JobID argument from SimpleSlotProvider
[ https://issues.apache.org/jira/browse/FLINK-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13000: --- Labels: pull-request-available (was: ) > Remove JobID argument from SimpleSlotProvider > - > > Key: FLINK-13000 > URL: https://issues.apache.org/jira/browse/FLINK-13000 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination, Tests >Affects Versions: 1.9.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > The JobID constructor argument is unused and can be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol opened a new pull request #11404: [FLINK-13000][tests] Remove unused JobID parameters
zentol opened a new pull request #11404: [FLINK-13000][tests] Remove unused JobID parameters URL: https://github.com/apache/flink/pull/11404 Removes unused JobID parameters from the `SimpleSlotProvider` and `ExecutionGraphTestUtils#createSimpleTestGraph`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
zhijiangW commented on a change in pull request #11400: [FLINK-16586][network]
Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
URL: https://github.com/apache/flink/pull/11400#discussion_r392247611
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironment.java
##
@@ -217,6 +218,7 @@ public ShuffleIOOwnerContext createShuffleIOOwnerContext(
for (InputGateDeploymentDescriptor igdd :
inputGateDeploymentDescriptors) {
SingleInputGate inputGate =
singleInputGateFactory.create(
ownerContext.getOwnerName(),
+ counter,
Review comment:
I guess your suggestion is for unifying all the related data structures
(collection, array) from `TaskDeploymentDescriptor.inputGates` to `List`?
Actually I do not see any correctness or performance problems ATM. This
unification really spans cross multiple components from scheduler to task
stack, and I do think it is really necessary to refactor it now. Or if we want
to do that, it should be a separate ticket out of this PR scope. We only make
use of previous `counter` to give a correct gate index while creating in
iteration for this PR motivation. WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16047) Blink planner produces wrong aggregate results with state clean up
[
https://issues.apache.org/jira/browse/FLINK-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058748#comment-17058748
]
Yu Li commented on FLINK-16047:
---
[~jark] Any update on this one? Thanks.
> Blink planner produces wrong aggregate results with state clean up
> --
>
> Key: FLINK-16047
> URL: https://issues.apache.org/jira/browse/FLINK-16047
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.9.0
>Reporter: Timo Walther
>Priority: Critical
> Fix For: 1.9.3, 1.10.1, 1.11.0
>
>
> It seems that FLINK-10674 has not been ported to the Blink planner.
> Because state clean up happens in processing time, it might be the case that
> retractions are arriving after the state has been cleaned up. Before these
> changes, a new accumulator was created and invalid retraction messages were
> emitted. This change drops retraction messages for which no accumulator
> exists.
> These lines are missing in
> {{org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction}}:
> {code}
> if (null == accumulators) {
> // Don't create a new accumulator for a retraction message. This
> // might happen if the retraction message is the first message for the
> // key or after a state clean up.
> if (!inputC.change) {
> return
> }
> // first accumulate message
> firstRow = true
> accumulators = function.createAccumulators()
> } else {
> firstRow = false
> }
> {code}
> The bug has not been verified. I spotted it only by looking at the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16589) Flink Table SQL fails/crashes with big queries with lots of fields
[
https://issues.apache.org/jira/browse/FLINK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058745#comment-17058745
]
Benchao Li commented on FLINK-16589:
[~ykt836] We did the code split only for CalcCodeGenerator in
https://issues.apache.org/jira/browse/FLINK-15430.
If we need to split for {{GroupAggsHandler}} too, I can help to do that.
> Flink Table SQL fails/crashes with big queries with lots of fields
> --
>
> Key: FLINK-16589
> URL: https://issues.apache.org/jira/browse/FLINK-16589
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Viet Pham
>Priority: Major
>
> Hi,
> My use case is a streaming application with a few streaming tables.
> I was trying to build a SELECT query (and registering it as a temporary view)
> with about 200 fields/expressions out of another streaming table. The
> application is successfully submitted to Flink cluster. However the worker
> processes keep crashing, with the exception as quoted below.
> It clearly mentioned in the log that this is a bug, so I fire this ticket. By
> the way, if I lower the number of fields down to 100 then it works nicely.
> Please advice.
> Thanks a lot for all the efforts bring Flink up. It is really amazing!
> {code:java}
> java.lang.RuntimeException: Could not instantiate generated class
> 'GroupAggsHandler$9687'at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)
> at
> org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)
> at
> org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 10 moreCaused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 12 moreCaused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException:
> Compiling "GroupAggsHandler$9687": Code of method
>
[GitHub] [flink] flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors
flinkbot edited a comment on issue #11400: [FLINK-16586][network] Build ResultSubpartitionInfo and InputChannelInfo in respective constructors URL: https://github.com/apache/flink/pull/11400#issuecomment-598611898 ## CI report: * a86f1e2e6785237893201dac302bc50041610d73 UNKNOWN * 3c660599680944e8c6f2b3b0ceaf6975b03fc5c4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153164755) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6282) * 5216b6412ed32669643e576b1879af2122f43e4b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16589) Flink Table SQL fails/crashes with big queries with lots of fields
[
https://issues.apache.org/jira/browse/FLINK-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058742#comment-17058742
]
Jingsong Lee commented on FLINK-16589:
--
If just too many fields in "select *" with "group by", it can be resolved, just
like `ProjectionCodeGenerator`, we can extract its "loop" code generation.
> Flink Table SQL fails/crashes with big queries with lots of fields
> --
>
> Key: FLINK-16589
> URL: https://issues.apache.org/jira/browse/FLINK-16589
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Viet Pham
>Priority: Major
>
> Hi,
> My use case is a streaming application with a few streaming tables.
> I was trying to build a SELECT query (and registering it as a temporary view)
> with about 200 fields/expressions out of another streaming table. The
> application is successfully submitted to Flink cluster. However the worker
> processes keep crashing, with the exception as quoted below.
> It clearly mentioned in the log that this is a bug, so I fire this ticket. By
> the way, if I lower the number of fields down to 100 then it works nicely.
> Please advice.
> Thanks a lot for all the efforts bring Flink up. It is really amazing!
> {code:java}
> java.lang.RuntimeException: Could not instantiate generated class
> 'GroupAggsHandler$9687'at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)
> at
> org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)
> at
> org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)
> ... 10 moreCaused by:
> org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)
> ... 12 moreCaused by:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)
> at
> org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at
> org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException:
> Compiling "GroupAggsHandler$9687": Code of method
> "retract(Lorg/apache/flink/ta
[GitHub] [flink] flinkbot edited a comment on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files
flinkbot edited a comment on issue #7702: [FLINK-11088][Security][YARN] Allow YARN to discover pre-installed keytab files URL: https://github.com/apache/flink/pull/7702#issuecomment-572195960 ## CI report: * d4a17beb028586a4b8b06a2eebfe7aa21ccf4fbb UNKNOWN * ae58c4fc1dcbe3ea0fa70326c09803c3e5792ede Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153089193) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6253) * 0e955bd5b9f490a8ed66013ccd8ce3f512955513 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14311) Streaming File Sink end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058738#comment-17058738
]
Yu Li commented on FLINK-14311:
---
Another instance in release-1.10 crone build:
https://api.travis-ci.org/v3/job/661626127/log.txt
> Streaming File Sink end-to-end test failed on Travis
>
>
> Key: FLINK-14311
> URL: https://issues.apache.org/jira/browse/FLINK-14311
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.1, 1.11.0
>
>
> The {{Streaming File Sink end-to-end test}} fails on Travis because it does
> not produce output for 10 minutes.
> https://api.travis-ci.org/v3/job/591992274/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11402: [FLINK-16166][build] Append additional javadoc options
flinkbot edited a comment on issue #11402: [FLINK-16166][build] Append additional javadoc options URL: https://github.com/apache/flink/pull/11402#issuecomment-598690738 ## CI report: * 3f91303e7b54b0790f556d22da4fedc4872ac137 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153178906) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11303: [FLINK-16245] Decoupling user classloader from context classloader.
flinkbot edited a comment on issue #11303: [FLINK-16245] Decoupling user classloader from context classloader. URL: https://github.com/apache/flink/pull/11303#issuecomment-594377249 ## CI report: * 5246383ea27ab1a98a53e58985ef5923e9e9a6c4 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153164641) Azure: [CANCELED](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6281) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] sjwiesman commented on issue #11367: [FLINK-16044] / [FLINK-16045] Make libraries/connectors documentation top-level sections
sjwiesman commented on issue #11367: [FLINK-16044] / [FLINK-16045] Make libraries/connectors documentation top-level sections URL: https://github.com/apache/flink/pull/11367#issuecomment-598721539 I don’t know anything about seo but we’ve had issues in the past with google indexing the ‘stable’ branch of the docs. This means search results might return pages that no longer exist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598702521 ## CI report: * c7595ee5eb7ac60dae52bcd1368878d70d372533 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153184710) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6287) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] libenchao commented on a change in pull request #11396: [FLINK-16170][connectors/elasticsearch]SearchTemplateRequest ClassNotFoundException when use flink-sql-connector-elasticsearch7
libenchao commented on a change in pull request #11396: [FLINK-16170][connectors/elasticsearch]SearchTemplateRequest ClassNotFoundException when use flink-sql-connector-elasticsearch7 URL: https://github.com/apache/flink/pull/11396#discussion_r392219528 ## File path: flink-connectors/flink-sql-connector-elasticsearch7/pom.xml ## @@ -135,10 +131,6 @@ under the License. org.elasticsearch org.apache.flink.elasticsearch7.shaded.org.elasticsearch - Review comment: Do we need to remove this one? If yes, how about es6? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16499) Flink shaded hadoop could not work when Yarn timeline service is enabled
[
https://issues.apache.org/jira/browse/FLINK-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058717#comment-17058717
]
Yang Wang commented on FLINK-16499:
---
[~chesnay] It seems that even we just use the Yarn client, we still need the
jersey-core as dependency when the Yarn timeline service enabled.
> Flink shaded hadoop could not work when Yarn timeline service is enabled
>
>
> Key: FLINK-16499
> URL: https://issues.apache.org/jira/browse/FLINK-16499
> Project: Flink
> Issue Type: Bug
> Components: BuildSystem / Shaded
>Reporter: Yang Wang
>Priority: Major
>
> When the Yarn timeline service is enabled (via
> {{yarn.timeline-service.enabled=true}} in yarn-site.xml), flink-shaded-hadoop
> could not work to submit Flink job to Yarn cluster. The following exception
> will be thrown.
>
> The root cause is the {{jersey-core-xx.jar}} is not bundled into
> {{flink-shaded-hadoop-xx}}{{.jar}}.
>
> {code:java}
> 2020-03-09 03:35:34,396 ERROR org.apache.flink.client.cli.CliFrontend
> [] - Fatal error while running command line interface.2020-03-09
> 03:35:34,396 ERROR org.apache.flink.client.cli.CliFrontend
> [] - Fatal error while running command line
> interface.java.lang.NoClassDefFoundError: javax/ws/rs/ext/MessageBodyReader
> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_242] at
> java.lang.ClassLoader.defineClass(ClassLoader.java:757) ~[?:1.8.0_242] at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> ~[?:1.8.0_242] at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_242]
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_242]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_242] at
> java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_242] at
> java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242] at
> java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_242] at
> java.lang.ClassLoader.loadClass(ClassLoader.java:419) ~[?:1.8.0_242] at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_242]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ~[?:1.8.0_242] at
> java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_242] at
> java.lang.ClassLoader.defineClass(ClassLoader.java:757) ~[?:1.8.0_242] at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> ~[?:1.8.0_242] at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_242]
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_242]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_242] at
> java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_242] at
> java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242] at
> java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_242] at
> java.lang.ClassLoader.loadClass(ClassLoader.java:419) ~[?:1.8.0_242] at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_242]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ~[?:1.8.0_242] at
> java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_242] at
> java.lang.ClassLoader.defineClass(ClassLoader.java:757) ~[?:1.8.0_242] at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> ~[?:1.8.0_242] at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_242]
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_242]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_242] at
> java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_242] at
> java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242] at
> java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_242] at
> java.lang.ClassLoader.loadClass(ClassLoader.java:419) ~[?:1.8.0_242] at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_242]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ~[?:1.8.0_242] at
> org.apache.hadoop.yarn.util.timeline.TimelineUtils.(TimelineUtils.java:50)
> ~[flink-shaded-hadoop-2-uber-2.8.3-7.0.jar:2.8.3-7.0] at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:179)
> ~[flink-shaded-hadoop-2-uber-2.8.3-7.0.jar:2.8.3-7.0] at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ~[flink-shaded-hadoop-2-uber-2.8.3-7.0.jar:2.8.3-7.0] at
> org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:71)
> ~[flink-dist_2.11-1.10.0-vvr-0.1-SNAPSHOT.jar:1.10.0-vvr-0.1-S
[GitHub] [flink] zentol commented on a change in pull request #11303: [FLINK-16245] Decoupling user classloader from context classloader.
zentol commented on a change in pull request #11303: [FLINK-16245] Decoupling
user classloader from context classloader.
URL: https://github.com/apache/flink/pull/11303#discussion_r392217711
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders.java
##
@@ -82,4 +89,61 @@ public static ResolveOrder fromString(String resolveOrder) {
super(urls, parent);
}
}
+
+ /**
+* Ensures that holding a reference on the context class loader
outliving the scope of user code does not prevent
+* the user classloader to be garbage collected (FLINK-16245).
+*
+* This classloader delegates to the actual user classloader. Upon
{@link #close()}, the delegate is nulled
+* and can be garbage collected. Additional class resolution will be
resolved solely through the bootstrap
+* classloader and most likely result in ClassNotFound exceptions.
+*/
+ private static class SafetyNetWrapperClassLoader extends URLClassLoader
+ implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(SafetyNetWrapperClassLoader.class);
+
+ private FlinkUserCodeClassLoader inner;
Review comment:
```suggestion
private final FlinkUserCodeClassLoader inner;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #11303: [FLINK-16245] Decoupling user classloader from context classloader.
zentol commented on a change in pull request #11303: [FLINK-16245] Decoupling
user classloader from context classloader.
URL: https://github.com/apache/flink/pull/11303#discussion_r392217711
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders.java
##
@@ -82,4 +89,61 @@ public static ResolveOrder fromString(String resolveOrder) {
super(urls, parent);
}
}
+
+ /**
+* Ensures that holding a reference on the context class loader
outliving the scope of user code does not prevent
+* the user classloader to be garbage collected (FLINK-16245).
+*
+* This classloader delegates to the actual user classloader. Upon
{@link #close()}, the delegate is nulled
+* and can be garbage collected. Additional class resolution will be
resolved solely through the bootstrap
+* classloader and most likely result in ClassNotFound exceptions.
+*/
+ private static class SafetyNetWrapperClassLoader extends URLClassLoader
+ implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(SafetyNetWrapperClassLoader.class);
+
+ private FlinkUserCodeClassLoader inner;
Review comment:
```suggestion
private final FlinkUserCodeClassLoader inner;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (FLINK-16554) Extract static classes from StreamTask
[ https://issues.apache.org/jira/browse/FLINK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijiang closed FLINK-16554. Resolution: Fixed Merged in master: 3b99f7e4d769f6e2b6e9fd8e36a542e063d6c369 > Extract static classes from StreamTask > -- > > Key: FLINK-16554 > URL: https://issues.apache.org/jira/browse/FLINK-16554 > Project: Flink > Issue Type: Task > Components: Runtime / Task >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > StreamTask is currently 1400+ LOC. > We can cut it to 1100+ by simply extracting these static classes into > separate files: > * `CheckpointingOperation` > * `AsyncCheckpointRunnable` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot edited a comment on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598702521 ## CI report: * c7595ee5eb7ac60dae52bcd1368878d70d372533 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/153184710) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6287) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16478) add restApi to modify loglevel
[ https://issues.apache.org/jira/browse/FLINK-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058707#comment-17058707 ] Yang Wang commented on FLINK-16478: --- [~chesnay] Thanks for you response. I think it is not enough. For k8s deployment, we could update the configmap and the changed log4j.properties will automatically be mount into the JM/TM. So it is enough in such case. However for Yarn deployment, it is not very easy to find the location of log4j.properties. Also we need to login to the Yarn cluster and have enough permission to change the file. It is usually impossible, especially in the production environment. If we could support update the log level by rest/webui, it will be very convenient. > add restApi to modify loglevel > --- > > Key: FLINK-16478 > URL: https://issues.apache.org/jira/browse/FLINK-16478 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: xiaodao >Priority: Minor > > sometimes we may need to change loglevel to get more information to resolved > bug, now we need to stop it and modify conf/log4j.properties and resubmit it > ,i think it's better to add rest api to modify loglevel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions
flinkbot edited a comment on issue #11253: [FLINK-16336][table] Add new type inference for temporal table functions URL: https://github.com/apache/flink/pull/11253#issuecomment-592517598 ## CI report: * d1368ad6d7262321e92a1e8c70211bc16a61dd05 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153170857) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6284) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW merged pull request #11382: [FLINK-16554][task] split StreamTask
zhijiangW merged pull request #11382: [FLINK-16554][task] split StreamTask URL: https://github.com/apache/flink/pull/11382 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11303: [FLINK-16245] Decoupling user classloader from context classloader.
AHeise commented on a change in pull request #11303: [FLINK-16245] Decoupling
user classloader from context classloader.
URL: https://github.com/apache/flink/pull/11303#discussion_r392210373
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders.java
##
@@ -82,4 +91,78 @@ public static ResolveOrder fromString(String resolveOrder) {
super(urls, parent);
}
}
+
+ /**
+* Ensures that holding a reference on the context class loader
outliving the scope of user code does not prevent
+* the user classloader to be garbage collected (FLINK-16245).
+*
+* This classloader delegates to the actual user classloader. Upon
{@link #close()}, the delegate is nulled
+* and can be garbage collected. Additional class resolution will be
resolved solely through the bootstrap
+* classloader and most likely result in ClassNotFound exceptions.
+*
+* @param the classloader type that also needs to be closeable.
+*/
+ private static class SafetyNetWrapperClassLoader extends CloseableClassLoader
+ implements Closeable {
+ private static final Logger LOG =
LoggerFactory.getLogger(SafetyNetWrapperClassLoader.class);
+
+ private T inner;
+
+ SafetyNetWrapperClassLoader(T inner) {
+ super(null);
+ this.inner = inner;
+ }
+
+ @Override
+ public void close() {
+ if (inner != null) {
+ try {
+ inner.close();
+ } catch (IOException e) {
+ LOG.warn("Could not close user
classloader", e);
+ }
+ }
+ inner = null;
+ }
+
+ @Override
+ protected Class loadClass(String name, boolean resolve)
throws ClassNotFoundException {
+ if (inner == null) {
+ return super.loadClass(name, resolve);
+ }
+
+ synchronized (getClassLoadingLock(name)) {
Review comment:
Added a common ancestor now and used delegation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-10742) Let Netty use Flink's buffers directly in credit-based mode
[ https://issues.apache.org/jira/browse/FLINK-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058701#comment-17058701 ] Zhijiang commented on FLINK-10742: -- Thanks for the kind reminder [~NicoK]. I think it is worth putting into release note to be aware by users. I would handle it later. > Let Netty use Flink's buffers directly in credit-based mode > --- > > Key: FLINK-10742 > URL: https://issues.apache.org/jira/browse/FLINK-10742 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.7.0 >Reporter: Nico Kruber >Assignee: Yun Gao >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > For credit-based flow control, we always have buffers available for data that > is sent to use. We could thus use them directly and not copy the network > stream into Netty buffers first and then into our buffers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot commented on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598702521 ## CI report: * c7595ee5eb7ac60dae52bcd1368878d70d372533 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
pnowojski commented on a change in pull request #11403:
[FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for
AbstractStreamOperator
URL: https://github.com/apache/flink/pull/11403#discussion_r392202032
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/StreamOperatorBase.java
##
@@ -0,0 +1,484 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.State;
+import org.apache.flink.api.common.state.StateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MetricOptions;
+import org.apache.flink.core.fs.CloseableRegistry;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.execution.Environment;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.metrics.groups.OperatorMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerJobMetricGroup;
+import org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.OperatorStateBackend;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.watermark.Watermark;
+import org.apache.flink.streaming.runtime.streamrecord.LatencyMarker;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.streaming.util.LatencyStats;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Locale;
+import java.util.Optional;
+
+/**
+ * New base class for all stream operators, replacing previous {@link
AbstractStreamOperator}.
+ * Currently intended to work with {@link MultipleInputStreamOperator}.
+ *
+ * One note-able difference in comparison to {@link AbstractStreamOperator}
is lack of
+ * {@link AbstractStreamOperator#setup(StreamTask, StreamConfig, Output)} in
favor of initialisation
+ * in the constructor, and removed some tight coupling with classes like
{@link StreamTask}.
+ *
+ * Methods are guaranteed not to be called concurrently.
+ *
+ * @param The output type of the operator
+ */
+@Experimental
+public abstract class StreamOperatorBase implements StreamOperator {
Review comment:
alternative name could be `AbstractStreamOperatorV2`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
pnowojski commented on a change in pull request #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#discussion_r392201502 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/StreamOperatorFactoryUtil.java ## @@ -57,7 +57,13 @@ ((ProcessingTimeServiceAware) operatorFactory).setProcessingTimeService(processingTimeService); } - OP op = operatorFactory.createStreamOperator(containingTask, configuration, output); + // TODO: what to do with ProcessingTimeServiceAware? + OP op = operatorFactory.createStreamOperator( + new StreamOperatorInitializer<>( + containingTask, + configuration, + output, + processingTimeService)); Review comment: @AHeise what do you think about including `processingTimeService` in `StreamOperatorInitializer` always, regardless of the `ProcessingTimeServiceAware`? Generally speaking what do you think about `StreamOperatorInitializer`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
flinkbot commented on issue #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403#issuecomment-598697714 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c7595ee5eb7ac60dae52bcd1368878d70d372533 (Fri Mar 13 12:30:44 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11401: [FLINK-15991][doc][zh] Translate memory configuration docs into Chinese.
flinkbot edited a comment on issue #11401: [FLINK-15991][doc][zh] Translate memory configuration docs into Chinese. URL: https://github.com/apache/flink/pull/11401#issuecomment-598647884 ## CI report: * 44c238715de2eb32237f37d3c38aa42f6765fd6c Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153164774) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6283) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11382: [FLINK-16554][task] split StreamTask
flinkbot edited a comment on issue #11382: [FLINK-16554][task] split StreamTask URL: https://github.com/apache/flink/pull/11382#issuecomment-597788287 ## CI report: * 55525e050ada84dcd0e1ea935dffe3b7f09faa42 UNKNOWN * 3b898115d9c7e949cc6751b911098e9b2ee0f295 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/153145930) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6278) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11402: [FLINK-16166][build] Append additional javadoc options
flinkbot edited a comment on issue #11402: [FLINK-16166][build] Append additional javadoc options URL: https://github.com/apache/flink/pull/11402#issuecomment-598690738 ## CI report: * 3f91303e7b54b0790f556d22da4fedc4872ac137 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/153178906) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16316) Provide new base class for StreamOperators
[
https://issues.apache.org/jira/browse/FLINK-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16316:
---
Labels: pull-request-available (was: )
> Provide new base class for StreamOperators
> --
>
> Key: FLINK-16316
> URL: https://issues.apache.org/jira/browse/FLINK-16316
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataStream
>Affects Versions: 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Current {{AbstractStreamOperator}} has at least couple of issues:
> # it's using deprecated {{SetupableStreamOperator}} interface, instead of
> relaying on {{StreamOperatorFactory}}.
> # it depends on {{OneInputStreamOperator}} and {{TwoInputStreamOperator}}
> interfaces, by providing indirectly implementation of methods like
> {{processLatencyMarker}}, {{processLatencyMarker1}} and
> {{processLatencyMarker2}}
> Both of those things makes {{AbstractStreamOperator}} an unsuitable base
> implementation for {{MultipleInputStreamOperator}}. We should provide a new
> base class (or hierarchy of base classes), without those two limitations.
> CC [~aljoscha]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] pnowojski opened a new pull request #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator
pnowojski opened a new pull request #11403: [FLINK-16316][operators] Implement new StreamOperatorBase as a replacement for AbstractStreamOperator URL: https://github.com/apache/flink/pull/11403 Implement new StreamOperatorBase as a replacement for AbstractStreamOperator The new base class for operators tries to address couple of limitations in the AbstractStreamOperator like: - lack of support for multiple inputs - setup(...) method ## Brief change log Please check individual commit messages. ## Verifying this change This change is cover by existing tests and adds couple of new ones where necessary. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
