[GitHub] [flink] flinkbot edited a comment on issue #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
flinkbot edited a comment on issue #11692: URL: https://github.com/apache/flink/pull/11692#issuecomment-611530487 ## CI report: * 4c854e0afe5cb934d919f37eb0891719063938ac Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160752603) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7683) * aa7f40b60eb7b67fd3e2ec6cd4be5890e2218ba8 UNKNOWN * 769e5d807f5c6671c0be5f39bc0309024d1caac2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
flinkbot edited a comment on issue #11615: URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339 ## CI report: * 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN * 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN * 778239dfa00a84ab8457d49eedc759b601028676 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160682224) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632) * 2a47b5818c6bf072ab36f47130392f46a9bd4df8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160982187) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7747) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dianfu commented on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616355772 It's a good idea to add description in the PR commit message and will do that. For this issue itself, I have already explained in the JIRA. Let me try to explain it in the PR again: It's a bug introduced by CALCITE-3149. Theoretically speaking, all the SQL jobs have chances to encounter this bug. We need to fix in the calcite eventually. @danny0405 is already helping on that. Regarding to the fix in this PR, it just adjust the test case a bit to make it doesn't encounter into the bug. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on issue #11815: [FLINK-17253] Support viewfs for hadoop version < 2.7
flinkbot commented on issue #11815: URL: https://github.com/apache/flink/pull/11815#issuecomment-616355624 ## CI report: * 82b7e411419bbd8a1011bdc78e2a916dbc1256a7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11814: [FLINK-17218][tests] Adding recoverable failures and correctness chec…
flinkbot edited a comment on issue #11814: URL: https://github.com/apache/flink/pull/11814#issuecomment-616347961 ## CI report: * 2b64d49dac7f89d4abf600fda1d7beb5b34d3fc5 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160982313) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7748) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11813: [FLINK-17249] Bump universal Kafka connector to Kafka 2.2.2
flinkbot edited a comment on issue #11813: URL: https://github.com/apache/flink/pull/11813#issuecomment-616239083 ## CI report: * 49a1530e3b0320a7053c977c2d36c1eed49245a2 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160974132) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7745) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-16408) Bind user code class loader to lifetime of a slot
[ https://issues.apache.org/jira/browse/FLINK-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann reassigned FLINK-16408: - Assignee: Till Rohrmann > Bind user code class loader to lifetime of a slot > - > > Key: FLINK-16408 > URL: https://issues.apache.org/jira/browse/FLINK-16408 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.9.2, 1.10.0 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > Fix For: 1.10.1, 1.11.0 > > > In order to avoid class leaks due to creating multiple user code class > loaders and loading class multiple times in a recovery case, I would suggest > to bind the lifetime of a user code class loader to the lifetime of a slot. > More precisely, the user code class loader should live at most as long as the > slot which is using it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17189) Table with processing time attribute can not be read from Hive catalog
[
https://issues.apache.org/jira/browse/FLINK-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087431#comment-17087431
]
Timo Walther commented on FLINK-17189:
--
Hi [~lirui],
yes, the {{TimestampKind}} is excluded by design. The data types of the table
schema should be only {{TIMESTAMP(3)}} in Hive. The information whether a
column is a time attribute is encoded in {{L_PROCTIME AS PROCTIME()}} for
processing time and {{WATERMARK FOR L_ORDERTIME AS L_ORDERTIME - INTERVAL '5'
MINUTE}} for event time. Those important schema parts should also be written
out (but in properties) and used when reading in a table again. We should have
the same logic already for event time. Just processing time seems to have
issues.
> Table with processing time attribute can not be read from Hive catalog
> --
>
> Key: FLINK-17189
> URL: https://issues.apache.org/jira/browse/FLINK-17189
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem, Table SQL / Planner
>Reporter: Timo Walther
>Priority: Major
> Fix For: 1.11.0, 1.10.2
>
>
> DDL:
> {code}
> CREATE TABLE PROD_LINEITEM (
> L_ORDERKEY INTEGER,
> L_PARTKEYINTEGER,
> L_SUPPKEYINTEGER,
> L_LINENUMBER INTEGER,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAXDOUBLE,
> L_CURRENCY STRING,
> L_RETURNFLAG STRING,
> L_LINESTATUS STRING,
> L_ORDERTIME TIMESTAMP(3),
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENTSTRING,
> WATERMARK FOR L_ORDERTIME AS L_ORDERTIME - INTERVAL '5' MINUTE,
> L_PROCTIME AS PROCTIME()
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'Lineitem',
> 'connector.properties.zookeeper.connect' = 'not-needed',
> 'connector.properties.bootstrap.servers' = 'kafka:9092',
> 'connector.startup-mode' = 'earliest-offset',
> 'format.type' = 'csv',
> 'format.field-delimiter' = '|'
> );
> {code}
> Query:
> {code}
> SELECT * FROM prod_lineitem;
> {code}
> Result:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.AssertionError: Conversion to relational algebra failed to preserve
> datatypes:
> validated type:
> RecordType(INTEGER L_ORDERKEY, INTEGER L_PARTKEY, INTEGER L_SUPPKEY, INTEGER
> L_LINENUMBER, DOUBLE L_QUANTITY, DOUBLE L_EXTENDEDPRICE, DOUBLE L_DISCOUNT,
> DOUBLE L_TAX, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_CURRENCY,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_RETURNFLAG,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_LINESTATUS, TIME
> ATTRIBUTE(ROWTIME) L_ORDERTIME, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> L_SHIPINSTRUCT, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_SHIPMODE,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_COMMENT, TIMESTAMP(3) NOT NULL
> L_PROCTIME) NOT NULL
> converted type:
> RecordType(INTEGER L_ORDERKEY, INTEGER L_PARTKEY, INTEGER L_SUPPKEY, INTEGER
> L_LINENUMBER, DOUBLE L_QUANTITY, DOUBLE L_EXTENDEDPRICE, DOUBLE L_DISCOUNT,
> DOUBLE L_TAX, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_CURRENCY,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_RETURNFLAG,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_LINESTATUS, TIME
> ATTRIBUTE(ROWTIME) L_ORDERTIME, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> L_SHIPINSTRUCT, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_SHIPMODE,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_COMMENT, TIME
> ATTRIBUTE(PROCTIME) NOT NULL L_PROCTIME) NOT NULL
> rel:
> LogicalProject(L_ORDERKEY=[$0], L_PARTKEY=[$1], L_SUPPKEY=[$2],
> L_LINENUMBER=[$3], L_QUANTITY=[$4], L_EXTENDEDPRICE=[$5], L_DISCOUNT=[$6],
> L_TAX=[$7], L_CURRENCY=[$8], L_RETURNFLAG=[$9], L_LINESTATUS=[$10],
> L_ORDERTIME=[$11], L_SHIPINSTRUCT=[$12], L_SHIPMODE=[$13], L_COMMENT=[$14],
> L_PROCTIME=[$15])
> LogicalWatermarkAssigner(rowtime=[L_ORDERTIME], watermark=[-($11,
> 30:INTERVAL MINUTE)])
> LogicalProject(L_ORDERKEY=[$0], L_PARTKEY=[$1], L_SUPPKEY=[$2],
> L_LINENUMBER=[$3], L_QUANTITY=[$4], L_EXTENDEDPRICE=[$5], L_DISCOUNT=[$6],
> L_TAX=[$7], L_CURRENCY=[$8], L_RETURNFLAG=[$9], L_LINESTATUS=[$10],
> L_ORDERTIME=[$11], L_SHIPINSTRUCT=[$12], L_SHIPMODE=[$13], L_COMMENT=[$14],
> L_PROCTIME=[PROCTIME()])
> LogicalTableScan(table=[[hcat, default, prod_lineitem, source:
> [KafkaTableSource(L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY,
> L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_CURRENCY, L_RETURNFLAG, L_LINESTATUS,
> L_ORDERTIME, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT)]]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dianfu edited a comment on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dianfu edited a comment on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616355772 It's a good idea to add description in the PR commit message and will do that. For this issue itself, I have already explained in the JIRA. Let me try to explain it in the PR again: It's a bug introduced by CALCITE-3149. It ensures that the RelDataType cache (stored in DATATYPE_CACHE) in RelDataTypeFactoryImpl could be garbage collected. However, the equality check still checks the [object reference](https://github.com/apache/calcite/blob/52a57078ba081b24b9d086ed363c715485d1a519/core/src/main/java/org/apache/calcite/rex/RexProgramBuilder.java#L948) instead using `object.equals`. It may cause issues in cases that the cache DATATYPE_CACHE has been garbage collected and at the same time there are still references to the old RelDataType. During debugging this problem, I have saw there are RelDataTypes which are not in the DATATYPE_CACHE cache and this is the root cause of the test failure reported in this JIRA. Theoretically speaking, all the SQL jobs have chances to encounter this bug. We need to fix in the calcite eventually. @danny0405 is already helping on that. Regarding to the fix in this PR, it just adjust the test case a bit to make it doesn't encounter into the bug. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16763) Should not use BatchTableEnvironment for Python UDF in the document of flink-1.10
[ https://issues.apache.org/jira/browse/FLINK-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16763: --- Labels: pull-request-available (was: ) > Should not use BatchTableEnvironment for Python UDF in the document of > flink-1.10 > - > > Key: FLINK-16763 > URL: https://issues.apache.org/jira/browse/FLINK-16763 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.10.1 > > > Currently, for flink-1.10, Python UDF is not supported in old planner under > batch mode, so we should not use BatchTableEnvironment as an example for > Python UDF in the document[1] of flink-1.10. The problem may also exist in > other places in the document and should be checked. > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/python/python_udfs.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 opened a new pull request #11816: [FLINK-16763][python][doc] Replace BatchTableEnvironment with StreamTableEnvironment for Python UDF examples
hequn8128 opened a new pull request #11816: URL: https://github.com/apache/flink/pull/11816 ## What is the purpose of the change Replace BatchTableEnvironment with StreamTableEnvironment for Python UDF examples since for flink-1.10, Python UDF is not supported in old planner under batch mode. ## Brief change log - Replace BatchTableEnvironment with StreamTableEnvironment for Python UDF examples in the document ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
dawidwys commented on a change in pull request #11692:
URL: https://github.com/apache/flink/pull/11692#discussion_r411147490
##
File path:
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/connector/source/abilities/PeriodicWatermarkAssignerProvider.java
##
@@ -28,19 +28,14 @@
* generating watermarks in {@link ScanTableSource}.
*/
@PublicEvolving
-public final class PeriodicWatermarkAssignerProvider extends
SupportsWatermarkPushDown.WatermarkProvider {
+public interface PeriodicWatermarkAssignerProvider extends
SupportsWatermarkPushDown.WatermarkProvider {
- private final AssignerWithPeriodicWatermarks periodicAssigner;
-
- private
PeriodicWatermarkAssignerProvider(AssignerWithPeriodicWatermarks
periodicAssigner) {
- this.periodicAssigner = periodicAssigner;
- }
-
- public static PeriodicWatermarkAssignerProvider
of(AssignerWithPeriodicWatermarks periodicAssigner) {
- return new PeriodicWatermarkAssignerProvider(periodicAssigner);
+ /**
+* Helper method for creating a static provider.
+*/
+ static PeriodicWatermarkAssignerProvider
of(AssignerWithPeriodicWatermarks periodicAssigner) {
Review comment:
This method does not make much sense, does it? Planner will always need
to create a Provider that implements at least two of the interfaces e.g.
`PeriodicWatermarkAssignerProvider` and FLIP-27 `WatermarkProvider`? Moreover
we probably want to have some supplier interface here. We don't necessarily
need to instantiate the `AssignerWithPeriodicWatermarks` if the source works
with the interface from FLIP-27.
Can we just remove this method? I really can't see any benefit of it
whatsoever.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on issue #11816: [FLINK-16763][python][doc] Replace BatchTableEnvironment with StreamTableEnvironment for Python UDF examples
flinkbot commented on issue #11816: URL: https://github.com/apache/flink/pull/11816#issuecomment-616359471 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0ada8ce36bd618fa5e9c869ca6cf0e4c059dda92 (Mon Apr 20 07:19:14 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17012) Expose stage of task initialization
[ https://issues.apache.org/jira/browse/FLINK-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087435#comment-17087435 ] Wenlong Lyu commented on FLINK-17012: - hi, [~sewen] thanks for the explanation. but I think that it may not be good to do much IO and user code initialization(including state intialization and Function#open) in the constructor, it can easily cause exceptions because of failure of external system and user code exception. It would be difficult to do clean up when exception happens in the constructor. > Expose stage of task initialization > --- > > Key: FLINK-17012 > URL: https://issues.apache.org/jira/browse/FLINK-17012 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Task >Reporter: Wenlong Lyu >Priority: Major > > Currently a task switches to running before fully initialized, does not take > state initialization and operator initialization(#open ) in to account, which > may take long time to finish. As a result, there would be a weird phenomenon > that all tasks are running but throughput is 0. > I think it could be good if we can expose the initialization stage of tasks. > What to you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #11397: [FLINK-16217] [sql-client] catch all exceptions to avoid SQL client crashed
twalthr commented on a change in pull request #11397:
URL: https://github.com/apache/flink/pull/11397#discussion_r411152388
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
##
@@ -196,8 +196,12 @@ public void open() {
if (line == null) {
continue;
}
- final Optional cmdCall =
parseCommand(line);
- cmdCall.ifPresent(this::callCommand);
+ try {
+ final Optional cmdCall =
parseCommand(line);
+ cmdCall.ifPresent(this::callCommand);
+ } catch (Throwable t) {
Review comment:
The SQL Client process should not continue if something unexpected
happened. The process might be in a bad state then. We should fail hard in
those cases. The current exception design is also considering this. In the
main() method a wrapping message will be displayed that says `This is a bug.
Please file an issue.`. We should rather fix `CliClient#callSet()` in this case
instead of messing with the exception design.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #11787: [FLINK-16746] Deprecate legacy heap memory option for JM and expose the new ones in docs
xintongsong commented on a change in pull request #11787:
URL: https://github.com/apache/flink/pull/11787#discussion_r411148809
##
File path: flink-jepsen/src/jepsen/flink/db.clj
##
@@ -41,7 +41,7 @@
{:high-availability "zookeeper"
:high-availability.zookeeper.quorum(zookeeper-quorum test)
:high-availability.storageDir "hdfs:///flink/ha"
- :jobmanager.heap.size "2048m"
+ :jobmanager.memory.process.size"2496m"
Review comment:
Same here? Why is the value change "2048 -> 2496"?
##
File path: flink-dist/src/main/resources/flink-conf.yaml
##
@@ -37,9 +37,11 @@ jobmanager.rpc.address: localhost
jobmanager.rpc.port: 6123
-# The heap size for the JobManager JVM
+# The total process memory size for the JobManager.
+#
+# Note this accounts for all memory usage within the JobManager process,
including JVM metaspace and other overhead.
-jobmanager.heap.size: 1024m
+jobmanager.memory.process.size: 1472m
Review comment:
Could you explain why is the default 1472m?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
dawidwys commented on a change in pull request #11692:
URL: https://github.com/apache/flink/pull/11692#discussion_r411153299
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/sink/abilities/SupportsPartitioning.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.connector.sink.abilities;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+
+import java.util.Map;
+
+/**
+ * Enables to write partitioned data in a {@link DynamicTableSink}.
+ *
+ * Partitions split the data stored in an external system into smaller
portions that are identified
+ * by one or more string-based partition keys. A single partition is
represented as a {@code Map < String, String >}
+ * which maps each partition key to a partition value. Partition keys and
their order are defined by the
+ * catalog table.
+ *
+ * For example, data can be partitioned by region and within a region
partitioned by month. The order
+ * of the partition keys (in the example: first by region then by month) is
defined by the catalog table. A
+ * list of partitions could be:
+ *
+ * List(
+ * ['region'='europe', 'month'='2020-01'],
+ * ['region'='europe', 'month'='2020-02'],
+ * ['region'='asia', 'month'='2020-01'],
+ * ['region'='asia', 'month'='2020-02']
+ * )
+ *
+ *
+ * Given the following partitioned table:
+ * {@code
+ * CREATE TABLE t (a INT, b STRING, c DOUBLE, region STRING, month STRING)
PARTITION BY (region, month);
+ * }
+ *
+ * We can insert data into static table partitions using the {@code
INSERT INTO ... PARTITION} syntax:
+ * {@code
+ * INSERT INTO t PARTITION (region='europe', month='2020-01') SELECT a, b,
c FROM my_view;
+ * }
+ *
+ * If all partition keys get a value assigned in the {@code PARTITION}
clause, the operation is considered
+ * as an "insertion into a static partition". In the above example, the query
result should be written
+ * into the static partition {@code region='europe', month='2020-01'} which
will be passed by the planner
+ * into {@link #applyStaticPartition(Map)}.
+ *
+ * Alternatively, we can insert data into dynamic table partitions
using the SQL syntax:
+ * {@code
+ * INSERT INTO t PARTITION (region='europe') SELECT a, b, c, month FROM
another_view;
+ * }
+ *
+ * If only a subset of all partition keys (a prefix part) get a value
assigned in the {@code PARTITION}
+ * clause, the operation is considered as an "insertion into a dynamic
partition". In the above example,
+ * the static partition part is {@code region='europe'} which will be passed
by the planner into
+ * {@link #applyStaticPartition(Map)}. The remaining values for partition keys
should be obtained from
+ * each individual record by the sink during runtime. In the example, the
{@code month} field is the dynamic
+ * partition key.
+ */
+@PublicEvolving
+public interface SupportsPartitioning {
+
+ /**
+* Provides the static part of a partition.
+*
+* A single partition maps each partition key to a partition value.
Depending on the user-defined
+* statement, the partition might not include all partition keys.
+*
+* See the documentation of {@link SupportsPartitioning} for more
information.
+*
+* @param partition user-defined (possibly partial) static partition
+*/
+ void applyStaticPartition(Map partition);
+
+ /**
+* Returns whether data needs to be grouped by partition before it is
consumed by the sink. By default,
+* this is not required from the runtime and records arrive in
arbitrary partition order.
+*
+* If this method returns true, the sink can expect that all records
will be grouped by the partition
+* keys before consumed by the sink. In other words: The sink will
receive all elements of one
+* partition and then all elements of another partition. Elements of
different partitions will not
Review comment:
The benefits I see in the approach I suggested are as follows:
* The planner makes the decision wh
[GitHub] [flink] dawidwys commented on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dawidwys commented on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616363030 I understood the underlying issue. I don't get though how this PR works around the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys edited a comment on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dawidwys edited a comment on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616363030 I understood the underlying issue. I don't get though how this PR works the issue around. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
flinkbot edited a comment on issue #11692: URL: https://github.com/apache/flink/pull/11692#issuecomment-611530487 ## CI report: * 4c854e0afe5cb934d919f37eb0891719063938ac Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160752603) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7683) * aa7f40b60eb7b67fd3e2ec6cd4be5890e2218ba8 UNKNOWN * 769e5d807f5c6671c0be5f39bc0309024d1caac2 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160984394) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7749) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys edited a comment on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dawidwys edited a comment on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616363030 I understood the underlying issue. I don't get though how this PR works the issue around. How do the adjustments make it less probable? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on issue #11816: [FLINK-16763][python][doc] Replace BatchTableEnvironment with StreamTableEnvironment for Python UDF examples
flinkbot commented on issue #11816: URL: https://github.com/apache/flink/pull/11816#issuecomment-616363629 ## CI report: * 0ada8ce36bd618fa5e9c869ca6cf0e4c059dda92 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11815: [FLINK-17253] Support viewfs for hadoop version < 2.7
flinkbot edited a comment on issue #11815: URL: https://github.com/apache/flink/pull/11815#issuecomment-616355624 ## CI report: * 82b7e411419bbd8a1011bdc78e2a916dbc1256a7 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160984484) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7750) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KurtYoung commented on a change in pull request #11397: [FLINK-16217] [sql-client] catch all exceptions to avoid SQL client crashed
KurtYoung commented on a change in pull request #11397:
URL: https://github.com/apache/flink/pull/11397#discussion_r411156938
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
##
@@ -196,8 +196,12 @@ public void open() {
if (line == null) {
continue;
}
- final Optional cmdCall =
parseCommand(line);
- cmdCall.ifPresent(this::callCommand);
+ try {
+ final Optional cmdCall =
parseCommand(line);
+ cmdCall.ifPresent(this::callCommand);
+ } catch (Throwable t) {
Review comment:
> The process might be in a bad state then
This makes sense, there is a risk that we will do more damage if don't
exist.
##
File path:
flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java
##
@@ -196,8 +196,12 @@ public void open() {
if (line == null) {
continue;
}
- final Optional cmdCall =
parseCommand(line);
- cmdCall.ifPresent(this::callCommand);
+ try {
+ final Optional cmdCall =
parseCommand(line);
+ cmdCall.ifPresent(this::callCommand);
+ } catch (Throwable t) {
Review comment:
> The process might be in a bad state then
This makes sense, there is a risk that we will do more damage if don't exit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
twalthr commented on a change in pull request #11692:
URL: https://github.com/apache/flink/pull/11692#discussion_r411158004
##
File path:
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/table/connector/source/abilities/PeriodicWatermarkAssignerProvider.java
##
@@ -28,19 +28,14 @@
* generating watermarks in {@link ScanTableSource}.
*/
@PublicEvolving
-public final class PeriodicWatermarkAssignerProvider extends
SupportsWatermarkPushDown.WatermarkProvider {
+public interface PeriodicWatermarkAssignerProvider extends
SupportsWatermarkPushDown.WatermarkProvider {
- private final AssignerWithPeriodicWatermarks periodicAssigner;
-
- private
PeriodicWatermarkAssignerProvider(AssignerWithPeriodicWatermarks
periodicAssigner) {
- this.periodicAssigner = periodicAssigner;
- }
-
- public static PeriodicWatermarkAssignerProvider
of(AssignerWithPeriodicWatermarks periodicAssigner) {
- return new PeriodicWatermarkAssignerProvider(periodicAssigner);
+ /**
+* Helper method for creating a static provider.
+*/
+ static PeriodicWatermarkAssignerProvider
of(AssignerWithPeriodicWatermarks periodicAssigner) {
Review comment:
Yes, I agree. We can solve this using some internal class later. I will
remove those methods.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] GJL commented on a change in pull request #11770: [FLINK-17014][runtime] Implement PipelinedRegionSchedulingStrategy
GJL commented on a change in pull request #11770:
URL: https://github.com/apache/flink/pull/11770#discussion_r411148799
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/PipelinedRegionSchedulingStrategyTest.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.scheduler.ExecutionVertexDeploymentOption;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Unit tests for {@link PipelinedRegionSchedulingStrategy}.
+ */
+public class PipelinedRegionSchedulingStrategyTest extends TestLogger {
+
+ private TestingSchedulerOperations testingSchedulerOperation;
+
+ private int parallelism = 2;
Review comment:
Consider making this static final or convert to a local variable.
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/TestingSchedulingPipelinedRegion.java
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A simple implementation of {@link SchedulingPipelinedRegion} for testing.
+ */
+public class TestingSchedulingPipelinedRegion implements
SchedulingPipelinedRegion {
+
+ private final Map
regionVertices = new HashMap<>();
+
+ private final Set consumedPartitions
= new HashSet<>();
+
+ public TestingSchedulingPipelinedRegion(final
Set vertices) {
+ for (TestingSchedulingExecutionVertex vertex : vertices) {
+ regionVertices.put(vertex.getId(), vertex);
+
+ for (TestingSchedulingResultPartition consumedPartition
: vertex.getConsumedResults()) {
+ if
(!vertices.contains(consumedPartition.getProducer())) {
+
consumedPartitions.add(consumedPartition);
+ }
+ }
+ }
+ }
+
+ @Override
+ public Iterable getVertices() {
+ return regionVertices.values();
Review comment:
Consider wrapping it in `Collections.unmodifiableCollection()` for
immutability or returning a copy. Same in `getConsumedResults()`.
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/TestingSchedulingTopology.java
##
@@ -78,6 +83,31 @@ public TestingSchedulingResultPartition
getResultPartition(final IntermediateRes
return resultPartition;
}
+ @Override
+ public Iterable getAllPipelinedRegions() {
+ return
vertexRegions.values().stream().collect(Collectors.toSet());
+ }
+
+ @Override
+ public Sche
[GitHub] [flink] hequn8128 commented on a change in pull request #11668: [FLINK-17013][python] Support Python UDTF in old planner under batch mode
hequn8128 commented on a change in pull request #11668:
URL: https://github.com/apache/flink/pull/11668#discussion_r411152627
##
File path: flink-python/pyflink/table/tests/test_udtf.py
##
@@ -41,44 +41,67 @@ def test_table_function(self):
DataTypes.BIGINT()))
t = self.t_env.from_elements([(1, 1, 3), (2, 1, 6), (3, 2, 9)], ['a',
'b', 'c'])
-t.join_lateral("multi_emit(a, multi_num(b)) as (x, y)") \
+t = t.join_lateral("multi_emit(a, multi_num(b)) as (x, y)") \
.left_outer_join_lateral("condition_multi_emit(x, y) as m") \
-.select("x, y, m") \
-.insert_into("Results")
-self.t_env.execute("test")
-actual = source_sink_utils.results()
+.select("x, y, m")
+actual = self._get_output(t)
self.assert_equals(actual,
["1,0,null", "1,1,null", "2,0,null", "2,1,null",
"3,0,0", "3,0,1",
"3,0,2", "3,1,1", "3,1,2", "3,2,2", "3,3,null"])
def test_table_function_with_sql_query(self):
-table_sink = source_sink_utils.TestAppendSink(
+self._register_table_sink(
['a', 'b', 'c'],
[DataTypes.BIGINT(), DataTypes.BIGINT(), DataTypes.BIGINT()])
-self.t_env.register_table_sink("Results", table_sink)
+
self.t_env.register_function(
"multi_emit", udtf(MultiEmit(), [DataTypes.BIGINT(),
DataTypes.BIGINT()],
[DataTypes.BIGINT(), DataTypes.BIGINT()]))
t = self.t_env.from_elements([(1, 1, 3), (2, 1, 6), (3, 2, 9)], ['a',
'b', 'c'])
self.t_env.register_table("MyTable", t)
-self.t_env.sql_query(
+t = self.t_env.sql_query(
"SELECT a, x, y FROM MyTable LEFT JOIN LATERAL TABLE(multi_emit(a,
b)) as T(x, y)"
-" ON TRUE") \
-.insert_into("Results")
-self.t_env.execute("test")
-actual = source_sink_utils.results()
+" ON TRUE")
+actual = self._get_output(t)
self.assert_equals(actual, ["1,1,0", "2,2,0", "3,3,0", "3,3,1"])
+def _register_table_sink(self, field_names: list, field_types: list):
+pass
+
+def _get_output(self, t):
+pass
+
class PyFlinkStreamUserDefinedTableFunctionTests(UserDefinedTableFunctionTests,
PyFlinkStreamTableTestCase):
-pass
+
+def _register_table_sink(self, field_names: list, field_types: list):
+table_sink = source_sink_utils.TestAppendSink(field_names, field_types)
+self.t_env.register_table_sink("Results", table_sink)
+
+def _get_output(self, t):
+t.insert_into("Results")
+self.t_env.execute("test")
+return source_sink_utils.results()
Review comment:
How about making this two as the default implementation in the base
class and only override them in the `PyFlinkBatchUserDefinedTableFunctionTests`?
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/plan/rules/AbstractPythonCorrelateRuleBase.java
##
@@ -69,35 +67,29 @@ public boolean matches(RelOptRuleCall call) {
return false;
}
- @Override
- public RelNode convert(RelNode rel) {
- DataStreamPythonCorrelateFactory factory = new
DataStreamPythonCorrelateFactory(rel);
- return factory.convertToCorrelate();
- }
-
/**
-* The factory is responsible to creating {@link
DataStreamPythonCorrelate}.
+* The abstract factory is responsible to creating {@link
DataSetPythonCorrelate} or {@link DataStreamPythonCorrelate}.
Review comment:
is responsible for creating
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/plan/rules/stream/DataStreamPythonCorrelateRule.java
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.plan.rules.stream;
+
+import org.apache.flink.table.plan.nodes.FlinkConventions;
+import org.apache.flink.table.plan.nodes.datastream.DataStreamPythonCorrelate;
+import org.apache.flink.t
[jira] [Created] (FLINK-17257) AbstractYarnClusterTest does not compile with Hadoop 2.10
Xintong Song created FLINK-17257:
Summary: AbstractYarnClusterTest does not compile with Hadoop 2.10
Key: FLINK-17257
URL: https://issues.apache.org/jira/browse/FLINK-17257
Project: Flink
Issue Type: Bug
Components: Deployment / YARN, Tests
Affects Versions: 1.9.3, 1.10.1, 1.11.0
Reporter: Xintong Song
Fix For: 1.11.0, 1.10.2, 1.9.4
In {{AbstractYarnClusterTest}}, we create {{ApplicationReport}} with the static
method {{ApplicationReport.newInstance}}, which is annotated as private and
unstable. This method is no longer compatible in Hadoop 2.10.
As a workaround, we can create {{ApplicationReport}} with its default
constructor and set only the fields that we need.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17258) Enable unaligned checkpoints in tests by default
Arvid Heise created FLINK-17258: --- Summary: Enable unaligned checkpoints in tests by default Key: FLINK-17258 URL: https://issues.apache.org/jira/browse/FLINK-17258 Project: Flink Issue Type: Sub-task Components: Tests Affects Versions: 1.11.0 Reporter: Arvid Heise Assignee: Arvid Heise Fix For: 1.11.0 To harden unaligned checkpoints as quickly as possible, we want to enable it by default for all tests similarly how we did it for credit-based flow control network stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17259) Have scala 2.12 support
João Boto created FLINK-17259: - Summary: Have scala 2.12 support Key: FLINK-17259 URL: https://issues.apache.org/jira/browse/FLINK-17259 Project: Flink Issue Type: Improvement Components: Stateful Functions Affects Versions: 2.0.0 Reporter: João Boto In statefun-flink is defined the scala.binary.version as 2.11 this force to use this the use of scala 2.11 should be the default 2.12? or have the option to chose the scala version -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-17257) AbstractYarnClusterTest does not compile with Hadoop 2.10
[
https://issues.apache.org/jira/browse/FLINK-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu reassigned FLINK-17257:
---
Assignee: Xintong Song
> AbstractYarnClusterTest does not compile with Hadoop 2.10
> -
>
> Key: FLINK-17257
> URL: https://issues.apache.org/jira/browse/FLINK-17257
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.9.3, 1.10.1, 1.11.0
>Reporter: Xintong Song
>Assignee: Xintong Song
>Priority: Major
> Fix For: 1.11.0, 1.10.2, 1.9.4
>
>
> In {{AbstractYarnClusterTest}}, we create {{ApplicationReport}} with the
> static method {{ApplicationReport.newInstance}}, which is annotated as
> private and unstable. This method is no longer compatible in Hadoop 2.10.
> As a workaround, we can create {{ApplicationReport}} with its default
> constructor and set only the fields that we need.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17012) Expose stage of task initialization
[ https://issues.apache.org/jira/browse/FLINK-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087462#comment-17087462 ] Stephan Ewen commented on FLINK-17012: -- It is true that this needs a constructor written in a way to "clean up after itself" on exception. But the code for that is pretty much the same (can actually be a bit simpler) as in an "initialize()" method and a "cleanup()" method where the "cleanup()" method needs to understand how far the "initialize()" method got. > Expose stage of task initialization > --- > > Key: FLINK-17012 > URL: https://issues.apache.org/jira/browse/FLINK-17012 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Task >Reporter: Wenlong Lyu >Priority: Major > > Currently a task switches to running before fully initialized, does not take > state initialization and operator initialization(#open ) in to account, which > may take long time to finish. As a result, there would be a weird phenomenon > that all tasks are running but throughput is 0. > I think it could be good if we can expose the initialization stage of tasks. > What to you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
flinkbot edited a comment on issue #11615: URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339 ## CI report: * 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN * 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN * 778239dfa00a84ab8457d49eedc759b601028676 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160682224) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632) * 2a47b5818c6bf072ab36f47130392f46a9bd4df8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160982187) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7747) * 3e865da06ab6dcc3a6ef1e76328e60fd06615144 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17012) Expose stage of task initialization
[ https://issues.apache.org/jira/browse/FLINK-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087463#comment-17087463 ] Stephan Ewen commented on FLINK-17012: -- [~pnowojski] Given that you work on the StreamTask a lot - would also be interesting to hear your thoughts on this. > Expose stage of task initialization > --- > > Key: FLINK-17012 > URL: https://issues.apache.org/jira/browse/FLINK-17012 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics, Runtime / Task >Reporter: Wenlong Lyu >Priority: Major > > Currently a task switches to running before fully initialized, does not take > state initialization and operator initialization(#open ) in to account, which > may take long time to finish. As a result, there would be a weird phenomenon > that all tasks are running but throughput is 0. > I think it could be good if we can expose the initialization stage of tasks. > What to you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11785: [FLINK-17206][table] refactor function catalog to support delayed UDF initialization.
flinkbot edited a comment on issue #11785: URL: https://github.com/apache/flink/pull/11785#issuecomment-615087743 ## CI report: * 131d640159c9f367ef0d682b4616228d6bed939d Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160979463) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7746) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11816: [FLINK-16763][python][doc] Replace BatchTableEnvironment with StreamTableEnvironment for Python UDF examples
flinkbot edited a comment on issue #11816: URL: https://github.com/apache/flink/pull/11816#issuecomment-616363629 ## CI report: * 0ada8ce36bd618fa5e9c869ca6cf0e4c059dda92 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160986902) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong opened a new pull request #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
xintongsong opened a new pull request #11817: URL: https://github.com/apache/flink/pull/11817 ## What is the purpose of the change In AbstractYarnClusterTest, we create ApplicationReport with the static method ApplicationReport.newInstance, which is annotated as private and unstable. This method is no longer compatible in Hadoop 2.10. As a workaround, we can create ApplicationReport with its default constructor and set only the fields that we need. ## Verifying this change This is a pure testing change which does not affect the production codes. Manually tested with Hadoop 2.4, 2.8, 2.10 and 3.2. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on issue #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
xintongsong commented on issue #11817: URL: https://github.com/apache/flink/pull/11817#issuecomment-616376921 cc @zhuzhurk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17257) AbstractYarnClusterTest does not compile with Hadoop 2.10
[
https://issues.apache.org/jira/browse/FLINK-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-17257:
---
Labels: pull-request-available (was: )
> AbstractYarnClusterTest does not compile with Hadoop 2.10
> -
>
> Key: FLINK-17257
> URL: https://issues.apache.org/jira/browse/FLINK-17257
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.9.3, 1.10.1, 1.11.0
>Reporter: Xintong Song
>Assignee: Xintong Song
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0, 1.10.2, 1.9.4
>
>
> In {{AbstractYarnClusterTest}}, we create {{ApplicationReport}} with the
> static method {{ApplicationReport.newInstance}}, which is annotated as
> private and unstable. This method is no longer compatible in Hadoop 2.10.
> As a workaround, we can create {{ApplicationReport}} with its default
> constructor and set only the fields that we need.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #11770: [FLINK-17014][runtime] Implement PipelinedRegionSchedulingStrategy
zhuzhurk commented on a change in pull request #11770:
URL: https://github.com/apache/flink/pull/11770#discussion_r411170916
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/PipelinedRegionSchedulingStrategyTest.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.scheduler.ExecutionVertexDeploymentOption;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Unit tests for {@link PipelinedRegionSchedulingStrategy}.
+ */
+public class PipelinedRegionSchedulingStrategyTest extends TestLogger {
+
+ private TestingSchedulerOperations testingSchedulerOperation;
+
+ private int parallelism = 2;
Review comment:
Ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on issue #11785: [FLINK-17206][table] refactor function catalog to support delayed UDF initialization.
dawidwys commented on issue #11785:
URL: https://github.com/apache/flink/pull/11785#issuecomment-616377734
Hi @WeiZhong94 thank you for working on this. Personally I would choose a
different approach in a couple of places. Let me describe how I see it.
The ultimate goal is to have:
```
private final Map tempSystemFunctions = new
LinkedHashMap<>();
private final Map tempCatalogFunctions =
new LinkedHashMap<>();
```
To get there we should:
1. We should not instantiate `FunctionDefinitions` by default. Similarly as
you are doing with the `UninstantiatedSystemFunction`. Instead of
`UninstantiatedSystemFunction` we should use `CatalogFunctionImpl` though.
2. We should add a `CatalogFunction` that internally holds an instantiated
`FunctionDefinition`. (for cases like `.select(new ScalarFunction(...))`) . We
could call it maybe: `InlineCatalogFunction`. Better name welcome. It could
look sth like this:
```
class InlineCatalogFunction implements CatalogFunction {
private final FunctionDefinition definition;
...
}
```
Having those we can use the maps as described in the beginning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-13860) Flink Apache Kudu Connector
[ https://issues.apache.org/jira/browse/FLINK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] João Boto closed FLINK-13860. - Resolution: Invalid > Flink Apache Kudu Connector > --- > > Key: FLINK-13860 > URL: https://issues.apache.org/jira/browse/FLINK-13860 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: João Boto >Priority: Major > > Hi.. > I'm the contributor and maintainer of this connector on Bahir-Flink project > [https://github.com/apache/bahir-flink/tree/master/flink-connector-kudu] > > but seems that flink-connectors on that project are less maintained an its > difficult to maintain the code up to date, as PR take a while to be merged > and never released any version, which makes it difficult to use easily > > I would like to contribute that code to flink allowing other to contribute > and use that connector > > [~fhueske] what do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
flinkbot commented on issue #11817: URL: https://github.com/apache/flink/pull/11817#issuecomment-616377968 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 19e4aa4ab7974adb5b771af4a9a51127c71e0bb4 (Mon Apr 20 07:58:01 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #11770: [FLINK-17014][runtime] Implement PipelinedRegionSchedulingStrategy
zhuzhurk commented on a change in pull request #11770:
URL: https://github.com/apache/flink/pull/11770#discussion_r411171776
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/TestingSchedulingPipelinedRegion.java
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A simple implementation of {@link SchedulingPipelinedRegion} for testing.
+ */
+public class TestingSchedulingPipelinedRegion implements
SchedulingPipelinedRegion {
+
+ private final Map
regionVertices = new HashMap<>();
+
+ private final Set consumedPartitions
= new HashSet<>();
+
+ public TestingSchedulingPipelinedRegion(final
Set vertices) {
+ for (TestingSchedulingExecutionVertex vertex : vertices) {
+ regionVertices.put(vertex.getId(), vertex);
+
+ for (TestingSchedulingResultPartition consumedPartition
: vertex.getConsumedResults()) {
+ if
(!vertices.contains(consumedPartition.getProducer())) {
+
consumedPartitions.add(consumedPartition);
+ }
+ }
+ }
+ }
+
+ @Override
+ public Iterable getVertices() {
+ return regionVertices.values();
Review comment:
Iterable is considered as unmodifiable so I think it's not very needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys removed a comment on issue #11785: [FLINK-17206][table] refactor function catalog to support delayed UDF initialization.
dawidwys removed a comment on issue #11785:
URL: https://github.com/apache/flink/pull/11785#issuecomment-616377734
Hi @WeiZhong94 thank you for working on this. Personally I would choose a
different approach in a couple of places. Let me describe how I see it.
The ultimate goal is to have:
```
private final Map tempSystemFunctions = new
LinkedHashMap<>();
private final Map tempCatalogFunctions =
new LinkedHashMap<>();
```
To get there we should:
1. We should not instantiate `FunctionDefinitions` by default. Similarly as
you are doing with the `UninstantiatedSystemFunction`. Instead of
`UninstantiatedSystemFunction` we should use `CatalogFunctionImpl` though.
2. We should add a `CatalogFunction` that internally holds an instantiated
`FunctionDefinition`. (for cases like `.select(new ScalarFunction(...))`) . We
could call it maybe: `InlineCatalogFunction`. Better name welcome. It could
look sth like this:
```
class InlineCatalogFunction implements CatalogFunction {
private final FunctionDefinition definition;
...
}
```
Having those we can use the maps as described in the beginning.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-11547) JsonMappingException in DynamoDBStreamsSchema
[
https://issues.apache.org/jira/browse/FLINK-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087466#comment-17087466
]
Jiawei Wu commented on FLINK-11547:
---
+1 I have seen a similiar issue. The proposed fix solved the issue. Is anyone
working on the PR?
> JsonMappingException in DynamoDBStreamsSchema
> -
>
> Key: FLINK-11547
> URL: https://issues.apache.org/jira/browse/FLINK-11547
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.8.0
>Reporter: Devin Thomson
>Priority: Major
> Attachments: full_flink_trace.txt
>
>
> If DynamoDBStreamsSchema is used as the deserializer for
> FlinkDynamoDBStreamsConsumer, an exception occurs during deserialization of a
> record. The stack trace is attached.
>
> This is a blocker for using DynamoDBStreamsSchema, but can be worked around
> by implementing a custom deserializer. The issue appears to be due to the
> usage of vanilla ObjectMapper:
>
> {code:java}
> private static final ObjectMapper MAPPER = new ObjectMapper();
> {code}
> When it should be using the appropriate mix-ins offered by the dynamodb
> stream adapter library:
> {code:java}
> private static final ObjectMapper MAPPER = new RecordObjectMapper();
> {code}
> This appears to resolve the issue, I tested by using my own deserializer
> implementation.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17260) StreamingKafkaITCase failure on Azure
Roman Khachatryan created FLINK-17260:
-
Summary: StreamingKafkaITCase failure on Azure
Key: FLINK-17260
URL: https://issues.apache.org/jira/browse/FLINK-17260
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka, Tests
Affects Versions: 1.11.0
Reporter: Roman Khachatryan
https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/7544/logs/165
{code:java}
2020-04-16T00:12:32.2848429Z [INFO] Running
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-04-16T00:14:47.9100927Z [ERROR] Tests run: 3, Failures: 1, Errors: 0,
Skipped: 0, Time elapsed: 135.621 s <<< FAILURE! - in
org.apache.flink.tests.util.k afka.StreamingKafkaITCase
2020-04-16T00:14:47.9103036Z [ERROR] testKafka[0:
kafka-version:0.10.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)
Time elapsed: 46.222 s <<< FAILURE!
2020-04-16T00:14:47.9104033Z java.lang.AssertionError:
expected:<[elephant,27,64213]> but was:<[]>
2020-04-16T00:14:47.9104638Zat org.junit.Assert.fail(Assert.java:88)
2020-04-16T00:14:47.9105148Zat
org.junit.Assert.failNotEquals(Assert.java:834)
2020-04-16T00:14:47.9105701Zat
org.junit.Assert.assertEquals(Assert.java:118)
2020-04-16T00:14:47.9106239Zat
org.junit.Assert.assertEquals(Assert.java:144)
2020-04-16T00:14:47.9107177Zat
org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:162)
2020-04-16T00:14:47.9107845Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-04-16T00:14:47.9108434Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-04-16T00:14:47.9109318Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-04-16T00:14:47.9109914Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-04-16T00:14:47.9110434Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-04-16T00:14:47.9110985Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-04-16T00:14:47.9111548Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-04-16T00:14:47.9112083Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-04-16T00:14:47.9112629Zat
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-04-16T00:14:47.9113145Zat
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-04-16T00:14:47.9113637Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-04-16T00:14:47.9114072Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-04-16T00:14:47.9114490Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-04-16T00:14:47.9115256Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-04-16T00:14:47.9115791Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-04-16T00:14:47.9116292Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-04-16T00:14:47.9116736Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-04-16T00:14:47.9117779Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-04-16T00:14:47.9118274Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-04-16T00:14:47.9118766Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-04-16T00:14:47.9119204Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-04-16T00:14:47.9119625Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2020-04-16T00:14:47.9120005Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2020-04-16T00:14:47.9120428Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-04-16T00:14:47.9120876Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-04-16T00:14:47.9121350Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-04-16T00:14:47.9121805Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-04-16T00:14:47.9122273Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-04-16T00:14:47.9122729Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-04-16T00:14:47.9123130Zat
org.junit.runners.Suite.runChild(Suite.java:128)
...
2020-04-16T00:14:47.9132530Z
2020-04-16T00:14:47.9134982Z [INFO] Running
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
2020-04-16T00:17:18.7332734Z [INFO] Tests run: 3, Failures: 0, Errors: 0,
Skipped: 0, Time elapsed: 150.813 s - in
org.apache.flink.tests.util.kafka.SQLClient KafkaITCase
2020-04-16T00:17:19.0840872Z [INFO]
2020-04-16T00:17:19.0841522Z [
[GitHub] [flink] zhuzhurk commented on a change in pull request #11770: [FLINK-17014][runtime] Implement PipelinedRegionSchedulingStrategy
zhuzhurk commented on a change in pull request #11770:
URL: https://github.com/apache/flink/pull/11770#discussion_r411173942
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/TestingSchedulingTopology.java
##
@@ -78,6 +83,31 @@ public TestingSchedulingResultPartition
getResultPartition(final IntermediateRes
return resultPartition;
}
+ @Override
+ public Iterable getAllPipelinedRegions() {
+ return
vertexRegions.values().stream().collect(Collectors.toSet());
Review comment:
Ok. Let's simplify it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] WeiZhong94 commented on issue #11768: [FLINK-16943][python] Support set the configuration option "pipeline.jars" in PyFlink.
WeiZhong94 commented on issue #11768: URL: https://github.com/apache/flink/pull/11768#issuecomment-616379782 @dianfu The documentation has been added in the latest commit, please take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17260) StreamingKafkaITCase failure on Azure
[
https://issues.apache.org/jira/browse/FLINK-17260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-17260:
--
Description:
[https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/7544/logs/165]
{code:java}
2020-04-16T00:12:32.2848429Z [INFO] Running
org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-04-16T00:14:47.9100927Z [ERROR] Tests run: 3, Failures: 1, Errors: 0,
Skipped: 0, Time elapsed: 135.621 s <<< FAILURE! - in
org.apache.flink.tests.util.k afka.StreamingKafkaITCase
2020-04-16T00:14:47.9103036Z [ERROR] testKafka[0:
kafka-version:0.10.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)
Time elapsed: 46.222 s <<< FAILURE!
2020-04-16T00:14:47.9104033Z java.lang.AssertionError:
expected:<[elephant,27,64213]> but was:<[]>
2020-04-16T00:14:47.9104638Zat org.junit.Assert.fail(Assert.java:88)
2020-04-16T00:14:47.9105148Zat
org.junit.Assert.failNotEquals(Assert.java:834)
2020-04-16T00:14:47.9105701Zat
org.junit.Assert.assertEquals(Assert.java:118)
2020-04-16T00:14:47.9106239Zat
org.junit.Assert.assertEquals(Assert.java:144)
2020-04-16T00:14:47.9107177Zat
org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:162)
2020-04-16T00:14:47.9107845Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-04-16T00:14:47.9108434Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-04-16T00:14:47.9109318Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-04-16T00:14:47.9109914Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-04-16T00:14:47.9110434Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-04-16T00:14:47.9110985Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-04-16T00:14:47.9111548Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-04-16T00:14:47.9112083Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-04-16T00:14:47.9112629Zat
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-04-16T00:14:47.9113145Zat
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
2020-04-16T00:14:47.9113637Zat
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-04-16T00:14:47.9114072Zat
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-04-16T00:14:47.9114490Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-04-16T00:14:47.9115256Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-04-16T00:14:47.9115791Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-04-16T00:14:47.9116292Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-04-16T00:14:47.9116736Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-04-16T00:14:47.9117779Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-04-16T00:14:47.9118274Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-04-16T00:14:47.9118766Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-04-16T00:14:47.9119204Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-04-16T00:14:47.9119625Zat
org.junit.runners.Suite.runChild(Suite.java:128)
2020-04-16T00:14:47.9120005Zat
org.junit.runners.Suite.runChild(Suite.java:27)
2020-04-16T00:14:47.9120428Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-04-16T00:14:47.9120876Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-04-16T00:14:47.9121350Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-04-16T00:14:47.9121805Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-04-16T00:14:47.9122273Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-04-16T00:14:47.9122729Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-04-16T00:14:47.9123130Zat
org.junit.runners.Suite.runChild(Suite.java:128)
...
2020-04-16T00:14:47.9132530Z
2020-04-16T00:14:47.9134982Z [INFO] Running
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
2020-04-16T00:17:18.7332734Z [INFO] Tests run: 3, Failures: 0, Errors: 0,
Skipped: 0, Time elapsed: 150.813 s - in
org.apache.flink.tests.util.kafka.SQLClient KafkaITCase
2020-04-16T00:17:19.0840872Z [INFO]
2020-04-16T00:17:19.0841522Z [INFO] Results:
2020-04-16T00:17:19.0841820Z [INFO]
2020-04-16T00:17:19.0842133Z [ERROR] Failures:
2020-04-16T00:17:19.0842565Z [ERROR] StreamingKafkaITCase.testKafka:162
expected:<[elephant,
[GitHub] [flink] rkhachatryan commented on issue #11754: [FLINK-17156][checkpointing] support cancellation of unaligned checkpoints
rkhachatryan commented on issue #11754: URL: https://github.com/apache/flink/pull/11754#issuecomment-616381588 > There is a test failure: https://dev.azure.com ... Created an issue for that https://issues.apache.org/jira/browse/FLINK-17260. Subsequent azure builds succeeded. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17261) Statefun docs broken
Chesnay Schepler created FLINK-17261:
Summary: Statefun docs broken
Key: FLINK-17261
URL: https://issues.apache.org/jira/browse/FLINK-17261
Project: Flink
Issue Type: Bug
Components: Documentation
Affects Versions: 2.0.0, 2.1.0
Reporter: Chesnay Schepler
https://ci.apache.org/builders/flink-statefun-docs-master/builds/56
{code} Liquid Exception: Liquid syntax error (line 67): Unknown tag 'higlight'
in deployment-and-operations/packaging.md
Liquid syntax error (line 67): Unknown tag 'higlight'
/home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
`unknown_tag'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #11770: [FLINK-17014][runtime] Implement PipelinedRegionSchedulingStrategy
zhuzhurk commented on a change in pull request #11770:
URL: https://github.com/apache/flink/pull/11770#discussion_r411177132
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/TestingSchedulingTopology.java
##
@@ -78,6 +83,31 @@ public TestingSchedulingResultPartition
getResultPartition(final IntermediateRes
return resultPartition;
}
+ @Override
+ public Iterable getAllPipelinedRegions() {
+ return
vertexRegions.values().stream().collect(Collectors.toSet());
+ }
+
+ @Override
+ public SchedulingPipelinedRegion
getPipelinedRegionOfVertex(ExecutionVertexID vertexId) {
+ return vertexRegions.get(vertexId);
+ }
+
+ void generatePipelinedRegions() {
Review comment:
Yes that would be better.
The only problem is that the regions cannot be refreshed if the topology is
changed after invoking `getAllPipelinedRegions()`. So I will clear the cached
regions on any topology changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17189) Table with processing time attribute can not be read from Hive catalog
[
https://issues.apache.org/jira/browse/FLINK-17189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087472#comment-17087472
]
Rui Li commented on FLINK-17189:
Hi [~twalthr], thanks for the explanations. Then I think we can store the
{{PROCTIME}} info in {{DescriptorProperties}}, similar to how we store
watermark specs. And I think we just need to store the names of fields which
are processing time. When deserializing the table schema, we check whether a
field is processing time and if it is, we update the filed type to set the
{{TimestampKind}}. Does that make sense to you?
> Table with processing time attribute can not be read from Hive catalog
> --
>
> Key: FLINK-17189
> URL: https://issues.apache.org/jira/browse/FLINK-17189
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem, Table SQL / Planner
>Reporter: Timo Walther
>Priority: Major
> Fix For: 1.11.0, 1.10.2
>
>
> DDL:
> {code}
> CREATE TABLE PROD_LINEITEM (
> L_ORDERKEY INTEGER,
> L_PARTKEYINTEGER,
> L_SUPPKEYINTEGER,
> L_LINENUMBER INTEGER,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAXDOUBLE,
> L_CURRENCY STRING,
> L_RETURNFLAG STRING,
> L_LINESTATUS STRING,
> L_ORDERTIME TIMESTAMP(3),
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENTSTRING,
> WATERMARK FOR L_ORDERTIME AS L_ORDERTIME - INTERVAL '5' MINUTE,
> L_PROCTIME AS PROCTIME()
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'Lineitem',
> 'connector.properties.zookeeper.connect' = 'not-needed',
> 'connector.properties.bootstrap.servers' = 'kafka:9092',
> 'connector.startup-mode' = 'earliest-offset',
> 'format.type' = 'csv',
> 'format.field-delimiter' = '|'
> );
> {code}
> Query:
> {code}
> SELECT * FROM prod_lineitem;
> {code}
> Result:
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.AssertionError: Conversion to relational algebra failed to preserve
> datatypes:
> validated type:
> RecordType(INTEGER L_ORDERKEY, INTEGER L_PARTKEY, INTEGER L_SUPPKEY, INTEGER
> L_LINENUMBER, DOUBLE L_QUANTITY, DOUBLE L_EXTENDEDPRICE, DOUBLE L_DISCOUNT,
> DOUBLE L_TAX, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_CURRENCY,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_RETURNFLAG,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_LINESTATUS, TIME
> ATTRIBUTE(ROWTIME) L_ORDERTIME, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> L_SHIPINSTRUCT, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_SHIPMODE,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_COMMENT, TIMESTAMP(3) NOT NULL
> L_PROCTIME) NOT NULL
> converted type:
> RecordType(INTEGER L_ORDERKEY, INTEGER L_PARTKEY, INTEGER L_SUPPKEY, INTEGER
> L_LINENUMBER, DOUBLE L_QUANTITY, DOUBLE L_EXTENDEDPRICE, DOUBLE L_DISCOUNT,
> DOUBLE L_TAX, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_CURRENCY,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_RETURNFLAG,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_LINESTATUS, TIME
> ATTRIBUTE(ROWTIME) L_ORDERTIME, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> L_SHIPINSTRUCT, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_SHIPMODE,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" L_COMMENT, TIME
> ATTRIBUTE(PROCTIME) NOT NULL L_PROCTIME) NOT NULL
> rel:
> LogicalProject(L_ORDERKEY=[$0], L_PARTKEY=[$1], L_SUPPKEY=[$2],
> L_LINENUMBER=[$3], L_QUANTITY=[$4], L_EXTENDEDPRICE=[$5], L_DISCOUNT=[$6],
> L_TAX=[$7], L_CURRENCY=[$8], L_RETURNFLAG=[$9], L_LINESTATUS=[$10],
> L_ORDERTIME=[$11], L_SHIPINSTRUCT=[$12], L_SHIPMODE=[$13], L_COMMENT=[$14],
> L_PROCTIME=[$15])
> LogicalWatermarkAssigner(rowtime=[L_ORDERTIME], watermark=[-($11,
> 30:INTERVAL MINUTE)])
> LogicalProject(L_ORDERKEY=[$0], L_PARTKEY=[$1], L_SUPPKEY=[$2],
> L_LINENUMBER=[$3], L_QUANTITY=[$4], L_EXTENDEDPRICE=[$5], L_DISCOUNT=[$6],
> L_TAX=[$7], L_CURRENCY=[$8], L_RETURNFLAG=[$9], L_LINESTATUS=[$10],
> L_ORDERTIME=[$11], L_SHIPINSTRUCT=[$12], L_SHIPMODE=[$13], L_COMMENT=[$14],
> L_PROCTIME=[PROCTIME()])
> LogicalTableScan(table=[[hcat, default, prod_lineitem, source:
> [KafkaTableSource(L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY,
> L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_CURRENCY, L_RETURNFLAG, L_LINESTATUS,
> L_ORDERTIME, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT)]]])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17262) Statefun snapshot deployments broken
Chesnay Schepler created FLINK-17262:
Summary: Statefun snapshot deployments broken
Key: FLINK-17262
URL: https://issues.apache.org/jira/browse/FLINK-17262
Project: Flink
Issue Type: Bug
Components: Release System
Affects Versions: 2.0.0, 2.1.0
Reporter: Chesnay Schepler
https://builds.apache.org/job/flink-statefun-snapshot-deployment-2.0/12/
{code}
Commit message: "[FLINK-17193] [python-k8s-example] Abort script on failure.
Build SDK distribution if was not previously built"
> git rev-list --no-walk d660668ff45312f7c3b10529b29b478efe220e57 # timeout=10
[flink-statefun-snapshot-deployment-2.0] $ /bin/bash
/tmp/jenkins419419675999100733.sh
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 341 100 3410 0914 0 --:--:-- --:--:-- --:--:-- 914
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
./tools/snapshots/deploy_snapshot_jars.sh: line 47: mvn: command not found
{code}
Happened twice in a row.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17261) Statefun docs broken
[
https://issues.apache.org/jira/browse/FLINK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-17261:
-
Affects Version/s: (was: 2.1.0)
(was: 2.0.0)
statefun-2.1.0
statefun-2.0.0
> Statefun docs broken
>
>
> Key: FLINK-17261
> URL: https://issues.apache.org/jira/browse/FLINK-17261
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Priority: Major
>
> https://ci.apache.org/builders/flink-statefun-docs-master/builds/56
> {code} Liquid Exception: Liquid syntax error (line 67): Unknown tag
> 'higlight' in deployment-and-operations/packaging.md
> Liquid syntax error (line 67): Unknown tag 'higlight'
> /home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
> `unknown_tag'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #11568: [FLINK-16779][table] Add RAW type support in DDL and functions
twalthr commented on a change in pull request #11568:
URL: https://github.com/apache/flink/pull/11568#discussion_r411178550
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/calcite/FlinkTypeFactory.scala
##
@@ -484,6 +493,9 @@ object FlinkTypeFactory {
// CURSOR for UDTF case, whose type info will never be used, just a
placeholder
case CURSOR => new TypeInformationRawType[Nothing](new NothingTypeInfo)
+ case null if relDataType.isInstanceOf[RawRelDataType] =>
Review comment:
@danny0405 thanks for the hint around `SqlTypeName.OTHER`. This one fits
better to the semantics of the RAW type than `SqlTypeName.ANY`. The family `
SqlTypeFamily.ANY` includes all types which is not correct for the RAW type.
But `SqlTypeName.OTHER` has no type family which is correct. I'm fine with
using `SqlTypeName.OTHER`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-17262) Statefun snapshot deployments broken
[
https://issues.apache.org/jira/browse/FLINK-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-17262:
-
Affects Version/s: (was: 2.1.0)
(was: 2.0.0)
statefun-2.1.0
statefun-2.0.0
> Statefun snapshot deployments broken
>
>
> Key: FLINK-17262
> URL: https://issues.apache.org/jira/browse/FLINK-17262
> Project: Flink
> Issue Type: Bug
> Components: Release System
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Priority: Major
>
> https://builds.apache.org/job/flink-statefun-snapshot-deployment-2.0/12/
> {code}
> Commit message: "[FLINK-17193] [python-k8s-example] Abort script on failure.
> Build SDK distribution if was not previously built"
> > git rev-list --no-walk d660668ff45312f7c3b10529b29b478efe220e57 #
> timeout=10
> [flink-statefun-snapshot-deployment-2.0] $ /bin/bash
> /tmp/jenkins419419675999100733.sh
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 100 341 100 3410 0914 0 --:--:-- --:--:-- --:--:-- 914
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> ./tools/snapshots/deploy_snapshot_jars.sh: line 47: mvn: command not found
> {code}
> Happened twice in a row.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
flinkbot edited a comment on issue #11615: URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339 ## CI report: * 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN * 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN * 2a47b5818c6bf072ab36f47130392f46a9bd4df8 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160982187) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7747) * 3e865da06ab6dcc3a6ef1e76328e60fd06615144 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160990470) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7751) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11554: [FLINK-15101][connector/common] Add SourceCoordinator implementation.
flinkbot edited a comment on issue #11554: URL: https://github.com/apache/flink/pull/11554#issuecomment-605459909 ## CI report: * bd27b1b4dd97173d85bb0867f7c377ac4c638af5 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156295604) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6793) * 14e9fe3bfdeeae480047848801243f9fbed03cb4 UNKNOWN * 6a0a147d96dc517574efb6a97786b8b7e8e9c10c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11727: [FLINK-17106][table] Support create and drop view in Flink SQL
flinkbot edited a comment on issue #11727: URL: https://github.com/apache/flink/pull/11727#issuecomment-613273432 ## CI report: * fa5592cec0a6a5dcecc4f45c4bf72caf6e166eb4 UNKNOWN * 3ac7c714eacc6416b3dfbe163b4581d18e67de8d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160967398) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7743) * a3901b149f603721be62fbb3d8bf46143d08c3d1 UNKNOWN * 25bb852676993fd14ebccf710b35a5f49bd56fd6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11696: [FLINK-16658][FLINK-16660] Introduce the ApplicationDispatcherBootstrap and wire it to StandaloneJobEntrypoint
flinkbot edited a comment on issue #11696: URL: https://github.com/apache/flink/pull/11696#issuecomment-611702183 ## CI report: * 10446262ad30a5f2a2a743065f1106d5242fead4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160726688) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7662) * b44d92e51473c208a84700f58cf453042b7a237b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11768: [FLINK-16943][python] Support set the configuration option "pipeline.jars" in PyFlink.
flinkbot edited a comment on issue #11768: URL: https://github.com/apache/flink/pull/11768#issuecomment-614482552 ## CI report: * 01280e72a363ee72180dd385dc95e0013f60650f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160718616) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7657) * 375c382d79b0eb32388aa1b6b9c2f365a91344ee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11784: [FLINK-17120][python] Add Cython support for operations
flinkbot edited a comment on issue #11784: URL: https://github.com/apache/flink/pull/11784#issuecomment-615011156 ## CI report: * 1ee639f83adcf607b722609782be88207de5fe49 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160666122) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7622) * 6f1db3ebd65dcbeaebb677460838b22745b9d89a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on issue #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
flinkbot commented on issue #11817: URL: https://github.com/apache/flink/pull/11817#issuecomment-616385883 ## CI report: * 19e4aa4ab7974adb5b771af4a9a51127c71e0bb4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16478) add restApi to modify loglevel
[ https://issues.apache.org/jira/browse/FLINK-16478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087476#comment-17087476 ] xiaodao commented on FLINK-16478: - Hi [~felixzheng],[~fly_in_gis], i write a design doc, Help me check it, thanks. https://docs.google.com/document/d/19AyuTHeERP6JKmtHYnCdBw29LnZpRkbTS7K12q4OfbA/edit?usp=sharing > add restApi to modify loglevel > --- > > Key: FLINK-16478 > URL: https://issues.apache.org/jira/browse/FLINK-16478 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: xiaodao >Priority: Minor > > sometimes we may need to change loglevel to get more information to resolved > bug, now we need to stop it and modify conf/log4j.properties and resubmit it > ,i think it's better to add rest api to modify loglevel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu commented on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dianfu commented on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616387332 There are two caches in RelDataTypeFactoryImpl: KEY2TYPE_CACHE and DATATYPE_CACHE. KEY2TYPE_CACHE caches the mapping of Key(consists of field names and field types, etc) to RelDataType and can be used for the canonization of row types per my understanding. DATATYPE_CACHE caches the RelDataType instances. PythonCalcSplitRule will split a Calc RelNode which contains both non-vectorized Python UDF and vectorized Python UDF into two Calc RelNodes. For the failure test case, the output type of the bottom Calc consists of two fields (f0: INTEGER, f1: INTEGER), let's call it row_type_0. This row type is already available in the cache (generated by other test cases, it's held in variable KEY2TYPE_CACHE) and so it will hit the cache when constructing this row type. However, during debugging, I found that the INTEGER type referenced by row_type_0 is already cleaned up from the cache DATATYPE_CACHE. Then when constructing the RexProgram for the top Calc, it creates another INTEGER type and failure happens. To work around this problem, we adjust the test case a bit to make the output row type of the bottom Calc consisting of three fields instead of two fields to make the cache hit fail. It seems a little hack, however, it did could solve this problem. I'm glad to try if there is more elegant way to address this problem wholly in Flink which could avoid this problem thoroughly. Do you have any suggestions? Glad to hear! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17261) Statefun docs broken
[
https://issues.apache.org/jira/browse/FLINK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-17261:
---
Assignee: Tzu-Li (Gordon) Tai
> Statefun docs broken
>
>
> Key: FLINK-17261
> URL: https://issues.apache.org/jira/browse/FLINK-17261
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>
> https://ci.apache.org/builders/flink-statefun-docs-master/builds/56
> {code} Liquid Exception: Liquid syntax error (line 67): Unknown tag
> 'higlight' in deployment-and-operations/packaging.md
> Liquid syntax error (line 67): Unknown tag 'higlight'
> /home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
> `unknown_tag'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-17261) Statefun docs broken
[
https://issues.apache.org/jira/browse/FLINK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-17261:
---
Assignee: (was: Tzu-Li (Gordon) Tai)
> Statefun docs broken
>
>
> Key: FLINK-17261
> URL: https://issues.apache.org/jira/browse/FLINK-17261
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Priority: Major
>
> https://ci.apache.org/builders/flink-statefun-docs-master/builds/56
> {code} Liquid Exception: Liquid syntax error (line 67): Unknown tag
> 'higlight' in deployment-and-operations/packaging.md
> Liquid syntax error (line 67): Unknown tag 'higlight'
> /home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
> `unknown_tag'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
zhuzhurk commented on a change in pull request #11817:
URL: https://github.com/apache/flink/pull/11817#discussion_r411183367
##
File path:
flink-yarn/src/test/java/org/apache/flink/yarn/AbstractYarnClusterTest.java
##
@@ -82,26 +83,21 @@ private ApplicationReport createApplicationReport(
ApplicationId applicationId,
YarnApplicationState yarnApplicationState,
FinalApplicationStatus finalApplicationStatus) {
- return ApplicationReport.newInstance(
- applicationId,
- ApplicationAttemptId.newInstance(applicationId, 0),
- "user",
- "queue",
- "name",
- "localhost",
- 42,
- null,
- yarnApplicationState,
- null,
- null,
- 1L,
- 2L,
- finalApplicationStatus,
- null,
- null,
- 1.0f,
- null,
- null);
+
+ ApplicationReport applicationReport =
Records.newRecord(ApplicationReport.class);
Review comment:
Seems `Records.newRecord(...)` is still unstable. Is there a stable
approach to create `ApplicationReport`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17261) Statefun docs broken
[
https://issues.apache.org/jira/browse/FLINK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087478#comment-17087478
]
Tzu-Li (Gordon) Tai commented on FLINK-17261:
-
cc [~sjwiesman]
seems to be caused by recent commit -
https://github.com/apache/flink-statefun/commit/3f4cddc58d6211aa0035a7c6b2ba3ddce186b17e#diff-4d66388f79ce9ddca88b6a4784d17ea6R74
> Statefun docs broken
>
>
> Key: FLINK-17261
> URL: https://issues.apache.org/jira/browse/FLINK-17261
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Priority: Major
>
> https://ci.apache.org/builders/flink-statefun-docs-master/builds/56
> {code} Liquid Exception: Liquid syntax error (line 67): Unknown tag
> 'higlight' in deployment-and-operations/packaging.md
> Liquid syntax error (line 67): Unknown tag 'higlight'
> /home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
> `unknown_tag'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
zhuzhurk commented on a change in pull request #11817:
URL: https://github.com/apache/flink/pull/11817#discussion_r411183367
##
File path:
flink-yarn/src/test/java/org/apache/flink/yarn/AbstractYarnClusterTest.java
##
@@ -82,26 +83,21 @@ private ApplicationReport createApplicationReport(
ApplicationId applicationId,
YarnApplicationState yarnApplicationState,
FinalApplicationStatus finalApplicationStatus) {
- return ApplicationReport.newInstance(
- applicationId,
- ApplicationAttemptId.newInstance(applicationId, 0),
- "user",
- "queue",
- "name",
- "localhost",
- 42,
- null,
- yarnApplicationState,
- null,
- null,
- 1L,
- 2L,
- finalApplicationStatus,
- null,
- null,
- 1.0f,
- null,
- null);
+
+ ApplicationReport applicationReport =
Records.newRecord(ApplicationReport.class);
Review comment:
Seems `Records.newRecord(...)` is still private and unstable. Is there a
stable approach to create `ApplicationReport`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] twalthr commented on issue #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
twalthr commented on issue #11692: URL: https://github.com/apache/flink/pull/11692#issuecomment-616388463 Thanks @dawidwys. I will merge this once the build is green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on a change in pull request #11554: [FLINK-15101][connector/common] Add SourceCoordinator implementation.
becketqin commented on a change in pull request #11554:
URL: https://github.com/apache/flink/pull/11554#discussion_r410904702
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java
##
@@ -0,0 +1,230 @@
+/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.connector.source.ReaderInfo;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.connector.source.SourceSplit;
+import org.apache.flink.api.connector.source.SplitEnumerator;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+import org.apache.flink.runtime.operators.coordination.OperatorEvent;
+import org.apache.flink.runtime.source.event.ReaderRegistrationEvent;
+import org.apache.flink.runtime.source.event.SourceEventWrapper;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.ObjectInput;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutput;
+import java.io.ObjectOutputStream;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+
+/**
+ * The default implementation of the {@link OperatorCoordinator} for the
{@link Source}.
+ *
+ * The SourceCoordinator provides an event loop style thread
model to interact with
+ * the Flink runtime. The coordinator ensures that all the state manipulations
are made by its event loop
+ * thread. It also helps keep track of the necessary split assignments history
per subtask to simplify the
+ * {@link SplitEnumerator} implementation.
+ *
+ * The coordinator maintains a {@link
org.apache.flink.api.connector.source.SplitEnumeratorContext
+ * SplitEnumeratorContxt} and shares it with the enumerator. When the
coordinator receives an action
+ * request from the Flink runtime, it sets up the context, and calls
corresponding method of the
+ * SplitEnumerator to take actions.
+ */
+@Internal
+public class SourceCoordinator
implements OperatorCoordinator {
+ private static final Logger LOG =
LoggerFactory.getLogger(OperatorCoordinator.class);
+ /** A single-thread executor to handle all the changes to the
coordinator. */
+ private final ExecutorService coordinatorExecutor;
+ /** The Source that is associated with this SourceCoordinator. */
+ private final Source source;
+ /** The serializer that handles the serde of the SplitEnumerator
checkpoints. */
+ private final SimpleVersionedSerializer
enumCheckpointSerializer;
+ /** The serializer for the SourceSplit of the associated Source. */
+ private final SimpleVersionedSerializer splitSerializer;
+ /** The context containing the states of the coordinator. */
+ private final SourceCoordinatorContext context;
+ /** The split enumerator created from the associated Source. */
+ private SplitEnumerator enumerator;
+ /** A flag marking whether the coordinator has started. */
+ private boolean started;
+
+ public SourceCoordinator(
+ ExecutorService coordinatorExecutor,
+ Source source,
+ SourceCoordinatorContext context) {
+ this.coordinatorExecutor = coordinatorExecutor;
+ this.source = source;
+ this.enumCheckpointSerializer =
source.getEnumeratorCheckpointSerializer();
+ this.splitSerializer = source.getSplitSerializer();
+ this.context = context;
+ this.enumerator = source.createEnumerator(context);
+ this.started = false;
+ }
+
+ @Override
+ public void start() throws Exception {
+ LOG.info("Starting split enumerator.");
Review comment:
I like the idea. Makes a lot of sense.
##
File path:
flink-runtime/src/main/java/org/apache/flink/run
[GitHub] [flink] aljoscha commented on issue #11811: [FLINK-16658][FLINK-16660] Introduce the ApplicationDispatcherBootstrap and wire it to StandaloneJobEntrypoint
aljoscha commented on issue #11811: URL: https://github.com/apache/flink/pull/11811#issuecomment-616389339 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17190) SQL client does not support views that reference a table from DDL
[
https://issues.apache.org/jira/browse/FLINK-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087479#comment-17087479
]
JinxinTang commented on FLINK-17190:
Could you please provide version info and is it can be reproduced in local
environment?
> SQL client does not support views that reference a table from DDL
> -
>
> Key: FLINK-17190
> URL: https://issues.apache.org/jira/browse/FLINK-17190
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Timo Walther
>Priority: Major
>
> It seems to be a classloading issue when the view references a DDL table.
> {code}
> CREATE TABLE PROD_LINEITEM (
> L_ORDERKEY INTEGER,
> L_PARTKEYINTEGER,
> L_SUPPKEYINTEGER,
> L_LINENUMBER INTEGER,
> L_QUANTITY DOUBLE,
> L_EXTENDEDPRICE DOUBLE,
> L_DISCOUNT DOUBLE,
> L_TAXDOUBLE,
> L_CURRENCY STRING,
> L_RETURNFLAG STRING,
> L_LINESTATUS STRING,
> L_ORDERTIME TIMESTAMP(3),
> L_SHIPINSTRUCT STRING,
> L_SHIPMODE STRING,
> L_COMMENTSTRING,
> WATERMARK FOR L_ORDERTIME AS L_ORDERTIME - INTERVAL '5' MINUTE,
> L_PROCTIME AS PROCTIME()
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'Lineitem',
> 'connector.properties.zookeeper.connect' = 'not-needed',
> 'connector.properties.bootstrap.servers' = 'kafka:9092',
> 'connector.startup-mode' = 'earliest-offset',
> 'format.type' = 'csv',
> 'format.field-delimiter' = '|'
> );
> CREATE VIEW v AS SELECT * FROM PROD_LINEITEM;
> {code}
> Result:
> {code}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Unexpected exception. This is a bug. Please consider filing an issue.
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:190)
> Caused by: org.apache.flink.table.api.TableException:
> findAndCreateTableSource failed.
> at
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:55)
> at
> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:92)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.findAndCreateTableSource(CatalogSourceTable.scala:156)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.tableSource$lzycompute(CatalogSourceTable.scala:65)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.tableSource(CatalogSourceTable.scala:65)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.scala:76)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3328)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2357)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2005)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:646)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3181)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:148)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:135)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:522)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:436)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:154)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:66)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:464)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.addView(LocalExecutor.java:300)
> at
> org.apache.flink.table.client.cli.CliClient.callCreateView(CliClient.java:579)
> at
> org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:308)
> at java.util.Optional.ifPresent(Optional.java:159)
> at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200)
> at org.apache.
[GitHub] [flink] rkhachatryan commented on issue #11802: [FLINK-17197][runtime] switch ContinuousFileReaderOperator state to FAILED on error
rkhachatryan commented on issue #11802: URL: https://github.com/apache/flink/pull/11802#issuecomment-616389998 Thanks for testing it @KurtYoung! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17262) Statefun snapshot deployments broken
[
https://issues.apache.org/jira/browse/FLINK-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-17262:
---
Assignee: Tzu-Li (Gordon) Tai
> Statefun snapshot deployments broken
>
>
> Key: FLINK-17262
> URL: https://issues.apache.org/jira/browse/FLINK-17262
> Project: Flink
> Issue Type: Bug
> Components: Release System
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>
> https://builds.apache.org/job/flink-statefun-snapshot-deployment-2.0/12/
> {code}
> Commit message: "[FLINK-17193] [python-k8s-example] Abort script on failure.
> Build SDK distribution if was not previously built"
> > git rev-list --no-walk d660668ff45312f7c3b10529b29b478efe220e57 #
> timeout=10
> [flink-statefun-snapshot-deployment-2.0] $ /bin/bash
> /tmp/jenkins419419675999100733.sh
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 100 341 100 3410 0914 0 --:--:-- --:--:-- --:--:-- 914
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> ./tools/snapshots/deploy_snapshot_jars.sh: line 47: mvn: command not found
> {code}
> Happened twice in a row.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on issue #11802: [FLINK-17197][runtime] switch ContinuousFileReaderOperator state to FAILED on error
rkhachatryan commented on issue #11802: URL: https://github.com/apache/flink/pull/11802#issuecomment-616391502 Azure failure is unrelated: https://issues.apache.org/jira/browse/FLINK-17205 Last push is a rebase onto the current master to get the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17261) Statefun docs broken
[
https://issues.apache.org/jira/browse/FLINK-17261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-17261:
---
Assignee: Tzu-Li (Gordon) Tai
> Statefun docs broken
>
>
> Key: FLINK-17261
> URL: https://issues.apache.org/jira/browse/FLINK-17261
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: statefun-2.0.0, statefun-2.1.0
>Reporter: Chesnay Schepler
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
>
> https://ci.apache.org/builders/flink-statefun-docs-master/builds/56
> {code} Liquid Exception: Liquid syntax error (line 67): Unknown tag
> 'higlight' in deployment-and-operations/packaging.md
> Liquid syntax error (line 67): Unknown tag 'higlight'
> /home/buildslave/slave/flink-statefun-docs-master/build/docs/.rubydeps/ruby/2.6.0/gems/liquid-4.0.3/lib/liquid/document.rb:23:in
> `unknown_tag'{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] twalthr commented on issue #11568: [FLINK-16779][table] Add RAW type support in DDL and functions
twalthr commented on issue #11568: URL: https://github.com/apache/flink/pull/11568#issuecomment-616393913 Having serde as part of a data type improves the usability a lot because we are hiding the complexity around this topic from users. A user must not care about serialization but can simply pass around a data type. Esp. when using complex nested POJOs where just one field needs special treatment, a dedicated type is very handy. In many cases, POJOs are predefined by some company-wide schema. So users are not allowed to change the type of a field to `byte[]` for a VARBINARY. I agree that a RAW string in a DDL looks terrible but in most of the cases such a string will be generated by some tool. In any case, we need a valid string that can be stored in a catalog to reduce friction in the end-to-end story. In the future, we can further improve the usability by proving a `DistinctType` for those type. Such as `CREATE TYPE MyRawType OF 'org.package.MyClass'` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15331) Create a table from a changelog
[ https://issues.apache.org/jira/browse/FLINK-15331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087492#comment-17087492 ] Forward Xu commented on FLINK-15331: big +1, Thanks for firing this，[~twalthr] and [~jark] > Create a table from a changelog > --- > > Key: FLINK-15331 > URL: https://issues.apache.org/jira/browse/FLINK-15331 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Timo Walther >Assignee: Jark Wu >Priority: Major > Fix For: 1.11.0 > > > This an umbrella issue for the feature of interpreting a changelog as a > table. It can be used to collect initial ideas and use cases until a FLIP is > proposed. > FLIP-105: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] alpinegizmo opened a new pull request #11818: [hotfix][docs] svg figure has overlapping text
alpinegizmo opened a new pull request #11818: URL: https://github.com/apache/flink/pull/11818 In this part of this svg figure, there's some misplaced text where it says "BucketingSink(path)": https://user-images.githubusercontent.com/43608/79730701-8054ed00-82f1-11ea-9d40-a7efff305ca2.png";> Here is it moved over so the text no longer overlaps: https://user-images.githubusercontent.com/43608/79730783-9ebae880-82f1-11ea-839a-776770e6f49c.png";> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dawidwys commented on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616394636 Thanks for the explanation @dianfu. Really helpful. Now I understand the change. Personally I don't have a better idea from the top of my head. Let's see what @danny0405 comes up with. +1 from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on issue #11771: [FLINK-17135][python][tests] Fix the test testPandasFunctionMixedWithGeneralPythonFunction to make it more stable
dianfu commented on issue #11771: URL: https://github.com/apache/flink/pull/11771#issuecomment-616394631 May be we can revert the change of calcite-3149 in Flink? Does that makes sense for you? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11554: [FLINK-15101][connector/common] Add SourceCoordinator implementation.
flinkbot edited a comment on issue #11554: URL: https://github.com/apache/flink/pull/11554#issuecomment-605459909 ## CI report: * bd27b1b4dd97173d85bb0867f7c377ac4c638af5 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156295604) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6793) * 14e9fe3bfdeeae480047848801243f9fbed03cb4 UNKNOWN * 6a0a147d96dc517574efb6a97786b8b7e8e9c10c Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160993263) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7752) * a11dbc2d6b25ff16ef3cff4ecc751538d6867e68 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11696: [FLINK-16658][FLINK-16660] Introduce the ApplicationDispatcherBootstrap and wire it to StandaloneJobEntrypoint
flinkbot edited a comment on issue #11696: URL: https://github.com/apache/flink/pull/11696#issuecomment-611702183 ## CI report: * 10446262ad30a5f2a2a743065f1106d5242fead4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160726688) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7662) * b44d92e51473c208a84700f58cf453042b7a237b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160993353) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7753) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11692: [FLINK-16992][table-common] Add all ability interfaces for table sources and sinks
flinkbot edited a comment on issue #11692: URL: https://github.com/apache/flink/pull/11692#issuecomment-611530487 ## CI report: * 4c854e0afe5cb934d919f37eb0891719063938ac Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160752603) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7683) * aa7f40b60eb7b67fd3e2ec6cd4be5890e2218ba8 UNKNOWN * 769e5d807f5c6671c0be5f39bc0309024d1caac2 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160984394) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7749) * 4aaae96bd8e3d98faada537f9f582e699925fbd6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11784: [FLINK-17120][python] Add Cython support for operations
flinkbot edited a comment on issue #11784: URL: https://github.com/apache/flink/pull/11784#issuecomment-615011156 ## CI report: * 1ee639f83adcf607b722609782be88207de5fe49 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160666122) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7622) * 6f1db3ebd65dcbeaebb677460838b22745b9d89a Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160993466) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7756) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11768: [FLINK-16943][python] Support set the configuration option "pipeline.jars" in PyFlink.
flinkbot edited a comment on issue #11768: URL: https://github.com/apache/flink/pull/11768#issuecomment-614482552 ## CI report: * 01280e72a363ee72180dd385dc95e0013f60650f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160718616) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7657) * 375c382d79b0eb32388aa1b6b9c2f365a91344ee Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160993435) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7755) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11727: [FLINK-17106][table] Support create and drop view in Flink SQL
flinkbot edited a comment on issue #11727: URL: https://github.com/apache/flink/pull/11727#issuecomment-613273432 ## CI report: * fa5592cec0a6a5dcecc4f45c4bf72caf6e166eb4 UNKNOWN * 3ac7c714eacc6416b3dfbe163b4581d18e67de8d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160967398) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7743) * a3901b149f603721be62fbb3d8bf46143d08c3d1 UNKNOWN * 25bb852676993fd14ebccf710b35a5f49bd56fd6 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160993385) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7754) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11811: [FLINK-16658][FLINK-16660] Introduce the ApplicationDispatcherBootstrap and wire it to StandaloneJobEntrypoint
flinkbot edited a comment on issue #11811: URL: https://github.com/apache/flink/pull/11811#issuecomment-616107576 ## CI report: * 40bd2693a11d52097fd55b23173710f35965dcb7 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160897124) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7732) * 205c6d4836622e84342b0194c789bb45e29f344b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11802: [FLINK-17197][runtime] switch ContinuousFileReaderOperator state to FAILED on error
flinkbot edited a comment on issue #11802: URL: https://github.com/apache/flink/pull/11802#issuecomment-615431873 ## CI report: * 1f63e9f1b9f77c0b16ce16a743de535333ba7fff Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160783547) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7690) * b3a5ba58720be5eee04eb2b13cd265ed24144a90 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on issue #11817: [FLINK-17257][yarn][test] Fix AbstractYarnClusterTest not compiling with hadoop 2.10.
flinkbot edited a comment on issue #11817: URL: https://github.com/apache/flink/pull/11817#issuecomment-616385883 ## CI report: * 19e4aa4ab7974adb5b771af4a9a51127c71e0bb4 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160993523) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7757) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on issue #11818: [hotfix][docs] svg figure has overlapping text
flinkbot commented on issue #11818: URL: https://github.com/apache/flink/pull/11818#issuecomment-616396660 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 94b7482267a9a50bbb00766abaa77cb7f4a2f25b (Mon Apr 20 08:33:32 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KurtYoung commented on issue #10455: [FLINK-15089][connectors] Puslar catalog
KurtYoung commented on issue #10455: URL: https://github.com/apache/flink/pull/10455#issuecomment-616398922 Hi @yjshen , sorry for the late response on this. I think right now the community is busy with finishing 1.11 planned features thus don't have much capacity for reviewing this. I would raise this during 1.12 feature planning and hopefully we can get this and pulsar connectors in Flink 1.12. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #11797: [FLINK-17169][table-blink] Refactor BaseRow to use RowKind instead of byte header
wuchong commented on a change in pull request #11797:
URL: https://github.com/apache/flink/pull/11797#discussion_r411198620
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/utils/WindowEmitStrategy.scala
##
@@ -63,7 +63,7 @@ class WindowEmitStrategy(
def produceUpdates: Boolean = {
if (isEventTime) {
- allowLateness > 0 || earlyFireDelayEnabled || lateFireDelayEnabled
+ earlyFireDelayEnabled || lateFireDelayEnabled
Review comment:
Yes. I think this is a bug when we merging blink code. A window
aggregation can produce updates only when early fire or late fire is enabled.
`allowLateness` is just a hint to the statebackend for how long to keep the
state when window is closed. If late data never trigger emission, the window
still produce insert-only messages.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] GJL commented on a change in pull request #11770: [FLINK-17014][runtime] Implement PipelinedRegionSchedulingStrategy
GJL commented on a change in pull request #11770:
URL: https://github.com/apache/flink/pull/11770#discussion_r411198677
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/strategy/TestingSchedulingPipelinedRegion.java
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler.strategy;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A simple implementation of {@link SchedulingPipelinedRegion} for testing.
+ */
+public class TestingSchedulingPipelinedRegion implements
SchedulingPipelinedRegion {
+
+ private final Map
regionVertices = new HashMap<>();
+
+ private final Set consumedPartitions
= new HashSet<>();
+
+ public TestingSchedulingPipelinedRegion(final
Set vertices) {
+ for (TestingSchedulingExecutionVertex vertex : vertices) {
+ regionVertices.put(vertex.getId(), vertex);
+
+ for (TestingSchedulingResultPartition consumedPartition
: vertex.getConsumedResults()) {
+ if
(!vertices.contains(consumedPartition.getProducer())) {
+
consumedPartitions.add(consumedPartition);
+ }
+ }
+ }
+ }
+
+ @Override
+ public Iterable getVertices() {
+ return regionVertices.values();
Review comment:
I don't think that's true, see `Iterator#remove()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-16947) ArtifactResolutionException: Could not transfer artifact. Entry [...] has not been leased from this pool
[
https://issues.apache.org/jira/browse/FLINK-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087501#comment-17087501
]
Chesnay Schepler commented on FLINK-16947:
--
Maven 3.2.5 uses wagon-http 2.8 which uses apache httpClient 4.3.2 which has a
related issue HTTPCLIENT-1453.
It would be good to understand though why we aren't seeing this Travis.
> ArtifactResolutionException: Could not transfer artifact. Entry [...] has
> not been leased from this pool
> -
>
> Key: FLINK-16947
> URL: https://issues.apache.org/jira/browse/FLINK-16947
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Piotr Nowojski
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6982&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> Build of flink-metrics-availability-test failed with:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (end-to-end-tests)
> on project flink-metrics-availability-test: Unable to generate classpath:
> org.apache.maven.artifact.resolver.ArtifactResolutionException: Could not
> transfer artifact org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> from/to google-maven-central
> (https://maven-central-eu.storage-download.googleapis.com/maven2/): Entry
> [id:13][route:{s}->https://maven-central-eu.storage-download.googleapis.com:443][state:null]
> has not been leased from this pool
> [ERROR] org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR]
> [ERROR] from the specified remote repositories:
> [ERROR] google-maven-central
> (https://maven-central-eu.storage-download.googleapis.com/maven2/,
> releases=true, snapshots=false),
> [ERROR] apache.snapshots (https://repository.apache.org/snapshots,
> releases=false, snapshots=true)
> [ERROR] Path to dependency:
> [ERROR] 1) dummy:dummy:jar:1.0
> [ERROR] 2) org.apache.maven.surefire:surefire-junit47:jar:2.22.1
> [ERROR] 3) org.apache.maven.surefire:common-junit48:jar:2.22.1
> [ERROR] 4) org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :flink-metrics-availability-test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai commented on issue #100: [FLINK-17060][docs] Replace "{flink}" with the Apache Flink trademark
tzulitai commented on issue #100: URL: https://github.com/apache/flink-statefun/pull/100#issuecomment-616400628 LGTM! Thanks for the fix @authuir. Merging this ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
