[GitHub] [flink] dianfu commented on pull request #13070: [FLINK-18831][docs][python] Improve the documentation about the operations in Python Table API
dianfu commented on pull request #13070: URL: https://github.com/apache/flink/pull/13070#issuecomment-669718510 @WeiZhong94 Thanks for the review. Updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #13070: [FLINK-18831][docs][python] Improve the documentation about the operations in Python Table API
dianfu commented on a change in pull request #13070:
URL: https://github.com/apache/flink/pull/13070#discussion_r466159475
##
File path: docs/dev/table/tableApi.md
##
@@ -1490,8 +1558,12 @@ result = left.join(right).where("a = d && rowtime1 >=
rowtime2 - 1.second
Joins a table with the results of a table function. Each row of the
left (outer) table is joined with all rows produced by the corresponding call
of the table function. A row of the left (outer) table is dropped, if its table
function call returns an empty result.
{% highlight python %}
-# register Java User-Defined Table Function
-table_env.register_java_function("split", "com.my.udf.MySplitUDTF")
+# register User-Defined Table Function
+@udtf(input_types=[DataTypes.BIGINT()],
+ result_types=[DataTypes.BIGINT(), DataTypes.BIGINT(),
DataTypes.BIGINT()])
+def split(x):
+return [Row(1, 2, 3)]
Review comment:
array also works
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * bce6761325c406591dac1f9ba2d755f5d76a3f78 UNKNOWN * cb4298bf796daf7c9e3ac2842459ea9221f7ac9e UNKNOWN * d489b53b960def9c1beb30f0b605e0debc528120 UNKNOWN * 8449e779e9d551934fde6300b13c9fbcd3b870b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5220) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5195) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18836) Python UDTF doesn't work well when the return type isn't generator
[
https://issues.apache.org/jira/browse/FLINK-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18836:
Issue Type: Bug (was: Improvement)
> Python UDTF doesn't work well when the return type isn't generator
> --
>
> Key: FLINK-18836
> URL: https://issues.apache.org/jira/browse/FLINK-18836
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Priority: Major
> Fix For: 1.12.0, 1.11.2
>
>
> For the following Python UDTF which return type is not a generator:
> {code}
> # test specify the input_types
> @udtf(input_types=[DataTypes.BIGINT()],
>result_types=[DataTypes.BIGINT(), DataTypes.BIGINT(),
> DataTypes.BIGINT()])
> def split(x):
> return Row(10, 10, 10)
> {code}
> When used in a job, the operator containing the UDTF will not emit data to
> the downstream operator and there is also no exception thrown. The job just
> finished without any result.
> We should properly handle this case: either support this use case or throw a
> proper exception if we don't want to support this case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18836) Python UDTF doesn't work well when the return type isn't generator
Dian Fu created FLINK-18836:
---
Summary: Python UDTF doesn't work well when the return type isn't
generator
Key: FLINK-18836
URL: https://issues.apache.org/jira/browse/FLINK-18836
Project: Flink
Issue Type: Improvement
Components: API / Python
Affects Versions: 1.11.0
Reporter: Dian Fu
Fix For: 1.12.0, 1.11.2
For the following Python UDTF which return type is not a generator:
{code}
# test specify the input_types
@udtf(input_types=[DataTypes.BIGINT()],
result_types=[DataTypes.BIGINT(), DataTypes.BIGINT(),
DataTypes.BIGINT()])
def split(x):
return Row(10, 10, 10)
{code}
When used in a job, the operator containing the UDTF will not emit data to the
downstream operator and there is also no exception thrown. The job just
finished without any result.
We should properly handle this case: either support this use case or throw a
proper exception if we don't want to support this case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * bce6761325c406591dac1f9ba2d755f5d76a3f78 UNKNOWN * cb4298bf796daf7c9e3ac2842459ea9221f7ac9e UNKNOWN * d489b53b960def9c1beb30f0b605e0debc528120 UNKNOWN * 8449e779e9d551934fde6300b13c9fbcd3b870b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5220) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5195) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12880: [FLINK-18555][table sql/api] Make TableConfig options can be configur…
flinkbot edited a comment on pull request #12880: URL: https://github.com/apache/flink/pull/12880#issuecomment-657437173 ## CI report: * c5a0c7fa6e6cbe59d50a43745ce76b4ab8864f2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5225) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] WeiZhong94 commented on a change in pull request #13070: [FLINK-18831][docs][python] Improve the documentation about the operations in Python Table API
WeiZhong94 commented on a change in pull request #13070:
URL: https://github.com/apache/flink/pull/13070#discussion_r466150697
##
File path: docs/dev/table/tableApi.md
##
@@ -1490,8 +1558,12 @@ result = left.join(right).where("a = d && rowtime1 >=
rowtime2 - 1.second
Joins a table with the results of a table function. Each row of the
left (outer) table is joined with all rows produced by the corresponding call
of the table function. A row of the left (outer) table is dropped, if its table
function call returns an empty result.
{% highlight python %}
-# register Java User-Defined Table Function
-table_env.register_java_function("split", "com.my.udf.MySplitUDTF")
+# register User-Defined Table Function
+@udtf(input_types=[DataTypes.BIGINT()],
+ result_types=[DataTypes.BIGINT(), DataTypes.BIGINT(),
DataTypes.BIGINT()])
+def split(x):
+return [Row(1, 2, 3)]
Review comment:
yield Row(1, 2, 3) ?
##
File path: docs/dev/table/tableApi.md
##
@@ -464,6 +485,49 @@ orders = table_env.from_path("Orders")
{% endhighlight %}
+
+
+FromElements
+Batch Streaming
+
+
+ Similar to the VALUES clause in a SQL query. Produces an inline
table out of the provided rows.
+{% highlight python %}
+table = tEnv.from_elements([(1, 'ABC'), (2, 'ABCDE')])
Review comment:
t_env?
##
File path: docs/dev/table/tableApi.md
##
@@ -101,25 +101,49 @@ val result = orders
-The Python Table API is enabled by `from pyflink.table import *`.
-
The following example shows how a Python Table API program is constructed and
how expressions are specified as strings.
{% highlight python %}
from pyflink.table import *
-from pyflink.dataset import *
# environment configuration
-env = ExecutionEnvironment.get_execution_environment()
-t_env = TableEnvironment.create(env, TableConfig())
+t_env = BatchTableEnvironment.create(
+
environment_settings=EnvironmentSettings.new_instance().in_batch_mode().use_blink_planner().build())
# register Orders table and Result table sink in table environment
-# ...
+source_data_path = "/path/to/source/directory/"
+result_data_path = "/path/to/result/directory/"
+source_ddl = """
Review comment:
It would be better if we use the format strings:
f"""
create table Orders(
a VARCHAR,
b BIGINT,
c BIGINT,
rowtime TIMESTAMP(3),
WATERMARK FOR rowtime AS rowtime - INTERVAL '1' SECOND
) with (
'connector' = 'filesystem',
'format' = 'csv',
'path' = '{source_data_path}'
)
"""
##
File path: docs/dev/table/tableApi.md
##
@@ -1490,8 +1558,12 @@ result = left.join(right).where("a = d && rowtime1 >=
rowtime2 - 1.second
Joins a table with the results of a table function. Each row of the
left (outer) table is joined with all rows produced by the corresponding call
of the table function. A row of the left (outer) table is dropped, if its table
function call returns an empty result.
{% highlight python %}
-# register Java User-Defined Table Function
-table_env.register_java_function("split", "com.my.udf.MySplitUDTF")
+# register User-Defined Table Function
+@udtf(input_types=[DataTypes.BIGINT()],
Review comment:
input_types is unnecessary now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
flinkbot edited a comment on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-667904442 ## CI report: * 128e47f58b081d6273e3c6c57ad0da76c990595f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5226) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12880: [FLINK-18555][table sql/api] Make TableConfig options can be configur…
flinkbot edited a comment on pull request #12880: URL: https://github.com/apache/flink/pull/12880#issuecomment-657437173 ## CI report: * 5e007a083820aed78298cd519ae17e3762b5fa53 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5207) * c5a0c7fa6e6cbe59d50a43745ce76b4ab8864f2d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5225) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
flinkbot edited a comment on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-667904442 ## CI report: * e274e166aed3ffacc4ac58409aaf3384c937cb66 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5146) * 128e47f58b081d6273e3c6c57ad0da76c990595f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13075: [FLINK-18833][doc][python] Improve the Python documentation about sql
flinkbot edited a comment on pull request #13075: URL: https://github.com/apache/flink/pull/13075#issuecomment-669675460 ## CI report: * 661f2d534b6eb4b6718bb86cbe6d34361c20758d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5224) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12880: [FLINK-18555][table sql/api] Make TableConfig options can be configur…
flinkbot edited a comment on pull request #12880: URL: https://github.com/apache/flink/pull/12880#issuecomment-657437173 ## CI report: * 5e007a083820aed78298cd519ae17e3762b5fa53 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5207) * c5a0c7fa6e6cbe59d50a43745ce76b4ab8864f2d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13075: [FLINK-18833][doc][python] Improve the Python documentation about sql
flinkbot commented on pull request #13075: URL: https://github.com/apache/flink/pull/13075#issuecomment-669675460 ## CI report: * 661f2d534b6eb4b6718bb86cbe6d34361c20758d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * bce6761325c406591dac1f9ba2d755f5d76a3f78 UNKNOWN * cb4298bf796daf7c9e3ac2842459ea9221f7ac9e UNKNOWN * d489b53b960def9c1beb30f0b605e0debc528120 UNKNOWN * 8449e779e9d551934fde6300b13c9fbcd3b870b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5220) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5195) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #13050: [FLINK-18750][table] SqlValidatorException thrown when select from a …
danny0405 commented on pull request #13050: URL: https://github.com/apache/flink/pull/13050#issuecomment-669674850 Finally i have added a new component named `Expander` for the sql identifier expanding, so that we do not need to depend on the Calcite upgrade, also this would avoid the expanding logic to be buggy again and again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-18832) BoundedBlockingSubpartition does not work with StreamTask
[
https://issues.apache.org/jira/browse/FLINK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijiang reassigned FLINK-18832:
Assignee: Zhijiang
> BoundedBlockingSubpartition does not work with StreamTask
> -
>
> Key: FLINK-18832
> URL: https://issues.apache.org/jira/browse/FLINK-18832
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.10.1, 1.12.0, 1.11.1
>Reporter: Till Rohrmann
>Assignee: Zhijiang
>Priority: Major
>
> The {{BoundedBlockingSubpartition}} does not work with a {{StreamTask}}
> because the {{StreamTask}} instantiates an {{OutputFlusher}} which
> concurrently accesses the {{BoundedBlockingSubpartition}}. This concurrency
> can lead to a double closing of the underlying {{BufferConsumer}} which
> manifests in this stack trace:
> {code}
> [9:15 PM] Caused by:
> org.apache.flink.shaded.netty4.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, increment: 1
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.retain0(ReferenceCountUpdater.java:123)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.retain(ReferenceCountUpdater.java:110)
> at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.retain(AbstractReferenceCountedByteBuf.java:80)
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.retainBuffer(NetworkBuffer.java:174)
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.retainBuffer(NetworkBuffer.java:47)
> at
> org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.retainBuffer(ReadOnlySlicedNetworkBuffer.java:127)
> at
> org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.retainBuffer(ReadOnlySlicedNetworkBuffer.java:41)
> at
> org.apache.flink.runtime.io.network.buffer.BufferConsumer.build(BufferConsumer.java:108)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.writeAndCloseBufferConsumer(BoundedBlockingSubpartition.java:156)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.flushCurrentBuffer(BoundedBlockingSubpartition.java:144)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.flush(BoundedBlockingSubpartition.java:135)
> at
> org.apache.flink.runtime.io.network.partition.ResultPartition.flushAll(ResultPartition.java:245)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.flushAll(RecordWriter.java:183)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.flush(RecordWriterOutput.java:156)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.flushOutputs(OperatorChain.java:344)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:602)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:544)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18832) BoundedBlockingSubpartition does not work with StreamTask
[
https://issues.apache.org/jira/browse/FLINK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171894#comment-17171894
]
Zhijiang edited comment on FLINK-18832 at 8/6/20, 4:15 AM:
---
Thanks for reporting this bug [~trohrmann]!
By design I guess the BoundedBlockingSubpartition assumes no concurrent issue
for `#flushCurrentBuffer` method. But actually the task thread and flusher
thread can touch this method concurrently. I can think of some options for
resolving it:
* Disable flusher thread for batch jobs, because it has no benefits for latency
concern as the downstream will only request partition after upstream finishes
based on current schedule way. Even it would bring harm for upstream writer to
spill partial buffer after flush triggered.
* From long term goal, the flusher thread should be delegated by mailbox model,
so we can avoid concurrent issue even if the flusher timeout valid for batch
jobs.
* Breaks the previous assumption to allow concurrent access of
`BoundedBlockingSubpartition#flushCurrentBuffer`.
If we can realize the second option soon, then we can bypass this bug. I
remembered [~pnowojski]already submitted the PR for it before, but have not
merged yet. If this way can not be realized in short time, then i prefer the
first option to work around. WDYT?
was (Author: zjwang):
Thanks for reporting this bug [~trohrmann]!
By design I guess the BoundedBlockingSubpartition assumes no concurrent issue
for `#flushCurrentBuffer` method. But actually the task thread and flusher
thread can touch this method concurrently. I can thought of some options for
resolving it:
* Disable flusher thread for batch jobs, because it has no benefits for latency
concern as the downstream will only request partition after upstream finishes
based on current schedule way. Even it would bring harm for upstream writer to
spill partial buffer after flush triggered.
* From long term goal, the flusher thread should be delegated by mailbox model,
so we can avoid concurrent issue even if the flusher timeout valid for batch
jobs.
* Breaks the previous assumption to allow concurrent access of
`BoundedBlockingSubpartition#flushCurrentBuffer`.
If we can realize the second option soon, then we can bypass this bug. I
remembered [~pnowojski]already submitted the PR for it before, but have not
merged yet. If this way can not be realized in short time, then i prefer the
first option to work around. WDYT?
> BoundedBlockingSubpartition does not work with StreamTask
> -
>
> Key: FLINK-18832
> URL: https://issues.apache.org/jira/browse/FLINK-18832
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.10.1, 1.12.0, 1.11.1
>Reporter: Till Rohrmann
>Priority: Major
>
> The {{BoundedBlockingSubpartition}} does not work with a {{StreamTask}}
> because the {{StreamTask}} instantiates an {{OutputFlusher}} which
> concurrently accesses the {{BoundedBlockingSubpartition}}. This concurrency
> can lead to a double closing of the underlying {{BufferConsumer}} which
> manifests in this stack trace:
> {code}
> [9:15 PM] Caused by:
> org.apache.flink.shaded.netty4.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, increment: 1
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.retain0(ReferenceCountUpdater.java:123)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.retain(ReferenceCountUpdater.java:110)
> at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.retain(AbstractReferenceCountedByteBuf.java:80)
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.retainBuffer(NetworkBuffer.java:174)
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.retainBuffer(NetworkBuffer.java:47)
> at
> org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.retainBuffer(ReadOnlySlicedNetworkBuffer.java:127)
> at
> org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.retainBuffer(ReadOnlySlicedNetworkBuffer.java:41)
> at
> org.apache.flink.runtime.io.network.buffer.BufferConsumer.build(BufferConsumer.java:108)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.writeAndCloseBufferConsumer(BoundedBlockingSubpartition.java:156)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.flushCurrentBuffer(BoundedBlockingSubpartition.java:144)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.flush(BoundedBlockingSubpartition.java:135)
> at
>
[jira] [Commented] (FLINK-18832) BoundedBlockingSubpartition does not work with StreamTask
[
https://issues.apache.org/jira/browse/FLINK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171894#comment-17171894
]
Zhijiang commented on FLINK-18832:
--
Thanks for reporting this bug [~trohrmann]!
By design I guess the BoundedBlockingSubpartition assumes no concurrent issue
for `#flushCurrentBuffer` method. But actually the task thread and flusher
thread can touch this method concurrently. I can thought of some options for
resolving it:
* Disable flusher thread for batch jobs, because it has no benefits for latency
concern as the downstream will only request partition after upstream finishes
based on current schedule way. Even it would bring harm for upstream writer to
spill partial buffer after flush triggered.
* From long term goal, the flusher thread should be delegated by mailbox model,
so we can avoid concurrent issue even if the flusher timeout valid for batch
jobs.
* Breaks the previous assumption to allow concurrent access of
`BoundedBlockingSubpartition#flushCurrentBuffer`.
If we can realize the second option soon, then we can bypass this bug. I
remembered [~pnowojski]already submitted the PR for it before, but have not
merged yet. If this way can not be realized in short time, then i prefer the
first option to work around. WDYT?
> BoundedBlockingSubpartition does not work with StreamTask
> -
>
> Key: FLINK-18832
> URL: https://issues.apache.org/jira/browse/FLINK-18832
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Runtime / Task
>Affects Versions: 1.10.1, 1.12.0, 1.11.1
>Reporter: Till Rohrmann
>Priority: Major
>
> The {{BoundedBlockingSubpartition}} does not work with a {{StreamTask}}
> because the {{StreamTask}} instantiates an {{OutputFlusher}} which
> concurrently accesses the {{BoundedBlockingSubpartition}}. This concurrency
> can lead to a double closing of the underlying {{BufferConsumer}} which
> manifests in this stack trace:
> {code}
> [9:15 PM] Caused by:
> org.apache.flink.shaded.netty4.io.netty.util.IllegalReferenceCountException:
> refCnt: 0, increment: 1
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.retain0(ReferenceCountUpdater.java:123)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.retain(ReferenceCountUpdater.java:110)
> at
> org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.retain(AbstractReferenceCountedByteBuf.java:80)
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.retainBuffer(NetworkBuffer.java:174)
> at
> org.apache.flink.runtime.io.network.buffer.NetworkBuffer.retainBuffer(NetworkBuffer.java:47)
> at
> org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.retainBuffer(ReadOnlySlicedNetworkBuffer.java:127)
> at
> org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.retainBuffer(ReadOnlySlicedNetworkBuffer.java:41)
> at
> org.apache.flink.runtime.io.network.buffer.BufferConsumer.build(BufferConsumer.java:108)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.writeAndCloseBufferConsumer(BoundedBlockingSubpartition.java:156)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.flushCurrentBuffer(BoundedBlockingSubpartition.java:144)
> at
> org.apache.flink.runtime.io.network.partition.BoundedBlockingSubpartition.flush(BoundedBlockingSubpartition.java:135)
> at
> org.apache.flink.runtime.io.network.partition.ResultPartition.flushAll(ResultPartition.java:245)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.flushAll(RecordWriter.java:183)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.flush(RecordWriterOutput.java:156)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.flushOutputs(OperatorChain.java:344)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:602)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:544)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18830) JoinCoGroupFunction and FlatJoinCoGroupFunction work incorrectly for outer join when one side of coGroup is empty
[
https://issues.apache.org/jira/browse/FLINK-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liupengcheng updated FLINK-18830:
-
Description:
Currently, The `JoinCoGroupFunction` and `FlatJoinCoGroupFunction` in
JoinedStreams does't respect the join type, it's been implemented as doing join
within a two-level loop. However, this is incorrect for outer join when one
side of the coGroup is empty.
```
public void coGroup(Iterable first, Iterable second,
Collector out) throws Exception {
for (T1 val1: first) {
for (T2 val2: second) {
wrappedFunction.join(val1, val2, out);
}
}
}
```
The above code is the current implementation, suppose the first input is
non-empty, and the second input is an empty iterator, then the join
function(`wrappedFunction`) will never be called. This will cause no data to be
emitted for a left outer join.
So I propose to consider join type here, and handle this case, e.g., for left
outer join, we can emit record with right side set to null here if the right
side is empty or can not find any match in the right side.
was:
Currently, The `JoinCoGroupFunction` and `FlatJoinCoGroupFunction` in
JoinedStreams does't respect the join type, it's been implemented as doing join
within a two-level loop. However, this is incorrect for outer join when one
side of the coGroup is empty.
```
public void coGroup(Iterable first, Iterable second,
Collector out) throws Exception {
for (T1 val1: first) {
for (T2 val2: second) {
wrappedFunction.join(val1, val2, out);
}
}
}
```
The above code is the current implementation, suppose the first input is
non-empty, and the second input is an empty iterator, then the join
function(`wrappedFunction`) will never be called. This will cause no data to be
emitted for a left outer join.
So I propose to consider join type here, and handle this case, e.g., for left
outer join, we can call join function with right side set to null here if the
right side is empty, so that the join function can handle real joins.
> JoinCoGroupFunction and FlatJoinCoGroupFunction work incorrectly for outer
> join when one side of coGroup is empty
> -
>
> Key: FLINK-18830
> URL: https://issues.apache.org/jira/browse/FLINK-18830
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.1
>Reporter: liupengcheng
>Priority: Major
>
> Currently, The `JoinCoGroupFunction` and `FlatJoinCoGroupFunction` in
> JoinedStreams does't respect the join type, it's been implemented as doing
> join within a two-level loop. However, this is incorrect for outer join when
> one side of the coGroup is empty.
> ```
> public void coGroup(Iterable first, Iterable second,
> Collector out) throws Exception {
> for (T1 val1: first) {
> for (T2 val2: second) {
> wrappedFunction.join(val1, val2, out);
> }
> }
> }
> ```
> The above code is the current implementation, suppose the first input is
> non-empty, and the second input is an empty iterator, then the join
> function(`wrappedFunction`) will never be called. This will cause no data to
> be emitted for a left outer join.
> So I propose to consider join type here, and handle this case, e.g., for left
> outer join, we can emit record with right side set to null here if the right
> side is empty or can not find any match in the right side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13017: [FLINK-18258][hive] Implement SHOW PARTITIONS for Hive dialect
flinkbot edited a comment on pull request #13017: URL: https://github.com/apache/flink/pull/13017#issuecomment-665564915 ## CI report: * 16cd7dffbff67fd8aa060e9dcfde4af700db0839 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4999) * 2f2a2d5d7ef165e0c7ce068535bc79e90aae9cca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5222) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * bce6761325c406591dac1f9ba2d755f5d76a3f78 UNKNOWN * cb4298bf796daf7c9e3ac2842459ea9221f7ac9e UNKNOWN * d489b53b960def9c1beb30f0b605e0debc528120 UNKNOWN * 8449e779e9d551934fde6300b13c9fbcd3b870b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5220) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5195) * 1c5870c57b64436014900d67be2005395e007a52 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13075: [FLINK-18833][doc][python] Improve the Python documentation about sql
flinkbot commented on pull request #13075: URL: https://github.com/apache/flink/pull/13075#issuecomment-669670559 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 661f2d534b6eb4b6718bb86cbe6d34361c20758d (Thu Aug 06 04:04:56 UTC 2020) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18833).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #13017: [FLINK-18258][hive] Implement SHOW PARTITIONS for Hive dialect
lirui-apache commented on a change in pull request #13017:
URL: https://github.com/apache/flink/pull/13017#discussion_r466122748
##
File path:
flink-table/flink-sql-parser/src/main/java/org/apache/flink/sql/parser/dql/SqlShowPartitions.java
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.sql.parser.dql;
+
+import org.apache.flink.sql.parser.SqlPartitionUtils;
+
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlIdentifier;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlNodeList;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * SHOW PARTITIONS sql call.
+ */
+public class SqlShowPartitions extends SqlCall {
+
+ public static final SqlSpecialOperator OPERATOR = new
SqlSpecialOperator("SHOW PARTITIONS", SqlKind.OTHER);
+
+ protected final SqlIdentifier tableIdentifier;
+ protected final SqlNodeList partitionSpec;
+
+ public SqlShowPartitions(SqlParserPos pos, SqlIdentifier tableName,
@Nullable SqlNodeList partitionSpec) {
+ super(pos);
+ this.tableIdentifier = requireNonNull(tableName, "tableName
should not be null");
+ this.partitionSpec = partitionSpec;
+ }
+
+ @Override
+ public SqlOperator getOperator() {
+ return OPERATOR;
+ }
+
+ @Override
+ public List getOperandList() {
+ List operands = new ArrayList<>();
+ operands.add(tableIdentifier);
+ operands.add(partitionSpec);
+ return operands;
+ }
+
+ @Override
+ public void unparse(SqlWriter writer, int leftPrec, int rightPrec) {
+ writer.keyword("SHOW PARTITIONS");
+ tableIdentifier.unparse(writer, leftPrec, rightPrec);
+ SqlNodeList partitionSpec = getPartitionSpec();
+ if (partitionSpec != null && partitionSpec.size() > 0) {
Review comment:
If `partitionSpec` is not null, I think it must not be empty. We can add
a check to verify that.
##
File path:
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveDialectITCase.java
##
@@ -450,6 +450,18 @@ public void testAddDropPartitions() throws Exception {
ObjectPath tablePath = new ObjectPath("default", "tbl");
assertEquals(2, hiveCatalog.listPartitions(tablePath).size());
+ List partitions =
Lists.newArrayList(tableEnv.executeSql("show partitions tbl").collect());
Review comment:
I think it'll be clearer to move these to a separate test case.
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -1021,6 +1022,28 @@ private TableResult executeOperation(Operation
operation) {
return buildShowResult("function name",
listFunctions());
} else if (operation instanceof ShowViewsOperation) {
return buildShowResult("view name", listViews());
+ } else if (operation instanceof ShowPartitionsOperation) {
+ String exMsg =
getDQLOpExecuteErrorMsg(operation.asSummaryString());
Review comment:
Why not just call `getDDLOpExecuteErrorMsg`?
##
File path:
flink-table/flink-sql-client/src/test/java/org/apache/flink/table/client/cli/SqlCommandParserTest.java
##
@@ -389,6 +409,15 @@ public static TestItem validSql(
return testItem;
}
+ public static TestItem validSql(
Review comment:
We already have a `TestItem::validSql` method that takes SQL dialect as
a parameter. Can you reuse that?
##
File path:
[jira] [Updated] (FLINK-18833) Improve the Python documentation about sql
[ https://issues.apache.org/jira/browse/FLINK-18833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18833: --- Labels: pull-request-available (was: ) > Improve the Python documentation about sql > -- > > Key: FLINK-18833 > URL: https://issues.apache.org/jira/browse/FLINK-18833 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > > Now, there are a few documentations are written for java/scala users, which > is not convenient for python users to read. > We need to use a more pythonic description in docs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #13075: [FLINK-18833][doc][python] Improve the Python documentation about sql
HuangXingBo opened a new pull request #13075: URL: https://github.com/apache/flink/pull/13075 ## What is the purpose of the change *This pull request will improve the Python documentation about sql* ## Brief change log - *Make description more pythonic in sql related docs* ## Verifying this change execute the script build_docs.sh ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-18834.
---
Resolution: Fixed
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu reopened FLINK-18834:
-
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18834:
Fix Version/s: 1.12.0
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-18834.
---
Resolution: Fixed
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171887#comment-17171887
]
Dian Fu commented on FLINK-18834:
-
master: c59d4b24bf9d20763e16f56b7ef4a4f42bcb0518
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dianfu merged pull request #13074: [FLINK-18834][doc] Fix the bug of wrong link of rocks db html in config doc
dianfu merged pull request #13074: URL: https://github.com/apache/flink/pull/13074 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu reassigned FLINK-18834:
---
Assignee: Huang Xingbo
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13074: [FLINK-18834][doc] Fix the bug of wrong link of rocks db html in config doc
flinkbot commented on pull request #13074: URL: https://github.com/apache/flink/pull/13074#issuecomment-669665515 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c7007ed90d82399d6c59af638a4facc94bcc484d (Thu Aug 06 03:44:03 UTC 2020) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18834).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18834:
---
Labels: pull-request-available (was: )
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed
flinkbot edited a comment on pull request #13073: URL: https://github.com/apache/flink/pull/13073#issuecomment-669650373 ## CI report: * 145c86b135ee64462c449dacc81643ee8908e68b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5219) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
flinkbot edited a comment on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669640616 ## CI report: * 69e9338aba347a925c5096dd4aafe439a90921ad Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5217) * 766339d0c1aac1c601b5796298f48b224584437c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5221) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13017: [FLINK-18258][hive] Implement SHOW PARTITIONS for Hive dialect
flinkbot edited a comment on pull request #13017: URL: https://github.com/apache/flink/pull/13017#issuecomment-665564915 ## CI report: * 16cd7dffbff67fd8aa060e9dcfde4af700db0839 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4999) * 2f2a2d5d7ef165e0c7ce068535bc79e90aae9cca UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo opened a new pull request #13074: [FLINK-18834][doc] Fix the bug of wrong link of rocks db html in config doc
HuangXingBo opened a new pull request #13074: URL: https://github.com/apache/flink/pull/13074 ## What is the purpose of the change *This pull request will fix the bug of wrong link of rocks db html in config doc* ## Brief change log - *change the link in config.md and config.zh.md* ## Verifying this change execute the script ./build_docs.sh ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18835) sql using group by, duplicated group fileld appears
[
https://issues.apache.org/jira/browse/FLINK-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YHF updated FLINK-18835:
Attachment: SumAnalysis.java
> sql using group by, duplicated group fileld appears
> ---
>
> Key: FLINK-18835
> URL: https://issues.apache.org/jira/browse/FLINK-18835
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: YHF
>Priority: Critical
> Attachments: SumAnalysis.java
>
>
> datasource is kafka,then create a temporary view, group by (fieldA,fieldB)
> using sql,
> then transform the result table to datastream using toRetractStream, then
> print the result,
> I find duplicated (fieldA,fieldB)
> see attachment for code
> group by(scanType,scanSite,cmtInf),but result is below
> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
> db=0E-18, dbjp=1.00, pjWei=27.07,
> dbWei=0E-18, mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29
> 11:33:57.679})
> 3> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
> db=0E-18, dbjp=1.00, pjWei=27.07,
> dbWei=0E-18, mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29
> 11:33:57.679})
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18835) sql using group by, duplicated group fileld appears
[
https://issues.apache.org/jira/browse/FLINK-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YHF updated FLINK-18835:
Attachment: (was: SumAnalysis.java)
> sql using group by, duplicated group fileld appears
> ---
>
> Key: FLINK-18835
> URL: https://issues.apache.org/jira/browse/FLINK-18835
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: YHF
>Priority: Critical
>
> datasource is kafka,then create a temporary view, group by (fieldA,fieldB)
> using sql,
> then transform the result table to datastream using toRetractStream, then
> print the result,
> I find duplicated (fieldA,fieldB)
> see attachment for code
> group by(scanType,scanSite,cmtInf),but result is below
> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
> db=0E-18, dbjp=1.00, pjWei=27.07,
> dbWei=0E-18, mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29
> 11:33:57.679})
> 3> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
> db=0E-18, dbjp=1.00, pjWei=27.07,
> dbWei=0E-18, mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29
> 11:33:57.679})
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18835) sql using group by, duplicated group fileld appears
[
https://issues.apache.org/jira/browse/FLINK-18835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YHF updated FLINK-18835:
Description:
datasource is kafka,then create a temporary view, group by (fieldA,fieldB)
using sql,
then transform the result table to datastream using toRetractStream, then print
the result,
I find duplicated (fieldA,fieldB)
see attachment for code
group by(scanType,scanSite,cmtInf),but result is below
(true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
db=0E-18, dbjp=1.00, pjWei=27.07, dbWei=0E-18,
mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29 11:33:57.679})
3> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
db=0E-18, dbjp=1.00, pjWei=27.07, dbWei=0E-18,
mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29 11:33:57.679})
was:
datasource is kafka,then create a temporary view, group by (fieldA,fieldB)
using sql,
then transform the result table to datastream using toRetractStream, then print
the result,
I find duplicated (fieldA,fieldB)
see attachment for code
> sql using group by, duplicated group fileld appears
> ---
>
> Key: FLINK-18835
> URL: https://issues.apache.org/jira/browse/FLINK-18835
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: YHF
>Priority: Critical
>
> datasource is kafka,then create a temporary view, group by (fieldA,fieldB)
> using sql,
> then transform the result table to datastream using toRetractStream, then
> print the result,
> I find duplicated (fieldA,fieldB)
> see attachment for code
> group by(scanType,scanSite,cmtInf),but result is below
> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
> db=0E-18, dbjp=1.00, pjWei=27.07,
> dbWei=0E-18, mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29
> 11:33:57.679})
> 3> (true,Otm\{, scanType=97, scanSite=14, cmtInf=24,jp=1.00,
> db=0E-18, dbjp=1.00, pjWei=27.07,
> dbWei=0E-18, mintime=2020-07-29 11:33:57.679, maxtime=2020-07-29
> 11:33:57.679})
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi closed pull request #13023: [hotfix][hive] Add Hive orc streaming sink cases
JingsongLi closed pull request #13023: URL: https://github.com/apache/flink/pull/13023 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18835) sql using group by, duplicated group fileld appears
YHF created FLINK-18835: --- Summary: sql using group by, duplicated group fileld appears Key: FLINK-18835 URL: https://issues.apache.org/jira/browse/FLINK-18835 Project: Flink Issue Type: Bug Components: Table SQL / Planner Affects Versions: 1.11.1 Reporter: YHF Attachments: SumAnalysis.java datasource is kafka,then create a temporary view, group by (fieldA,fieldB) using sql, then transform the result table to datastream using toRetractStream, then print the result, I find duplicated (fieldA,fieldB) see attachment for code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
flinkbot edited a comment on pull request #13024: URL: https://github.com/apache/flink/pull/13024#issuecomment-666148245 ## CI report: * eb611d5b39f997fa3e986b5d163cb65a44b4b0ba UNKNOWN * b2ae9dd70445adfb6e7b44deae46225841537b7d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5218) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
flinkbot edited a comment on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669640616 ## CI report: * 69e9338aba347a925c5096dd4aafe439a90921ad Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5217) * 766339d0c1aac1c601b5796298f48b224584437c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * bce6761325c406591dac1f9ba2d755f5d76a3f78 UNKNOWN * cb4298bf796daf7c9e3ac2842459ea9221f7ac9e UNKNOWN * d489b53b960def9c1beb30f0b605e0debc528120 UNKNOWN * 8449e779e9d551934fde6300b13c9fbcd3b870b3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5220) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5195) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-18834:
-
Fix Version/s: (was: 1.12.0)
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17075) Add task status reconciliation between TM and JM
[
https://issues.apache.org/jira/browse/FLINK-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171881#comment-17171881
]
Zhu Zhu commented on FLINK-17075:
-
Thanks for the explanation [~trohrmann]
I will move it to 1.10.3.
> Add task status reconciliation between TM and JM
>
>
> Key: FLINK-17075
> URL: https://issues.apache.org/jira/browse/FLINK-17075
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
>
> In order to harden the TM and JM communication I suggest to let the
> {{TaskExecutor}} send the task statuses back to the {{JobMaster}} as part of
> the heartbeat payload (similar to FLINK-11059). This would allow to reconcile
> the states of both components in case that a status update message was lost
> as described by a user on the ML.
> https://lists.apache.org/thread.html/ra9ed70866381f0ef0f4779633346722ccab3dc0d6dbacce04080b74e%40%3Cuser.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
Huang Xingbo created FLINK-18834:
Summary: Could not locate the included file
'generated/rocks_db_native_metric_configuration.html'
Key: FLINK-18834
URL: https://issues.apache.org/jira/browse/FLINK-18834
Project: Flink
Issue Type: Bug
Components: Documentation
Reporter: Huang Xingbo
Fix For: 1.12.0
When I execute the script ./build_docs.sh, it throws the following exception:
{code:java}
Liquid Exception: Could not locate the included file
'generated/rocks_db_native_metric_configuration.html' in any of
["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one of
those directories and, if it is a symlink, does not point outside your site
source. in ops/config.md
{code}
It seems `rocks_db_native_metric_configuration` has been renamed to
`rocksdb_native_metric_configuration.html`, but it has not been synchronized in
ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18834) Could not locate the included file 'generated/rocks_db_native_metric_configuration.html'
[
https://issues.apache.org/jira/browse/FLINK-18834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-18834:
-
Affects Version/s: 1.12.0
> Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html'
> -
>
> Key: FLINK-18834
> URL: https://issues.apache.org/jira/browse/FLINK-18834
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Huang Xingbo
>Priority: Major
> Fix For: 1.12.0
>
>
> When I execute the script ./build_docs.sh, it throws the following exception:
> {code:java}
> Liquid Exception: Could not locate the included file
> 'generated/rocks_db_native_metric_configuration.html' in any of
> ["/Users/duanchen/sourcecode/flink/docs/_includes"]. Ensure it exists in one
> of those directories and, if it is a symlink, does not point outside your
> site source. in ops/config.md
> {code}
> It seems `rocks_db_native_metric_configuration` has been renamed to
> `rocksdb_native_metric_configuration.html`, but it has not been synchronized
> in ops/config.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed
flinkbot edited a comment on pull request #13073: URL: https://github.com/apache/flink/pull/13073#issuecomment-669650373 ## CI report: * 145c86b135ee64462c449dacc81643ee8908e68b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5219) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
flinkbot edited a comment on pull request #13024: URL: https://github.com/apache/flink/pull/13024#issuecomment-666148245 ## CI report: * 8731d0680f929b9754a7fc5014a890db4c5bbc74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5049) * eb611d5b39f997fa3e986b5d163cb65a44b4b0ba UNKNOWN * b2ae9dd70445adfb6e7b44deae46225841537b7d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5218) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
flinkbot edited a comment on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-659940433 ## CI report: * da7fb96fc00acda2a4c103f5f177efb9bd9be8be UNKNOWN * bce6761325c406591dac1f9ba2d755f5d76a3f78 UNKNOWN * cb4298bf796daf7c9e3ac2842459ea9221f7ac9e UNKNOWN * d489b53b960def9c1beb30f0b605e0debc528120 UNKNOWN * 8449e779e9d551934fde6300b13c9fbcd3b870b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5195) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5220) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
PatrickRen commented on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669655364 Looks like CI failed. Fixed in 766339d This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry commented on pull request #13067: [FLINK-18821][network] Report PartitionRequest failures to subsequent…
wsry commented on pull request #13067: URL: https://github.com/apache/flink/pull/13067#issuecomment-669655074 The fix itself LGTM, but I am afraid there is another issue we may also need to fix. It seems to me the exceptionally completed future in the ```clients``` map is not cleaned up correctly. Am I right? If so, I think it is better to also fix it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18741) ProcessWindowFunction's process function exception
[
https://issues.apache.org/jira/browse/FLINK-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171876#comment-17171876
]
mzz edited comment on FLINK-18741 at 8/6/20, 2:53 AM:
--
[~aljoscha]
thx for your reply. it doesn't throwing any exception .
When I use ProcessWindowFunction , override process function， I traverse the
iterator in this function, the values outside are reset to their initial values。
the iterator is elements，define a variable in the process function and assign
the initial value. Then, the iterator is traversed to give the variable + 1.
However, when the window ends, the value of the variable will be reset to the
initial value.
*code:*
{code:java}
elements.foreach(e => {
if ("adreq".equals(e._3)) {
requestNum += 1
println(key._1, requestNum)
// The values printed here like :
//(key,1)
//(key,2)
//(key,3)
}
})
//But print outside the for loop always like :
//(key,0)
println(requestNum, key._1)
{code}
was (Author: mzz_q):
[~aljoscha]
thx for your reply.
When I use ProcessWindowFunction , override process function， I traverse the
iterator in this function, the values outside are reset to their initial values。
the iterator is elements，define a variable in the process function and assign
the initial value. Then, the iterator is traversed to give the variable + 1.
However, when the window ends, the value of the variable will be reset to the
initial value.
*code:*
{code:java}
elements.foreach(e => {
if ("adreq".equals(e._3)) {
requestNum += 1
println(key._1, requestNum)
// The values printed here like :
//(key,1)
//(key,2)
//(key,3)
}
})
//But print outside the for loop always like :
//(key,0)
println(requestNum, key._1)
{code}
> ProcessWindowFunction's process function exception
> ---
>
> Key: FLINK-18741
> URL: https://issues.apache.org/jira/browse/FLINK-18741
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0
>Reporter: mzz
>Priority: Critical
>
> I use ProcessWindowFunction to achieve PV calculation, but when rewriting
> process, the user-defined state value cannot be returned。
> code：
> {code:java}
> tem.keyBy(x =>
> (x._1, x._2, x._4, x._5, x._6, x._7, x._8))
> .timeWindow(Time.seconds(15 * 60)) //15 min window
> .process(new ProcessWindowFunction[(String, String, String, String,
> String, String, String, String, String), CkResult, (String, String, String,
> String, String, String, String), TimeWindow] {
> var clickCount: ValueState[Long] = _
> * var requestCount: ValueState[Long] = _
> * var returnCount: ValueState[Long] = _
> var videoCount: ValueState[Long] = _
> var noVideoCount: ValueState[Long] = _
> override def open(parameters: Configuration): Unit = {
> clickCount = getRuntimeContext.getState(new
> ValueStateDescriptor("clickCount", classOf[Long]))
>* requestCount = getRuntimeContext.getState(new
> ValueStateDescriptor("requestCount", classOf[Long]))*
> returnCount = getRuntimeContext.getState(new
> ValueStateDescriptor("returnCount", classOf[Long]))
> videoCount = getRuntimeContext.getState(new
> ValueStateDescriptor("videoCount", classOf[Long]))
> noVideoCount = getRuntimeContext.getState(new
> ValueStateDescriptor("noVideoCount", classOf[Long]))
> }
> override def process(key: (String, String, String, String, String,
> String, String), context: Context, elements: Iterable[(String, String,
> String, String, String, String, String, String, String)], out:
> Collector[CkResult]) = {
> try {
> var clickNum: Long = clickCount.value
> val dateNow =
> LocalDateTime.now().format(DateTimeFormatter.ofPattern("MMdd")).toLong
> var requestNum: Long = requestCount.value
> var returnNum: Long = returnCount.value
> var videoNum: Long = videoCount.value
> var noVideoNum: Long = noVideoCount.value
> if (requestNum == null) {
> requestNum = 0
> }
>
> val ecpm = key._7.toDouble.formatted("%.2f").toFloat
> val created_at = getSecondTimestampTwo(new Date)
>
> * elements.foreach(e => {
> if ("adreq".equals(e._3)) {
> requestNum += 1
> println(key._1, requestNum)
> }
> })
> requestCount.update(requestNum)
> println(requestNum, key._1)*
>
> out.collect(CkResult(dateNow, (created_at - getZero_time) / (60 *
> 15), key._2, key._3, key._4, key._5, key._3 + "_" + key._4 + "_" + key._5,
> key._6,
[jira] [Comment Edited] (FLINK-18741) ProcessWindowFunction's process function exception
[
https://issues.apache.org/jira/browse/FLINK-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171876#comment-17171876
]
mzz edited comment on FLINK-18741 at 8/6/20, 2:52 AM:
--
[~aljoscha]
thx for your reply.
When I use ProcessWindowFunction , override process function， I traverse the
iterator in this function, the values outside are reset to their initial values。
the iterator is elements，define a variable in the process function and assign
the initial value. Then, the iterator is traversed to give the variable + 1.
However, when the window ends, the value of the variable will be reset to the
initial value.
*code:*
{code:java}
elements.foreach(e => {
if ("adreq".equals(e._3)) {
requestNum += 1
println(key._1, requestNum)
// The values printed here like :
//(key,1)
//(key,2)
//(key,3)
}
})
//But print outside the for loop always like :
//(key,0)
println(requestNum, key._1)
{code}
was (Author: mzz_q):
When I use ProcessWindowFunction , override process function， I traverse the
iterator in this function, the values outside are reset to their initial values。
the iterator is elements，define a variable in the process function and assign
the initial value. Then, the iterator is traversed to give the variable + 1.
However, when the window ends, the value of the variable will be reset to the
initial value.
*code:*
{code:java}
elements.foreach(e => {
if ("adreq".equals(e._3)) {
requestNum += 1
println(key._1, requestNum)
// The values printed here like :
//(key,1)
//(key,2)
//(key,3)
}
})
//But print outside the for loop always like :
//(key,0)
println(requestNum, key._1)
{code}
> ProcessWindowFunction's process function exception
> ---
>
> Key: FLINK-18741
> URL: https://issues.apache.org/jira/browse/FLINK-18741
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0
>Reporter: mzz
>Priority: Critical
>
> I use ProcessWindowFunction to achieve PV calculation, but when rewriting
> process, the user-defined state value cannot be returned。
> code：
> {code:java}
> tem.keyBy(x =>
> (x._1, x._2, x._4, x._5, x._6, x._7, x._8))
> .timeWindow(Time.seconds(15 * 60)) //15 min window
> .process(new ProcessWindowFunction[(String, String, String, String,
> String, String, String, String, String), CkResult, (String, String, String,
> String, String, String, String), TimeWindow] {
> var clickCount: ValueState[Long] = _
> * var requestCount: ValueState[Long] = _
> * var returnCount: ValueState[Long] = _
> var videoCount: ValueState[Long] = _
> var noVideoCount: ValueState[Long] = _
> override def open(parameters: Configuration): Unit = {
> clickCount = getRuntimeContext.getState(new
> ValueStateDescriptor("clickCount", classOf[Long]))
>* requestCount = getRuntimeContext.getState(new
> ValueStateDescriptor("requestCount", classOf[Long]))*
> returnCount = getRuntimeContext.getState(new
> ValueStateDescriptor("returnCount", classOf[Long]))
> videoCount = getRuntimeContext.getState(new
> ValueStateDescriptor("videoCount", classOf[Long]))
> noVideoCount = getRuntimeContext.getState(new
> ValueStateDescriptor("noVideoCount", classOf[Long]))
> }
> override def process(key: (String, String, String, String, String,
> String, String), context: Context, elements: Iterable[(String, String,
> String, String, String, String, String, String, String)], out:
> Collector[CkResult]) = {
> try {
> var clickNum: Long = clickCount.value
> val dateNow =
> LocalDateTime.now().format(DateTimeFormatter.ofPattern("MMdd")).toLong
> var requestNum: Long = requestCount.value
> var returnNum: Long = returnCount.value
> var videoNum: Long = videoCount.value
> var noVideoNum: Long = noVideoCount.value
> if (requestNum == null) {
> requestNum = 0
> }
>
> val ecpm = key._7.toDouble.formatted("%.2f").toFloat
> val created_at = getSecondTimestampTwo(new Date)
>
> * elements.foreach(e => {
> if ("adreq".equals(e._3)) {
> requestNum += 1
> println(key._1, requestNum)
> }
> })
> requestCount.update(requestNum)
> println(requestNum, key._1)*
>
> out.collect(CkResult(dateNow, (created_at - getZero_time) / (60 *
> 15), key._2, key._3, key._4, key._5, key._3 + "_" + key._4 + "_" + key._5,
> key._6, key._1, requestCount.value, returnCount.value, fill_rate,
>
[jira] [Commented] (FLINK-18741) ProcessWindowFunction's process function exception
[
https://issues.apache.org/jira/browse/FLINK-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171876#comment-17171876
]
mzz commented on FLINK-18741:
-
When I use ProcessWindowFunction , override process function， I traverse the
iterator in this function, the values outside are reset to their initial values。
the iterator is elements，define a variable in the process function and assign
the initial value. Then, the iterator is traversed to give the variable + 1.
However, when the window ends, the value of the variable will be reset to the
initial value.
*code:*
{code:java}
elements.foreach(e => {
if ("adreq".equals(e._3)) {
requestNum += 1
println(key._1, requestNum)
// The values printed here like :
//(key,1)
//(key,2)
//(key,3)
}
})
//But print outside the for loop always like :
//(key,0)
println(requestNum, key._1)
{code}
> ProcessWindowFunction's process function exception
> ---
>
> Key: FLINK-18741
> URL: https://issues.apache.org/jira/browse/FLINK-18741
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.10.0
>Reporter: mzz
>Priority: Critical
>
> I use ProcessWindowFunction to achieve PV calculation, but when rewriting
> process, the user-defined state value cannot be returned。
> code：
> {code:java}
> tem.keyBy(x =>
> (x._1, x._2, x._4, x._5, x._6, x._7, x._8))
> .timeWindow(Time.seconds(15 * 60)) //15 min window
> .process(new ProcessWindowFunction[(String, String, String, String,
> String, String, String, String, String), CkResult, (String, String, String,
> String, String, String, String), TimeWindow] {
> var clickCount: ValueState[Long] = _
> * var requestCount: ValueState[Long] = _
> * var returnCount: ValueState[Long] = _
> var videoCount: ValueState[Long] = _
> var noVideoCount: ValueState[Long] = _
> override def open(parameters: Configuration): Unit = {
> clickCount = getRuntimeContext.getState(new
> ValueStateDescriptor("clickCount", classOf[Long]))
>* requestCount = getRuntimeContext.getState(new
> ValueStateDescriptor("requestCount", classOf[Long]))*
> returnCount = getRuntimeContext.getState(new
> ValueStateDescriptor("returnCount", classOf[Long]))
> videoCount = getRuntimeContext.getState(new
> ValueStateDescriptor("videoCount", classOf[Long]))
> noVideoCount = getRuntimeContext.getState(new
> ValueStateDescriptor("noVideoCount", classOf[Long]))
> }
> override def process(key: (String, String, String, String, String,
> String, String), context: Context, elements: Iterable[(String, String,
> String, String, String, String, String, String, String)], out:
> Collector[CkResult]) = {
> try {
> var clickNum: Long = clickCount.value
> val dateNow =
> LocalDateTime.now().format(DateTimeFormatter.ofPattern("MMdd")).toLong
> var requestNum: Long = requestCount.value
> var returnNum: Long = returnCount.value
> var videoNum: Long = videoCount.value
> var noVideoNum: Long = noVideoCount.value
> if (requestNum == null) {
> requestNum = 0
> }
>
> val ecpm = key._7.toDouble.formatted("%.2f").toFloat
> val created_at = getSecondTimestampTwo(new Date)
>
> * elements.foreach(e => {
> if ("adreq".equals(e._3)) {
> requestNum += 1
> println(key._1, requestNum)
> }
> })
> requestCount.update(requestNum)
> println(requestNum, key._1)*
>
> out.collect(CkResult(dateNow, (created_at - getZero_time) / (60 *
> 15), key._2, key._3, key._4, key._5, key._3 + "_" + key._4 + "_" + key._5,
> key._6, key._1, requestCount.value, returnCount.value, fill_rate,
> noVideoCount.value + videoCount.value,
> expose_rate, clickCount.value, click_rate, ecpm,
> (noVideoCount.value * ecpm + videoCount.value * ecpm /
> 1000.toFloat).formatted("%.2f").toFloat, created_at))
> }
> catch {
> case e: Exception => println(key, e)
> }
> }
> })
> {code}
> {code:java}
> elements.foreach(e => {
> if ("adreq".equals(e._3)) {
> requestNum += 1
> println(key._1, requestNum)
> // The values printed here like :
> //(key,1)
> //(key,2)
> //(key,3)
> }
> })
> //But print outside the for loop always like :
> //(key,0)
> println(requestNum, key._1)
> {code}
> who can help me ,plz thx。
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path
[ https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171875#comment-17171875 ] Kai Chen commented on FLINK-18785: -- hi [~trohrmann] . Updated jobmanager.log `jobmanager.log.attemp2-13` and yarn am attempts `flink_savepoint_path_do_not_exits.jpg` > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path > -- > > Key: FLINK-18785 > URL: https://issues.apache.org/jira/browse/FLINK-18785 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN, Runtime / Checkpointing, Runtime / > Coordination >Affects Versions: 1.10.0, 1.10.1 > Environment: flink on yarn > flink-1.10.x > jdk8 > flink-conf.yaml yarn.application-attempts: 2 (or just delete this config) > yarn-2.7.2 >Reporter: Kai Chen >Priority: Major > Attachments: flink_savepoint_path_do_not_exits.jpg, > image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1, > jobmanager.log.attemp2-13 > > > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path. > I just run this cmd: > bin/flink run -m yarn-cluster -s "hdfs:///do/not/exist/path" > examples/streaming/ > WindowJoin.jar > when i visit UI，i meet this: > !image-2020-07-31-19-04-19-241.png! > in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader > election > > Here is the stack trace in `jobmanager.err`: > ERROR ConnectionState Authentication failed > ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Could not start > recovered job 94b0911af12b61d3ee905baf1. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218) > at > org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown > Source) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown > Source) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1584) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at >
[GitHub] [flink] Thesharing commented on pull request #12917: [FLINK-18355][tests] Simplify tests of SlotPoolImpl
Thesharing commented on pull request #12917: URL: https://github.com/apache/flink/pull/12917#issuecomment-669652177 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path
[ https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Chen updated FLINK-18785: - Attachment: jobmanager.log.attemp2-13 > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path > -- > > Key: FLINK-18785 > URL: https://issues.apache.org/jira/browse/FLINK-18785 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN, Runtime / Checkpointing, Runtime / > Coordination >Affects Versions: 1.10.0, 1.10.1 > Environment: flink on yarn > flink-1.10.x > jdk8 > flink-conf.yaml yarn.application-attempts: 2 (or just delete this config) > yarn-2.7.2 >Reporter: Kai Chen >Priority: Major > Attachments: flink_savepoint_path_do_not_exits.jpg, > image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1, > jobmanager.log.attemp2-13 > > > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path. > I just run this cmd: > bin/flink run -m yarn-cluster -s "hdfs:///do/not/exist/path" > examples/streaming/ > WindowJoin.jar > when i visit UI，i meet this: > !image-2020-07-31-19-04-19-241.png! > in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader > election > > Here is the stack trace in `jobmanager.err`: > ERROR ConnectionState Authentication failed > ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Could not start > recovered job 94b0911af12b61d3ee905baf1. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218) > at > org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown > Source) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown > Source) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1584) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) > ... 4 more > Caused by: java.lang.RuntimeException: >
[jira] [Updated] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path
[ https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Chen updated FLINK-18785: - Attachment: flink_savepoint_path_do_not_exits.jpg > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path > -- > > Key: FLINK-18785 > URL: https://issues.apache.org/jira/browse/FLINK-18785 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN, Runtime / Checkpointing, Runtime / > Coordination >Affects Versions: 1.10.0, 1.10.1 > Environment: flink on yarn > flink-1.10.x > jdk8 > flink-conf.yaml yarn.application-attempts: 2 (or just delete this config) > yarn-2.7.2 >Reporter: Kai Chen >Priority: Major > Attachments: flink_savepoint_path_do_not_exits.jpg, > image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1, > jobmanager.log.attemp2-13 > > > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path. > I just run this cmd: > bin/flink run -m yarn-cluster -s "hdfs:///do/not/exist/path" > examples/streaming/ > WindowJoin.jar > when i visit UI，i meet this: > !image-2020-07-31-19-04-19-241.png! > in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader > election > > Here is the stack trace in `jobmanager.err`: > ERROR ConnectionState Authentication failed > ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Could not start > recovered job 94b0911af12b61d3ee905baf1. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218) > at > org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown > Source) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown > Source) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1584) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) > ... 4 more > Caused by:
[jira] [Updated] (FLINK-18785) flink goes into dead lock leader election when restoring from a do-not-exist checkpoint/savepoint path
[ https://issues.apache.org/jira/browse/FLINK-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Chen updated FLINK-18785: - Attachment: (was: jobmanager.log.attemp2) > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path > -- > > Key: FLINK-18785 > URL: https://issues.apache.org/jira/browse/FLINK-18785 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN, Runtime / Checkpointing, Runtime / > Coordination >Affects Versions: 1.10.0, 1.10.1 > Environment: flink on yarn > flink-1.10.x > jdk8 > flink-conf.yaml yarn.application-attempts: 2 (or just delete this config) > yarn-2.7.2 >Reporter: Kai Chen >Priority: Major > Attachments: image-2020-07-31-19-04-19-241.png, jobmanager.log.attemp1 > > > flink goes into dead lock leader election when restoring from a do-not-exist > checkpoint/savepoint path. > I just run this cmd: > bin/flink run -m yarn-cluster -s "hdfs:///do/not/exist/path" > examples/streaming/ > WindowJoin.jar > when i visit UI，i meet this: > !image-2020-07-31-19-04-19-241.png! > in flink-1.9.3, the program just exits. But in 1.10.x, it stucks in leader > election > > Here is the stack trace in `jobmanager.err`: > ERROR ConnectionState Authentication failed > ERROR ClusterEntrypoint Fatal error occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Could not start > recovered job 94b0911af12b61d3ee905baf1. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$handleRecoveredJobStartError$0(Dispatcher.java:218) > at > org.apache.flink.runtime.dispatcher.Dispatcher$$Lambda$128/130098676.apply(Unknown > Source) > at > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > at > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$60/278409878.apply(Unknown > Source) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1584) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) > ... 4 more > Caused by: java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could
[GitHub] [flink] hequn8128 commented on a change in pull request #13066: [FLINK-18765][python] Support map() and flat_map() for Python DataStream API.
hequn8128 commented on a change in pull request #13066:
URL: https://github.com/apache/flink/pull/13066#discussion_r466112647
##
File path: flink-python/pyflink/datastream/data_stream.py
##
@@ -160,3 +167,120 @@ def set_buffer_timeout(self, timeout_millis: int):
"""
self._j_data_stream.setBufferTimeout(timeout_millis)
return self
+
+def map(self, func, type_info=None):
+"""
+Applies a Map transformation on a DataStream. The transformation calls
a MapFunction for
+each element of the DataStream. Each MapFunction call returns exactly
one element. The user
+can also extend RichMapFunction to gain access to other features
provided by the
+RichFunction interface.
+
+Note that If user does not specify the output data type, the output
data will be serialized
+as pickle primitive byte array.
+
+:param func: The MapFunction that is called for each element of the
DataStream.
+:param type_info: The type information of the MapFunction output data.
+:return: The transformed DataStream.
+"""
+if not isinstance(func, MapFunction):
+if callable(func):
+func = MapFunctionWrapper(func)
+else:
+raise TypeError("The input muster be MapFunction or a callable
function")
Review comment:
The input must be a MapFunction
##
File path: flink-python/pyflink/datastream/data_stream.py
##
@@ -160,3 +167,120 @@ def set_buffer_timeout(self, timeout_millis: int):
"""
self._j_data_stream.setBufferTimeout(timeout_millis)
return self
+
+def map(self, func, type_info=None):
Review comment:
Add type hint.
```
from typing import Union, Callable
def map(self, func: Union[Callable, MapFunction], type_info: TypeInformation
= None) -> 'DataStream':
```
##
File path: flink-python/pyflink/datastream/data_stream.py
##
@@ -160,3 +167,120 @@ def set_buffer_timeout(self, timeout_millis: int):
"""
self._j_data_stream.setBufferTimeout(timeout_millis)
return self
+
+def map(self, func, type_info=None):
+"""
+Applies a Map transformation on a DataStream. The transformation calls
a MapFunction for
+each element of the DataStream. Each MapFunction call returns exactly
one element. The user
+can also extend RichMapFunction to gain access to other features
provided by the
+RichFunction interface.
+
+Note that If user does not specify the output data type, the output
data will be serialized
+as pickle primitive byte array.
+
+:param func: The MapFunction that is called for each element of the
DataStream.
+:param type_info: The type information of the MapFunction output data.
+:return: The transformed DataStream.
+"""
+if not isinstance(func, MapFunction):
+if callable(func):
+func = MapFunctionWrapper(func)
+else:
+raise TypeError("The input muster be MapFunction or a callable
function")
+func_name = "m_map_" + str(uuid.uuid1())
+j_python_data_stream_scalar_function_operator, output_type_info = \
+self._get_java_python_function_operator(func,
+type_info,
+func_name,
+flink_fn_execution_pb2
+
.UserDefinedDataStreamFunction.MAP)
+return DataStream(self._j_data_stream.transform(
+func_name,
+output_type_info.get_java_type_info(),
+j_python_data_stream_scalar_function_operator
+))
+
+def flat_map(self, func, type_info=None):
Review comment:
Type hint.
##
File path: flink-python/pyflink/datastream/data_stream.py
##
@@ -160,3 +167,120 @@ def set_buffer_timeout(self, timeout_millis: int):
"""
self._j_data_stream.setBufferTimeout(timeout_millis)
return self
+
+def map(self, func, type_info=None):
+"""
+Applies a Map transformation on a DataStream. The transformation calls
a MapFunction for
+each element of the DataStream. Each MapFunction call returns exactly
one element. The user
+can also extend RichMapFunction to gain access to other features
provided by the
+RichFunction interface.
+
+Note that If user does not specify the output data type, the output
data will be serialized
+as pickle primitive byte array.
+
Review comment:
Add an example in the python docs.
##
File path: flink-python/pyflink/datastream/data_stream.py
##
@@ -160,3 +167,120 @@ def set_buffer_timeout(self, timeout_millis: int):
"""
[GitHub] [flink] flinkbot commented on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed
flinkbot commented on pull request #13073: URL: https://github.com/apache/flink/pull/13073#issuecomment-669650373 ## CI report: * 145c86b135ee64462c449dacc81643ee8908e68b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
flinkbot edited a comment on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669640616 ## CI report: * 69e9338aba347a925c5096dd4aafe439a90921ad Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5217) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
flinkbot edited a comment on pull request #13024: URL: https://github.com/apache/flink/pull/13024#issuecomment-666148245 ## CI report: * 8731d0680f929b9754a7fc5014a890db4c5bbc74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5049) * eb611d5b39f997fa3e986b5d163cb65a44b4b0ba UNKNOWN * b2ae9dd70445adfb6e7b44deae46225841537b7d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18833) Improve the Python documentation about sql
Huang Xingbo created FLINK-18833: Summary: Improve the Python documentation about sql Key: FLINK-18833 URL: https://issues.apache.org/jira/browse/FLINK-18833 Project: Flink Issue Type: Improvement Components: API / Python, Documentation Reporter: Huang Xingbo Fix For: 1.12.0, 1.11.2 Now, there are a few documentations are written for java/scala users, which is not convenient for python users to read. We need to use a more pythonic description in docs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed
flinkbot commented on pull request #13073: URL: https://github.com/apache/flink/pull/13073#issuecomment-669647740 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 145c86b135ee64462c449dacc81643ee8908e68b (Thu Aug 06 02:35:12 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18820).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18820) SourceOperator should send MAX_WATERMARK to downstream operator when closed
[ https://issues.apache.org/jira/browse/FLINK-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuannan Su updated FLINK-18820: --- Description: SourceOperator should send MAX_WATERMARK to the downstream operator when closed. Otherwise, the watermark of the downstream operator may not advance. (was: SourceOperator should send MAX_WATERMARK to the downstream operator when closed. Otherwise, the watermark of the downstream operator will not advance. ) > SourceOperator should send MAX_WATERMARK to downstream operator when closed > --- > > Key: FLINK-18820 > URL: https://issues.apache.org/jira/browse/FLINK-18820 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Reporter: Xuannan Su >Priority: Major > Labels: pull-request-available > > SourceOperator should send MAX_WATERMARK to the downstream operator when > closed. Otherwise, the watermark of the downstream operator may not advance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Sxnan opened a new pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed
Sxnan opened a new pull request #13073: URL: https://github.com/apache/flink/pull/13073 ## What is the purpose of the change *This pull request make the SourceOperator send the Watermark.MAX_VALUE to the downstream when closing.* ## Brief change log - *Send the Watermark.MAX_VALUE in the close method of SourceOperator ## Verifying this change This change added tests and can be verified as follows: - *Added test that validates that Watermark.MAX_VALUE is sent when closing* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18820) SourceOperator should send MAX_WATERMARK to downstream operator when closed
[ https://issues.apache.org/jira/browse/FLINK-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18820: --- Labels: pull-request-available (was: ) > SourceOperator should send MAX_WATERMARK to downstream operator when closed > --- > > Key: FLINK-18820 > URL: https://issues.apache.org/jira/browse/FLINK-18820 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Reporter: Xuannan Su >Priority: Major > Labels: pull-request-available > > SourceOperator should send MAX_WATERMARK to the downstream operator when > closed. Otherwise, the watermark of the downstream operator will not advance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
flinkbot edited a comment on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669640616 ## CI report: * 69e9338aba347a925c5096dd4aafe439a90921ad Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5217) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangzzu commented on a change in pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
wangzzu commented on a change in pull request #13024:
URL: https://github.com/apache/flink/pull/13024#discussion_r466108172
##
File path:
flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java
##
@@ -293,10 +294,15 @@ private void runApplicationEntryPoint(
}
if (serializedThrowable.isPresent()) {
- Throwable throwable =
- serializedThrowable
- .get()
-
.deserializeError(application.getUserCodeClassLoader());
+ Throwable throwable;
+ try {
+ throwable = serializedThrowable
+ .get()
+
.deserializeError(application.getUserCodeClassLoader());
+ } catch (ProgramInvocationException e) {
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
flinkbot edited a comment on pull request #13024: URL: https://github.com/apache/flink/pull/13024#issuecomment-666148245 ## CI report: * 8731d0680f929b9754a7fc5014a890db4c5bbc74 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5049) * eb611d5b39f997fa3e986b5d163cb65a44b4b0ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
becketqin commented on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669645332 LGTM. We probably also want to have uber jars for other formats as well. @wuchong Do you want to take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangzzu commented on pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
wangzzu commented on pull request #13024: URL: https://github.com/apache/flink/pull/13024#issuecomment-669645338 @aljoscha Thanks for reviewing. I have modified the implementation, @wangyang0918 @aljoscha Could you help review the PR again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13066: [FLINK-18765][python] Support map() and flat_map() for Python DataStream API.
hequn8128 commented on a change in pull request #13066:
URL: https://github.com/apache/flink/pull/13066#discussion_r466100622
##
File path:
flink-python/src/main/java/org/apache/flink/datastream/runtime/functions/python/DataStreamPythonFunction.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.runtime.functions.python;
+
+import org.apache.flink.table.functions.python.PythonEnv;
+import org.apache.flink.table.functions.python.PythonFunction;
+
+/**
+ * DataStreamPythonFunction maintain the serialized python function and its
function type, which will be used in
Review comment:
DataStreamPythonFunction => {@link DataStreamPythonFunction}. Same for
other places.
##
File path:
flink-python/src/main/java/org/apache/flink/datastream/runtime/runners/python/beam/BeamDataStreamPythonStatelessFunctionRunner.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.runtime.runners.python.beam;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.datastream.runtime.typeutils.python.PythonTypeUtils;
+import org.apache.flink.fnexecution.v1.FlinkFnApi;
+import org.apache.flink.python.env.PythonEnvironmentManager;
+import org.apache.flink.python.metric.FlinkMetricContainer;
+import
org.apache.flink.streaming.api.runners.python.beam.BeamPythonStatelessFunctionRunner;
+
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+
+import javax.annotation.Nullable;
+
+import java.util.Map;
+
+/**
+ * DataStreamPythonFunctionRunner is responsible for starting a beam python
harness to execute user defined python
+ * function.
+ */
+public class BeamDataStreamPythonStatelessFunctionRunner extends
BeamPythonStatelessFunctionRunner {
Review comment:
private static final long serialVersionUID = 1L;
##
File path:
flink-python/src/main/java/org/apache/flink/datastream/runtime/functions/python/DataStreamPythonFunction.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.datastream.runtime.functions.python;
+
+import org.apache.flink.table.functions.python.PythonEnv;
+import org.apache.flink.table.functions.python.PythonFunction;
+
+/**
+ * DataStreamPythonFunction maintain the serialized python function and its
function type, which will be used in
Review comment:
maintains
##
File path:
flink-python/src/main/java/org/apache/flink/datastream/runtime/operators/python/DataStreamPythonStatelessFunctionOperator.java
##
@@ -0,0 +1,154 @@
+package
[GitHub] [flink] wangyang0918 commented on a change in pull request #12240: [FLINK-15792][k8s] Make Flink logs accessible via kubectl logs per default
wangyang0918 commented on a change in pull request #12240:
URL: https://github.com/apache/flink/pull/12240#discussion_r466107541
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##
@@ -213,10 +211,11 @@ private static String getLogging(String logFile, String
confDir, boolean hasLogb
if (hasLogback || hasLog4j) {
logging.append("-Dlog.file=").append(logFile);
if (hasLogback) {
- logging.append("
-Dlogback.configurationFile=file:").append(confDir).append("/logback.xml");
+ logging.append("
-Dlogback.configurationFile=file:").append(confDir).append("/").append(CONFIG_FILE_LOGBACK_NAME);
}
if (hasLog4j) {
- logging.append("
-Dlog4j.configurationFile=file:").append(confDir).append("/log4j.properties");
+ logging.append("
-Dlog4j.configuration=file:").append(confDir).append("/").append(CONFIG_FILE_LOG4J_NAME)
+ .append("
-Dlog4j.configurationFile=file:").append(confDir).append("/").append(CONFIG_FILE_LOG4J_NAME);
}
Review comment:
For standalone and Yarn deployment, we could support both log4j1 and
log4j2. We should do this for K8s deployment. `log4j.configuration` is for
log4j1 and `log4j.configurationFile` is for log4j2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wangzzu commented on a change in pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
wangzzu commented on a change in pull request #13024: URL: https://github.com/apache/flink/pull/13024#discussion_r466107357 ## File path: flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java ## @@ -704,7 +708,7 @@ protected void executeProgram(final Configuration configuration, final PackagedP * * @return A PackagedProgram (upon success) */ - PackagedProgram buildProgram(final ProgramOptions runOptions) + PackagedProgram buildProgram(final ProgramOptions runOptions, boolean setConfiguration) Review comment: fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangzzu commented on a change in pull request #13024: [FLINK-18742][flink-clients] Some configuration args do not take effect at client
wangzzu commented on a change in pull request #13024: URL: https://github.com/apache/flink/pull/13024#discussion_r466107256 ## File path: flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java ## @@ -516,7 +590,7 @@ private static void checkJarFile(URL jarfile) throws ProgramInvocationException private List userClassPaths = Collections.emptyList(); - private Configuration configuration = new Configuration(); + private Configuration configuration = null; Review comment: fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
flinkbot commented on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669640616 ## CI report: * 69e9338aba347a925c5096dd4aafe439a90921ad UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on a change in pull request #13051: [FLINK-18760][runtime] Redundant task managers should be released when there's no job running in session cluster
KarmaGYZ commented on a change in pull request #13051:
URL: https://github.com/apache/flink/pull/13051#discussion_r466102349
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##
@@ -1256,7 +1256,10 @@ void checkTaskManagerTimeoutsAndRedundancy() {
}
int slotsDiff = redundantTaskManagerNum *
numSlotsPerWorker - freeSlots.size();
- if (slotsDiff > 0) {
+ if (freeSlots.size() == slots.size()) {
Review comment:
If job's failover takes a long time, we could run into this scenario as
well. In that case, the redundant TM mechanism will not take effect as
expected. WDYT about this?
cc @tillrohrmann
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
flinkbot commented on pull request #13072: URL: https://github.com/apache/flink/pull/13072#issuecomment-669638015 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 69e9338aba347a925c5096dd4aafe439a90921ad (Thu Aug 06 02:00:19 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-18802).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18802) Create an uber jar when packaging flink-avro
[ https://issues.apache.org/jira/browse/FLINK-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18802: --- Labels: pull-request-available (was: ) > Create an uber jar when packaging flink-avro > > > Key: FLINK-18802 > URL: https://issues.apache.org/jira/browse/FLINK-18802 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > Currently users have to provide dependencies such as avro, jackson-core-asl, > jackson-mapper-asl and joda-time in the job jar for DataStream jobs, or > manually copy them into flink/lib in SQL jobs when using avro formatting. > It's better to generate an uber jar including these dependencies when > packaging flink-avro module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] PatrickRen opened a new pull request #13072: [FLINK-18802][formats] Package uber jar including all dependencies for flink-avro
PatrickRen opened a new pull request #13072: URL: https://github.com/apache/flink/pull/13072 ## What is the purpose of the change This pull request changes the pom.xml of flink-avro module in order to create a uber jar in packaging phase, including dependencies such as avro, jackson-core-asl, jackson-mapper-asl and joda-time. ## Brief change log - Add another execution in maven-assembly-plugin to create a jar-with-dependencies - Remove the "provided" scope and "optional" tag of joda-time dependency to include it in the uber jar ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hequn8128 commented on a change in pull request #13066: [FLINK-18765][python] Support map() and flat_map() for Python DataStream API.
hequn8128 commented on a change in pull request #13066:
URL: https://github.com/apache/flink/pull/13066#discussion_r465607850
##
File path: flink-python/pyflink/proto/flink-fn-execution.proto
##
@@ -147,3 +163,50 @@ message Schema {
repeated Field fields = 1;
}
+
+// A representation of the data type information of a data stream.
+message TypeInfo {
Review comment:
Please remove the useless type, e.g., VOID, LOCAL_DATE, etc.
##
File path: flink-python/pyflink/fn_execution/operation_utils.py
##
@@ -20,11 +20,13 @@
import cloudpickle
from typing import Any, Tuple, Dict, List
+from pyflink.fn_execution import flink_fn_execution_pb2
from pyflink.serializers import PickleSerializer
from pyflink.table.udf import DelegationTableFunction, DelegatingScalarFunction
SCALAR_FUNCTION_URN = "flink:transform:scalar_function:v1"
TABLE_FUNCTION_URN = "flink:transform:table_function:v1"
+DATA_STREAM_FUNCTION_URN = "flink:transform:datastream_stateless_function:v1"
Review comment:
DATA_STREAM_STATELESS_FUNCTION_URN?
##
File path: flink-python/pyflink/proto/flink-fn-execution.proto
##
@@ -52,6 +52,22 @@ message UserDefinedFunctions {
bool metric_enabled = 2;
}
+// User defined data stream function definition.
Review comment:
data stream => DataStream. And for other places.
##
File path: flink-python/pyflink/fn_execution/coders.py
##
@@ -271,6 +309,21 @@ def __init__(self, precision, scale):
def get_impl(self):
return coder_impl.DecimalCoderImpl(self.precision, self.scale)
+def get_slow_impl(self):
+return coder_impl_slow.DecimalCoderImpl(self.precision, self.scale)
+
+
+class BasicDecimalCoder(FieldCoder):
Review comment:
Rename to `BigDecimalCoder`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-18819) All the PubSub tests are failing
[
https://issues.apache.org/jira/browse/FLINK-18819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171862#comment-17171862
]
Dian Fu commented on FLINK-18819:
-
Downgrade to "Critical" as it doesn't occur any more except the above instances.
> All the PubSub tests are failing
>
>
> Key: FLINK-18819
> URL: https://issues.apache.org/jira/browse/FLINK-18819
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub, Tests
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5177=logs=08866332-78f7-59e4-4f7e-49a56faa3179=7f606211-1454-543c-70ab-c7a028a1ce8c
> {code}
> 2020-08-04T21:18:11.4228635Z 63538 [main] INFO
> com.spotify.docker.client.DefaultDockerClient [] - Starting container with
> Id: a40e3b941371ddd82f4efe0ec6f72b89f36868316e5d8cc52dacc64f0bc79029
> 2020-08-04T21:18:13.1868394Z [ERROR] Tests run: 3, Failures: 1, Errors: 2,
> Skipped: 0, Time elapsed: 64.94 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest
> 2020-08-04T21:18:13.1869974Z [ERROR]
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest Time
> elapsed: 64.938 s <<< FAILURE!
> 2020-08-04T21:18:13.1870671Z java.lang.AssertionError: We expect 1 port to be
> mapped expected:<1> but was:<0>
> 2020-08-04T21:18:13.1871219Z at org.junit.Assert.fail(Assert.java:88)
> 2020-08-04T21:18:13.1871693Z at
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2020-08-04T21:18:13.1872167Z at
> org.junit.Assert.assertEquals(Assert.java:645)
> 2020-08-04T21:18:13.1872808Z at
> org.apache.flink.streaming.connectors.gcp.pubsub.emulator.GCloudEmulatorManager.launchDocker(GCloudEmulatorManager.java:141)
> 2020-08-04T21:18:13.1873573Z at
> org.apache.flink.streaming.connectors.gcp.pubsub.emulator.GCloudUnitTestBase.launchGCloudEmulator(GCloudUnitTestBase.java:45)
> 2020-08-04T21:18:13.1874205Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-08-04T21:18:13.1874742Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-08-04T21:18:13.1875353Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-08-04T21:18:13.1875912Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-08-04T21:18:13.1876472Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-08-04T21:18:13.1877081Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-08-04T21:18:13.1877695Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-08-04T21:18:13.1878283Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 2020-08-04T21:18:13.1878876Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-08-04T21:18:13.1879425Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-08-04T21:18:13.1880172Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-08-04T21:18:13.1880815Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-08-04T21:18:13.1881464Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-08-04T21:18:13.1882083Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-08-04T21:18:13.1882736Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-08-04T21:18:13.1883399Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-08-04T21:18:13.1883999Z at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> 2020-08-04T21:18:13.1884593Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> 2020-08-04T21:18:13.1884989Z
> 2020-08-04T21:18:13.1885496Z [ERROR]
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest Time
> elapsed: 64.939 s <<< ERROR!
> 2020-08-04T21:18:13.1886024Z java.lang.NullPointerException
> 2020-08-04T21:18:13.1886588Z at
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest.tearDown(EmulatedPubSubSinkTest.java:62)
> 2020-08-04T21:18:13.1887193Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-08-04T21:18:13.1887864Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-08-04T21:18:13.1888483Z at
>
[jira] [Updated] (FLINK-18819) All the PubSub tests are failing
[
https://issues.apache.org/jira/browse/FLINK-18819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18819:
Priority: Critical (was: Blocker)
> All the PubSub tests are failing
>
>
> Key: FLINK-18819
> URL: https://issues.apache.org/jira/browse/FLINK-18819
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub, Tests
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5177=logs=08866332-78f7-59e4-4f7e-49a56faa3179=7f606211-1454-543c-70ab-c7a028a1ce8c
> {code}
> 2020-08-04T21:18:11.4228635Z 63538 [main] INFO
> com.spotify.docker.client.DefaultDockerClient [] - Starting container with
> Id: a40e3b941371ddd82f4efe0ec6f72b89f36868316e5d8cc52dacc64f0bc79029
> 2020-08-04T21:18:13.1868394Z [ERROR] Tests run: 3, Failures: 1, Errors: 2,
> Skipped: 0, Time elapsed: 64.94 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest
> 2020-08-04T21:18:13.1869974Z [ERROR]
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest Time
> elapsed: 64.938 s <<< FAILURE!
> 2020-08-04T21:18:13.1870671Z java.lang.AssertionError: We expect 1 port to be
> mapped expected:<1> but was:<0>
> 2020-08-04T21:18:13.1871219Z at org.junit.Assert.fail(Assert.java:88)
> 2020-08-04T21:18:13.1871693Z at
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2020-08-04T21:18:13.1872167Z at
> org.junit.Assert.assertEquals(Assert.java:645)
> 2020-08-04T21:18:13.1872808Z at
> org.apache.flink.streaming.connectors.gcp.pubsub.emulator.GCloudEmulatorManager.launchDocker(GCloudEmulatorManager.java:141)
> 2020-08-04T21:18:13.1873573Z at
> org.apache.flink.streaming.connectors.gcp.pubsub.emulator.GCloudUnitTestBase.launchGCloudEmulator(GCloudUnitTestBase.java:45)
> 2020-08-04T21:18:13.1874205Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-08-04T21:18:13.1874742Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-08-04T21:18:13.1875353Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-08-04T21:18:13.1875912Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-08-04T21:18:13.1876472Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-08-04T21:18:13.1877081Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-08-04T21:18:13.1877695Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-08-04T21:18:13.1878283Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> 2020-08-04T21:18:13.1878876Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-08-04T21:18:13.1879425Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-08-04T21:18:13.1880172Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-08-04T21:18:13.1880815Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-08-04T21:18:13.1881464Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-08-04T21:18:13.1882083Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-08-04T21:18:13.1882736Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-08-04T21:18:13.1883399Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-08-04T21:18:13.1883999Z at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> 2020-08-04T21:18:13.1884593Z at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> 2020-08-04T21:18:13.1884989Z
> 2020-08-04T21:18:13.1885496Z [ERROR]
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest Time
> elapsed: 64.939 s <<< ERROR!
> 2020-08-04T21:18:13.1886024Z java.lang.NullPointerException
> 2020-08-04T21:18:13.1886588Z at
> org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSinkTest.tearDown(EmulatedPubSubSinkTest.java:62)
> 2020-08-04T21:18:13.1887193Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-08-04T21:18:13.1887864Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-08-04T21:18:13.1888483Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-08-04T21:18:13.1889046Z at
>
[jira] [Commented] (FLINK-18681) The jar package version conflict causes the task to continue to increase and grab resources
[
https://issues.apache.org/jira/browse/FLINK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171859#comment-17171859
]
Tao Yang commented on FLINK-18681:
--
Hi, [~apach...@163.com], the RM log seems totally abnormal since no any INFO
logs in it. The normal RM log will show submission process and all allocated
containers for new apps which can't be found in the RM log you provided. Could
you please check if the RM log file are filtered and only WARN/ERROR logs left?
> The jar package version conflict causes the task to continue to increase and
> grab resources
> ---
>
> Key: FLINK-18681
> URL: https://issues.apache.org/jira/browse/FLINK-18681
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: wangtaiyang
>Priority: Major
> Attachments: appId.log, dependency.log,
> image-2020-07-28-15-32-51-851.png,
> yarn-hadoop-resourcemanager-x.x.x.15.log.2020-07-22-17.log
>
>
> When I submit a flink task to yarn, the default resource configuration is
> 1G&1core, but in fact this task will always increase resources 2core, 3core,
> and so on. . . 200core. . . Then I went to look at the JM log and found the
> following error:
> {code:java}
> //代码占位符
> java.lang.NoSuchMethodError:
> org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;java.lang.NoSuchMethodError:
>
> org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
> at
> org.apache.flink.runtime.entrypoint.parser.CommandLineOptions.(CommandLineOptions.java:28)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1] at
> org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:648)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1] at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
> ~[?:1.8.0_191]
> ...
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.flink.runtime.entrypoint.parser.CommandLineOptionsjava.lang.NoClassDefFoundError:
> Could not initialize class
> org.apache.flink.runtime.entrypoint.parser.CommandLineOptions at
> org.apache.flink.runtime.clusterframework.BootstrapTools.lambda$getDynamicPropertiesAsString$0(BootstrapTools.java:648)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1] at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
> ~[?:1.8.0_191] at
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
> ~[?:1.8.0_191] at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> ~[?:1.8.0_191]{code}
> Finally, it is confirmed that it is caused by the commands-cli version
> conflict, but the task reporting error has not stopped and will continue to
> grab resources and increase. Is this a bug?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] Myracle commented on pull request #13051: [FLINK-18760][runtime] Redundant task managers should be released when there's no job running in session cluster
Myracle commented on pull request #13051: URL: https://github.com/apache/flink/pull/13051#issuecomment-669609819 @xintongsong @KarmaGYZ Can you review my code? Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18796) FlinkKinesisProducer.backpressureLatch should be volatile
[ https://issues.apache.org/jira/browse/FLINK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171766#comment-17171766 ] Chesnay Schepler commented on FLINK-18796: -- Isn't the same true for the FlinkKinesisProducer? The KinesisProducer internally uses a ThreadPoolExecutor to complete the callbacks, which creates threads lazily when required, so until we start processing data (i.e., call invoke), no thread has been created yet. > FlinkKinesisProducer.backpressureLatch should be volatile > - > > Key: FLINK-18796 > URL: https://issues.apache.org/jira/browse/FLINK-18796 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.11.1 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > (confirm first) > > cc: [~rmetzger] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18796) FlinkKinesisProducer.backpressureLatch should be volatile
[ https://issues.apache.org/jira/browse/FLINK-18796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171753#comment-17171753 ] Roman Khachatryan commented on FLINK-18796: --- CassandraSinkBase is a bit different in that it starts other threads after assigning this state, so there is a HB relationship. I'm also not sure that there is any noticeable performance difference, as there are already volatile accesses there (and eventually IO). > FlinkKinesisProducer.backpressureLatch should be volatile > - > > Key: FLINK-18796 > URL: https://issues.apache.org/jira/browse/FLINK-18796 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis >Affects Versions: 1.11.1 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > (confirm first) > > cc: [~rmetzger] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13071: [FLINK-18751][Coordination] Implement SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13071: URL: https://github.com/apache/flink/pull/13071#issuecomment-669310861 ## CI report: * 8a735691f03d9abd842d7917303ccf9c9fe21e56 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5212) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13042: [FLINK-16510] Allow configuring shutdown behavior to avoid JVM freeze
flinkbot edited a comment on pull request #13042: URL: https://github.com/apache/flink/pull/13042#issuecomment-667268558 ## CI report: * 0d7596f938ca8c655fef4c011d67e921d3a4b946 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5092) * 94ce334e46321af04393fdcf7219f4c8c80ba655 UNKNOWN * 8826ac20d30fc87d026d2406bf4de1bdcb4ddeb4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5214) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13054: [FLINK-15803][table] Support DataViews in FLIP-65 aggregate functions
flinkbot edited a comment on pull request #13054: URL: https://github.com/apache/flink/pull/13054#issuecomment-668030431 ## CI report: * 26381042613df4f94aa57feb72a50665836ad7d1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5211) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13042: [FLINK-16510] Allow configuring shutdown behavior to avoid JVM freeze
flinkbot edited a comment on pull request #13042: URL: https://github.com/apache/flink/pull/13042#issuecomment-667268558 ## CI report: * 0d7596f938ca8c655fef4c011d67e921d3a4b946 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5092) * 94ce334e46321af04393fdcf7219f4c8c80ba655 UNKNOWN * d1104b36d044153572bd61172b03c7af3cf8c9b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5213) * 8826ac20d30fc87d026d2406bf4de1bdcb4ddeb4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5214) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13042: [FLINK-16510] Allow configuring shutdown behavior to avoid JVM freeze
flinkbot edited a comment on pull request #13042: URL: https://github.com/apache/flink/pull/13042#issuecomment-667268558 ## CI report: * 0d7596f938ca8c655fef4c011d67e921d3a4b946 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5092) * 94ce334e46321af04393fdcf7219f4c8c80ba655 UNKNOWN * d1104b36d044153572bd61172b03c7af3cf8c9b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5213) * 8826ac20d30fc87d026d2406bf4de1bdcb4ddeb4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13042: [FLINK-16510] Allow configuring shutdown behavior to avoid JVM freeze
zentol commented on a change in pull request #13042:
URL: https://github.com/apache/flink/pull/13042#discussion_r465931031
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/ClusterOptions.java
##
@@ -78,4 +79,13 @@
.text("Enable the slot spread out allocation
strategy. This strategy tries to spread out " +
"the slots evenly across all available
%s.", code("TaskExecutors"))
.build());
+
+ @Documentation.Section(Documentation.Sections.EXPERT_CLUSTER)
+ public static final ConfigOption HALT_ON_FATAL_ERROR =
+ key("cluster.processes.halt-on-fatal-error")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Whether processes should halt on
fatal errors instead of performing a graceful shutdown. " +
+ "In some environments (e.g. Java 8 with the G1
garbage collector), a regular graceful shutdown can lead " +
+ "to a JVM deadlock. See:
https://issues.apache.org/jira/browse/FLINK-16510;);
Review comment:
```suggestion
.withDescription(Description.builder().text(
"Whether processes should halt on fatal errors
instead of performing a graceful shutdown. " +
"In some environments (e.g. Java 8 with
the G1 garbage collector), a regular graceful shutdown can lead " +
"to a JVM deadlock. See %s for
details.",
link("https://issues.apache.org/jira/browse/FLINK-16510;, "FLINK-16510")
.build());
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
