[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[
https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-7104:
Description:
Actually, the dir of {{junit2471804890898765737}} had already not existed
anymore. But the program still access it. Then, cause build and test failure.
The reason for this issue is that when one of the threads deletes the generated
file, the other thread still continues to delete the file that had been
deleted, thats leading to AccessDeniedException.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
was:
Actually, the dir of {{junit2471804890898765737}} had already not existed
anymore. But the program still access it. Then, cause build and test failure.
The reason for this issue is that when one of the threads deletes the generated
file, the other thread still continues to delete a file that has been deleted,
thats leading to AccessDeniedException.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
> AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently
> call
> -
>
> Key: FLINK-7
[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[
https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-7104:
Component/s: Tests
> AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently
> call
> -
>
> Key: FLINK-7104
> URL: https://issues.apache.org/jira/browse/FLINK-7104
> Project: Flink
> Issue Type: Bug
> Components: Build System, Core, Tests
>Reporter: mingleizhang
>Priority: Critical
>
> Actually, the dir of {{junit2471804890898765737}} had already not existed
> anymore. But the program still access it. Then, cause build and test failure.
> The reason for this issue is that when one of the threads deletes the
> generated file, the other thread still continues to delete a file that has
> been deleted, thats leading to AccessDeniedException.
> {code:java}
> java.nio.file.AccessDeniedException:
> D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
> at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> at java.nio.file.Files.delete(Files.java:1126)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
> at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
> at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
> at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
> at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
> at
> org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
> at
> org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[
https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-7104:
Description:
Actually, the dir of {{junit2471804890898765737}} had already not existed
anymore. But the program still access it. Then, cause build and test failure.
The reason for this issue is that when one of the threads deletes the generated
file, the other thread still continues to delete a file that has been deleted,
thats leading to AccessDeniedException.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
was:
Actually, the dir of {{junit2471804890898765737}} had already not existed
anymore. But the program still access it. Then, cause build failure.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
> AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently
> call
> -
>
> Key: FLINK-7104
> URL: https://issues.apache.org/jira/browse/FLINK-7104
> Project: Flink
> Issue Type: Bug
> Components: Build System, Core
>Reporter: mingleizhang
>
[jira] [Commented] (FLINK-6499) Migrate state configuration options
[
https://issues.apache.org/jira/browse/FLINK-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074293#comment-16074293
]
ASF GitHub Bot commented on FLINK-6499:
---
Github user zjureel commented on the issue:
https://github.com/apache/flink/pull/4173
@tzulitai Thank you for merging this :)
> Migrate state configuration options
> ---
>
> Key: FLINK-6499
> URL: https://issues.apache.org/jira/browse/FLINK-6499
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
>
> {code}
> /** The default directory for savepoints. */
> @PublicEvolving
> public static final String SAVEPOINT_DIRECTORY_KEY =
> "state.savepoints.dir";
> /** The default directory used for persistent checkpoints. */
> @PublicEvolving
> public static final String CHECKPOINTS_DIRECTORY_KEY =
> "state.checkpoints.dir";
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4173: [FLINK-6499] Migrate state configuration options
Github user zjureel commented on the issue: https://github.com/apache/flink/pull/4173 @tzulitai Thank you for merging this :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4187: [FLINK-6998][Kafka Connector] Add kafka offset commit met...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4187 Hi @zhenzhongxu, could you rebase on to the latest master? Currently the PR contains commits unrelated to the change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6998) Kafka connector needs to expose metrics for failed/successful offset commits in the Kafka Consumer callback
[ https://issues.apache.org/jira/browse/FLINK-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074290#comment-16074290 ] ASF GitHub Bot commented on FLINK-6998: --- Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4187 Hi @zhenzhongxu, could you rebase on to the latest master? Currently the PR contains commits unrelated to the change. > Kafka connector needs to expose metrics for failed/successful offset commits > in the Kafka Consumer callback > --- > > Key: FLINK-6998 > URL: https://issues.apache.org/jira/browse/FLINK-6998 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Zhenzhong Xu >Assignee: Zhenzhong Xu > > Propose to add "kafkaCommitsSucceeded" and "kafkaCommitsFailed" metrics in > KafkaConsumerThread class. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6747) Table API / SQL Docs: Streaming Page
[
https://issues.apache.org/jira/browse/FLINK-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074237#comment-16074237
]
ASF GitHub Bot commented on FLINK-6747:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125539538
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
+
+// convert result Table into a DataStream
+DataStream stream = tableEnv.toAppendStream(result, Row.class,
qConfig);
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// obtain query configuration from TableEnvironment
+val qConfig: StreamQueryConfig = tableEnv.queryConfig
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12))
+...
+
+// define query
+val result: Table = ???
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig)
--- End diff --
Add a `sink: TableSink[Row] = ???` just a suggest.
> Table API / SQL Docs: Streaming Page
>
>
> Key: FLINK-6747
> URL: https://issues.apache.org/jira/browse/FLINK-6747
> Project: Flink
> Issue Type: Task
> Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> Extend {{./docs/dev/table/streaming.md}} page.
> Missing are sections about
> - Dynamic Tables
> - QueryConfiguration (state retention time)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6747) Table API / SQL Docs: Streaming Page
[
https://issues.apache.org/jira/browse/FLINK-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074239#comment-16074239
]
ASF GitHub Bot commented on FLINK-6747:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125540625
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
--- End diff --
I think add line break look more comfortable，but i am fine to keep current
style.(same as bellow)
> Table API / SQL Docs: Streaming Page
>
>
> Key: FLINK-6747
> URL: https://issues.apache.org/jira/browse/FLINK-6747
> Project: Flink
> Issue Type: Task
> Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> Extend {{./docs/dev/table/streaming.md}} page.
> Missing are sections about
> - Dynamic Tables
> - QueryConfiguration (state retention time)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6747) Table API / SQL Docs: Streaming Page
[
https://issues.apache.org/jira/browse/FLINK-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074236#comment-16074236
]
ASF GitHub Bot commented on FLINK-6747:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125553376
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
+
+// convert result Table into a DataStream
+DataStream stream = tableEnv.toAppendStream(result, Row.class,
qConfig);
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// obtain query configuration from TableEnvironment
+val qConfig: StreamQueryConfig = tableEnv.queryConfig
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12))
+...
+
+// define query
+val result: Table = ???
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig)
+
+// convert result Table into a DataStream
+val stream: DataStream[Row] = result.toAppendStream[Row](qConfig)
+
+{% endhighlight %}
+
+
+
+In the the following we describe the parameters of the `QueryConfig` and
how they affect the accuracy and resource consumption of a query.
+
+### Idle State Retention Time
+
+Many queries aggregate or join records on one or more key attributes. When
such a query is executed on a stream, the resulting continuous query needs to
collect records or maintain partial results per key. If the key domain of the
input stream is evolving, i.e., the active key values are changing over time,
the continuous query accumulates more and more state as distinct keys are
observed. However, often keys become inactive after some time and their
corresponding state becomes stale and useless.
+
+For example the following query computes the number of clicks per session.
+
+```
+SELECT sessionId, COUNT(*) FROM clicks GROUP BY sessionId;
+```
+
+The `sessionId` attribute is used as a grouping key and the continuous
query maintains a count for each session it observes. The `sessionId` attribute
is evolving over time and `sessionId` values are only active until the session
ends, i.e., for a limited period of time. However, the continuous query cannot
know about this property of `sessionId` and has to expect that any `sessionId`
value can occur at any time. Therefore, it maintains the current count for each
observed `sessionId` value. Consequently,
[jira] [Commented] (FLINK-6747) Table API / SQL Docs: Streaming Page
[
https://issues.apache.org/jira/browse/FLINK-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074238#comment-16074238
]
ASF GitHub Bot commented on FLINK-6747:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125539485
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
--- End diff --
Do we need add `TableSink sink = ...` just a suggest.
> Table API / SQL Docs: Streaming Page
>
>
> Key: FLINK-6747
> URL: https://issues.apache.org/jira/browse/FLINK-6747
> Project: Flink
> Issue Type: Task
> Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> Extend {{./docs/dev/table/streaming.md}} page.
> Missing are sections about
> - Dynamic Tables
> - QueryConfiguration (state retention time)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6747) Table API / SQL Docs: Streaming Page
[
https://issues.apache.org/jira/browse/FLINK-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074235#comment-16074235
]
ASF GitHub Bot commented on FLINK-6747:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125538537
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
+
+// convert result Table into a DataStream
+DataStream stream = tableEnv.toAppendStream(result, Row.class,
qConfig);
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// obtain query configuration from TableEnvironment
+val qConfig: StreamQueryConfig = tableEnv.queryConfig
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12))
+...
+
+// define query
+val result: Table = ???
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig)
+
+// convert result Table into a DataStream
--- End diff --
DataStream -> DataStream[Row] ?
> Table API / SQL Docs: Streaming Page
>
>
> Key: FLINK-6747
> URL: https://issues.apache.org/jira/browse/FLINK-6747
> Project: Flink
> Issue Type: Task
> Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> Extend {{./docs/dev/table/streaming.md}} page.
> Missing are sections about
> - Dynamic Tables
> - QueryConfiguration (state retention time)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4256: [FLINK-6747] [docs] Add documentation for QueryCon...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125539485
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
--- End diff --
Do we need add `TableSink sink = ...` just a suggest.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4256: [FLINK-6747] [docs] Add documentation for QueryCon...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125539538
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
+
+// convert result Table into a DataStream
+DataStream stream = tableEnv.toAppendStream(result, Row.class,
qConfig);
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// obtain query configuration from TableEnvironment
+val qConfig: StreamQueryConfig = tableEnv.queryConfig
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12))
+...
+
+// define query
+val result: Table = ???
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig)
--- End diff --
Add a `sink: TableSink[Row] = ???` just a suggest.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4256: [FLINK-6747] [docs] Add documentation for QueryCon...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125540625
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
--- End diff --
I think add line break look more comfortableï¼but i am fine to keep
current style.(same as bellow)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4256: [FLINK-6747] [docs] Add documentation for QueryCon...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125553376
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
+
+// convert result Table into a DataStream
+DataStream stream = tableEnv.toAppendStream(result, Row.class,
qConfig);
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// obtain query configuration from TableEnvironment
+val qConfig: StreamQueryConfig = tableEnv.queryConfig
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12))
+...
+
+// define query
+val result: Table = ???
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig)
+
+// convert result Table into a DataStream
+val stream: DataStream[Row] = result.toAppendStream[Row](qConfig)
+
+{% endhighlight %}
+
+
+
+In the the following we describe the parameters of the `QueryConfig` and
how they affect the accuracy and resource consumption of a query.
+
+### Idle State Retention Time
+
+Many queries aggregate or join records on one or more key attributes. When
such a query is executed on a stream, the resulting continuous query needs to
collect records or maintain partial results per key. If the key domain of the
input stream is evolving, i.e., the active key values are changing over time,
the continuous query accumulates more and more state as distinct keys are
observed. However, often keys become inactive after some time and their
corresponding state becomes stale and useless.
+
+For example the following query computes the number of clicks per session.
+
+```
+SELECT sessionId, COUNT(*) FROM clicks GROUP BY sessionId;
+```
+
+The `sessionId` attribute is used as a grouping key and the continuous
query maintains a count for each session it observes. The `sessionId` attribute
is evolving over time and `sessionId` values are only active until the session
ends, i.e., for a limited period of time. However, the continuous query cannot
know about this property of `sessionId` and has to expect that any `sessionId`
value can occur at any time. Therefore, it maintains the current count for each
observed `sessionId` value. Consequently, the total state size of the query is
continuously growing as more and more `sessionId` values are observed.
+
+The *Idle State Retention Time* defines for how long the state of a key
may not be updated before it is removed. For the previous exa
[GitHub] flink pull request #4256: [FLINK-6747] [docs] Add documentation for QueryCon...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4256#discussion_r125538537
--- Diff: docs/dev/table/streaming.md ---
@@ -351,13 +351,109 @@ val windowedTable = tEnv
Query Configuration
---
-In stream processing, compuations are constantly happening and there are
many use cases that require to update previously emitted results. There are
many ways in which a query can compute and emit updates. These do not affect
the semantics of the query but might lead to approximated results.
+Table API and SQL queries have the same semantics regardless whether their
input is bounded batch input or unbounded stream input. In many cases,
continuous queries on streaming input are capable of computing accurate results
that are identical to offline computed results. However, this is not possible
in general case because continuous queries have to restrict the size of state
they maintain in order to avoid to run out of storage and to be able to process
unbounded streaming data over a long period of time. Consequently, a continuous
query might only be able to provide approximated results depending on the
characteristics of the input data and the query itself.
-Flink's Table API and SQL interface use a `QueryConfig` to control the
computation and emission of results and updates.
+Flink's Table API and SQL interface provide parameters to tune the
accuracy and resource consumption of continuous queries. The parameters are
specified via a `QueryConfig` object. The `QueryConfig` can be obtained from
the `TableEnvironment` and is passed back when a `Table` is translated, i.e.,
when it is [transformed into a
DataStream](common.html#convert-a-table-into-a-datastream-or-dataset) or
[emitted via a TableSink](common.html#emit-a-table).
-### State Retention
+
+
+{% highlight java %}
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv =
TableEnvironment.getTableEnvironment(env);
+
+// obtain query configuration from TableEnvironment
+StreamQueryConfig qConfig = tableEnv.queryConfig();
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12));
+...
+
+// define query
+Table result = ...
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig);
+
+// convert result Table into a DataStream
+DataStream stream = tableEnv.toAppendStream(result, Row.class,
qConfig);
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// obtain query configuration from TableEnvironment
+val qConfig: StreamQueryConfig = tableEnv.queryConfig
+// set query parameters
+qConfig.withIdleStateRetentionTime(Time.hours(12))
+...
+
+// define query
+val result: Table = ???
+
+// emit result Table via a TableSink
+result.writeToSink(sink, qConfig)
+
+// convert result Table into a DataStream
--- End diff --
DataStream -> DataStream[Row] ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Comment Edited] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074221#comment-16074221 ] Luke edited comment on FLINK-6988 at 7/5/17 4:31 AM: - Is there plan for flink producer connector to support multiple topics transactional commit in one sink? Since now kafka support transactional behavior, it can be very useful in use case where records send to different topics can be committed or rollback in same transaction state. was (Author: luke_dev86): Is there plan for flink producer connector to support multiple topics in one sink? Since now kafka support transactional behavior, it can be very useful in use case where records send to different topics can be committed or rollback in same transaction state. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to an output Kafka topic using that > transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6499) Migrate state configuration options
[
https://issues.apache.org/jira/browse/FLINK-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074230#comment-16074230
]
ASF GitHub Bot commented on FLINK-6499:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4173
Thanks for the contribution @zjureel.
LGTM, merging this ..
> Migrate state configuration options
> ---
>
> Key: FLINK-6499
> URL: https://issues.apache.org/jira/browse/FLINK-6499
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
>
> {code}
> /** The default directory for savepoints. */
> @PublicEvolving
> public static final String SAVEPOINT_DIRECTORY_KEY =
> "state.savepoints.dir";
> /** The default directory used for persistent checkpoints. */
> @PublicEvolving
> public static final String CHECKPOINTS_DIRECTORY_KEY =
> "state.checkpoints.dir";
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4173: [FLINK-6499] Migrate state configuration options
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4173 Thanks for the contribution @zjureel. LGTM, merging this .. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6988) Add Apache Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074221#comment-16074221 ] Luke commented on FLINK-6988: - Is there plan for flink producer connector to support multiple topics in one sink? Since now kafka support transactional behavior, it can be very useful in use case where records send to different topics can be committed or rollback in same transaction state. > Add Apache Kafka 0.11 connector > --- > > Key: FLINK-6988 > URL: https://issues.apache.org/jira/browse/FLINK-6988 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.3.1 >Reporter: Piotr Nowojski >Assignee: Piotr Nowojski > > Kafka 0.11 (it will be released very soon) add supports for transactions. > Thanks to that, Flink might be able to implement Kafka sink supporting > "exactly-once" semantic. API changes and whole transactions support is > described in > [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging]. > The goal is to mimic implementation of existing BucketingSink. New > FlinkKafkaProducer011 would > * upon creation begin transaction, store transaction identifiers into the > state and would write all incoming data to an output Kafka topic using that > transaction > * on `snapshotState` call, it would flush the data and write in state > information that current transaction is pending to be committed > * on `notifyCheckpointComplete` we would commit this pending transaction > * in case of crash between `snapshotState` and `notifyCheckpointComplete` we > either abort this pending transaction (if not every participant successfully > saved the snapshot) or restore and commit it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[
https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-7104:
Description:
Actually, the dir of {{junit2471804890898765737}} had already not existed
anymore. But the program still access it. Then, cause build failure.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
was:
Actually, the dir of {{junit7630178798271244089}} had already not existed
anymore. But the program still access it.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
> AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently
> call
> -
>
> Key: FLINK-7104
> URL: https://issues.apache.org/jira/browse/FLINK-7104
> Project: Flink
> Issue Type: Bug
> Components: Build System, Core
>Reporter: mingleizhang
>Priority: Critical
>
> Actually, the dir of {{junit2471804890898765737}} had already not existed
> anymore. But the program still access it. Then, cause build failure.
> {code:java}
> java.nio.file.AccessDeniedException:
> D:
[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[
https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mingleizhang updated FLINK-7104:
Description:
Actually, the dir of {{junit7630178798271244089}} had already not existed
anymore. But the program still access it.
{code:java}
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
{code}
was:
java.nio.file.AccessDeniedException:
D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111)
at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219)
at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144)
at
org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159)
at
org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)
> AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently
> call
> -
>
> Key: FLINK-7104
> URL: https://issues.apache.org/jira/browse/FLINK-7104
> Project: Flink
> Issue Type: Bug
> Components: Build System, Core
>Reporter: mingleizhang
>Priority: Critical
>
> Actually, the dir of {{junit7630178798271244089}} had already not existed
> anymore. But the program still access it.
> {code:java}
> java.nio.file.AccessDeniedException:
> D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf
[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[ https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingleizhang updated FLINK-7104: Component/s: Build System > AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently > call > - > > Key: FLINK-7104 > URL: https://issues.apache.org/jira/browse/FLINK-7104 > Project: Flink > Issue Type: Bug > Components: Build System, Core >Reporter: mingleizhang >Priority: Critical > > java.nio.file.AccessDeniedException: > D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94 > at > sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) > at > sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) > at > sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) > at > sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159) > at > org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
[ https://issues.apache.org/jira/browse/FLINK-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingleizhang updated FLINK-7104: Description: java.nio.file.AccessDeniedException: D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94 at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) at java.nio.file.Files.delete(Files.java:1126) at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115) at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) at org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159) at org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74) > AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently > call > - > > Key: FLINK-7104 > URL: https://issues.apache.org/jira/browse/FLINK-7104 > Project: Flink > Issue Type: Bug > Components: Build System, Core >Reporter: mingleizhang >Priority: Critical > > java.nio.file.AccessDeniedException: > D:\Users\rice.zhang\AppData\Local\Temp\junit2471804890898765737\junit7630178798271244089\018f436fbebbe13597ea323f72953d27\31c397e4e80ebaf0d1809c96ddc3a25b\29aab6838e71db153c456b2da5b541b7\6abf9809045c5fc61c09e33369144e94 > at > sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) > at > sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) > at > sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) > at > sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:115) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:111) > at org.apache.flink.util.FileUtils.cleanDirectory(FileUtils.java:219) > at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:144) > at > org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:159) > at > org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLINK-7104) AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call
mingleizhang created FLINK-7104: --- Summary: AccessDeniedException get when FileUtilsTest#testDeleteDirectoryConcurrently call Key: FLINK-7104 URL: https://issues.apache.org/jira/browse/FLINK-7104 Project: Flink Issue Type: Bug Components: Core Reporter: mingleizhang Priority: Critical -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6969) Add support for deferred computation for group window aggregates
[
https://issues.apache.org/jira/browse/FLINK-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074169#comment-16074169
]
ASF GitHub Bot commented on FLINK-6969:
---
Github user wuchong commented on the issue:
https://github.com/apache/flink/pull/4183
The proposal makes sense to me.
BTW, the concepts of `earlyResultTimeOffset` and `updateRate` also works on
window group aggregates, right?
> Add support for deferred computation for group window aggregates
>
>
> Key: FLINK-6969
> URL: https://issues.apache.org/jira/browse/FLINK-6969
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: sunjincheng
>
> Deferred computation is a strategy to deal with late arriving data and avoid
> updates of previous results. Instead of computing a result as soon as it is
> possible (i.e., when a corresponding watermark was received), deferred
> computation adds a configurable amount of slack time in which late data is
> accepted before the result is compute. For example, instead of computing a
> tumbling window of 1 hour at each full hour, we can add a deferred
> computation interval of 15 minute to compute the result quarter past each
> full hour.
> This approach adds latency but can reduce the number of update esp. in use
> cases where the user cannot influence the generation of watermarks. It is
> also useful if the data is emitted to a system that cannot update result
> (files or Kafka). The deferred computation interval should be configured via
> the {{QueryConfig}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4183: [FLINK-6969][table]Add support for deferred computation f...
Github user wuchong commented on the issue: https://github.com/apache/flink/pull/4183 The proposal makes sense to me. BTW, the concepts of `earlyResultTimeOffset` and `updateRate` also works on window group aggregates, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6958) Async I/O timeout not work
[ https://issues.apache.org/jira/browse/FLINK-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074164#comment-16074164 ] Jark Wu commented on FLINK-6958: I agree with you. In case of regular shutdown, we should wait all remaining timers fire. > Async I/O timeout not work > -- > > Key: FLINK-6958 > URL: https://issues.apache.org/jira/browse/FLINK-6958 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.2.1 >Reporter: feng xiaojie >Assignee: Jark Wu > > when use Async I/O with UnorderedStreamElementQueue, the queue will always > full if you don't call the AsyncCollector.collect to ack them. > Timeout shall collect these entries when the timeout trigger,but it isn't work > I debug find, > when time out, it will call resultFuture.completeExceptionally(error); > but not call UnorderedStreamElementQueue.onCompleteHandler > it will cause that async i/o hang always -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6407) Upgrade AVRO dependency version to 1.8.x
[ https://issues.apache.org/jira/browse/FLINK-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingleizhang updated FLINK-6407: Fix Version/s: 1.4.0 > Upgrade AVRO dependency version to 1.8.x > > > Key: FLINK-6407 > URL: https://issues.apache.org/jira/browse/FLINK-6407 > Project: Flink > Issue Type: Wish > Components: Batch Connectors and Input/Output Formats >Affects Versions: 1.2.1 >Reporter: Miguel >Assignee: mingleizhang >Priority: Minor > Fix For: 1.4.0 > > > Avro 1.7.6 and 1.7.7 does not support Map that uses java Enum keys (it is > limited to String type keys). It was solved in Avro 1.8.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-6934) Consider moving LRUCache class
[ https://issues.apache.org/jira/browse/FLINK-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingleizhang updated FLINK-6934: Fix Version/s: 1.3.2 > Consider moving LRUCache class > -- > > Key: FLINK-6934 > URL: https://issues.apache.org/jira/browse/FLINK-6934 > Project: Flink > Issue Type: Improvement > Components: Local Runtime >Reporter: mingleizhang >Assignee: mingleizhang > Fix For: 1.3.2 > > > LRUCache class is not used any more. So, I would suggest remove it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6747) Table API / SQL Docs: Streaming Page
[
https://issues.apache.org/jira/browse/FLINK-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074083#comment-16074083
]
ASF GitHub Bot commented on FLINK-6747:
---
GitHub user fhueske opened a pull request:
https://github.com/apache/flink/pull/4256
[FLINK-6747] [docs] Add documentation for QueryConfig.
Needs to be merge to `master` and `release-1.3`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/flink tableStreamDocs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4256.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4256
commit c5775d998d89ad86895e339466aecd47e112c6dd
Author: Fabian Hueske
Date: 2017-07-04T21:28:22Z
[FLINK-6747] [docs] Add documentation for QueryConfig.
> Table API / SQL Docs: Streaming Page
>
>
> Key: FLINK-6747
> URL: https://issues.apache.org/jira/browse/FLINK-6747
> Project: Flink
> Issue Type: Task
> Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Fabian Hueske
>
> Extend {{./docs/dev/table/streaming.md}} page.
> Missing are sections about
> - Dynamic Tables
> - QueryConfiguration (state retention time)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4256: [FLINK-6747] [docs] Add documentation for QueryCon...
GitHub user fhueske opened a pull request: https://github.com/apache/flink/pull/4256 [FLINK-6747] [docs] Add documentation for QueryConfig. Needs to be merge to `master` and `release-1.3`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fhueske/flink tableStreamDocs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4256.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4256 commit c5775d998d89ad86895e339466aecd47e112c6dd Author: Fabian Hueske Date: 2017-07-04T21:28:22Z [FLINK-6747] [docs] Add documentation for QueryConfig. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()
[
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822346#comment-15822346
]
Ted Yu edited comment on FLINK-5486 at 7/4/17 9:03 PM:
---
Lock on State.bucketStates should be held in the following method:
{code}
private void handleRestoredBucketState(State restoredState) {
Preconditions.checkNotNull(restoredState);
for (BucketState bucketState : restoredState.bucketStates.values()) {
{code}
was (Author: yuzhih...@gmail.com):
Lock on State.bucketStates should be held in the following method:
{code}
private void handleRestoredBucketState(State restoredState) {
Preconditions.checkNotNull(restoredState);
for (BucketState bucketState : restoredState.bucketStates.values()) {
{code}
> Lack of synchronization in BucketingSink#handleRestoredBucketState()
>
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Ted Yu
>
> Here is related code:
> {code}
>
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
> synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
> }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside
> the synchronization block. Otherwise during the processing of
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be
> cleared.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-4534) Lack of synchronization in BucketingSink#restoreState()
[
https://issues.apache.org/jira/browse/FLINK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-4534:
--
Description:
Iteration over state.bucketStates is protected by synchronization in other
methods, except for the following in restoreState():
{code}
for (BucketState bucketState : state.bucketStates.values()) {
{code}
and following in close():
{code}
for (Map.Entry> entry :
state.bucketStates.entrySet()) {
closeCurrentPartFile(entry.getValue());
{code}
w.r.t. bucketState.pendingFilesPerCheckpoint , there is similar issue starting
line 752:
{code}
Set pastCheckpointIds =
bucketState.pendingFilesPerCheckpoint.keySet();
LOG.debug("Moving pending files to final location on restore.");
for (Long pastCheckpointId : pastCheckpointIds) {
{code}
was:
Iteration over state.bucketStates is protected by synchronization in other
methods, except for the following in restoreState():
{code}
for (BucketState bucketState : state.bucketStates.values()) {
{code}
and following in close():
{code}
for (Map.Entry> entry :
state.bucketStates.entrySet()) {
closeCurrentPartFile(entry.getValue());
{code}
w.r.t. bucketState.pendingFilesPerCheckpoint , there is similar issue starting
line 752:
{code}
Set pastCheckpointIds =
bucketState.pendingFilesPerCheckpoint.keySet();
LOG.debug("Moving pending files to final location on restore.");
for (Long pastCheckpointId : pastCheckpointIds) {
{code}
> Lack of synchronization in BucketingSink#restoreState()
> ---
>
> Key: FLINK-4534
> URL: https://issues.apache.org/jira/browse/FLINK-4534
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
>Reporter: Ted Yu
>
> Iteration over state.bucketStates is protected by synchronization in other
> methods, except for the following in restoreState():
> {code}
> for (BucketState bucketState : state.bucketStates.values()) {
> {code}
> and following in close():
> {code}
> for (Map.Entry> entry :
> state.bucketStates.entrySet()) {
> closeCurrentPartFile(entry.getValue());
> {code}
> w.r.t. bucketState.pendingFilesPerCheckpoint , there is similar issue
> starting line 752:
> {code}
> Set pastCheckpointIds =
> bucketState.pendingFilesPerCheckpoint.keySet();
> LOG.debug("Moving pending files to final location on restore.");
> for (Long pastCheckpointId : pastCheckpointIds) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6105) Properly handle InterruptedException in HadoopInputFormatBase
[
https://issues.apache.org/jira/browse/FLINK-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-6105:
--
Description:
When catching InterruptedException, we should throw InterruptedIOException
instead of IOException.
The following example is from HadoopInputFormatBase :
{code}
try {
splits = this.mapreduceInputFormat.getSplits(jobContext);
} catch (InterruptedException e) {
throw new IOException("Could not get Splits.", e);
}
{code}
There may be other places where IOE is thrown.
was:
When catching InterruptedException, we should throw InterruptedIOException
instead of IOException.
The following example is from HadoopInputFormatBase :
{code}
try {
splits = this.mapreduceInputFormat.getSplits(jobContext);
} catch (InterruptedException e) {
throw new IOException("Could not get Splits.", e);
}
{code}
There may be other places where IOE is thrown.
> Properly handle InterruptedException in HadoopInputFormatBase
> -
>
> Key: FLINK-6105
> URL: https://issues.apache.org/jira/browse/FLINK-6105
> Project: Flink
> Issue Type: Bug
> Components: DataStream API
>Reporter: Ted Yu
>
> When catching InterruptedException, we should throw InterruptedIOException
> instead of IOException.
> The following example is from HadoopInputFormatBase :
> {code}
> try {
> splits = this.mapreduceInputFormat.getSplits(jobContext);
> } catch (InterruptedException e) {
> throw new IOException("Could not get Splits.", e);
> }
> {code}
> There may be other places where IOE is thrown.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-5541) Missing null check for localJar in FlinkSubmitter#submitTopology()
[
https://issues.apache.org/jira/browse/FLINK-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-5541:
--
Description:
{code}
if (localJar == null) {
try {
for (final URL url : ((ContextEnvironment)
ExecutionEnvironment.getExecutionEnvironment())
.getJars()) {
// TODO verify that there is only one jar
localJar = new File(url.toURI()).getAbsolutePath();
}
} catch (final URISyntaxException e) {
// ignore
} catch (final ClassCastException e) {
// ignore
}
}
logger.info("Submitting topology " + name + " in distributed mode with
conf " + serConf);
client.submitTopologyWithOpts(name, localJar, topology);
{code}
Since the try block may encounter URISyntaxException / ClassCastException, we
should check that localJar is not null before calling submitTopologyWithOpts().
was:
{code}
if (localJar == null) {
try {
for (final URL url : ((ContextEnvironment)
ExecutionEnvironment.getExecutionEnvironment())
.getJars()) {
// TODO verify that there is only one jar
localJar = new File(url.toURI()).getAbsolutePath();
}
} catch (final URISyntaxException e) {
// ignore
} catch (final ClassCastException e) {
// ignore
}
}
logger.info("Submitting topology " + name + " in distributed mode with
conf " + serConf);
client.submitTopologyWithOpts(name, localJar, topology);
{code}
Since the try block may encounter URISyntaxException / ClassCastException, we
should check that localJar is not null before calling submitTopologyWithOpts().
> Missing null check for localJar in FlinkSubmitter#submitTopology()
> --
>
> Key: FLINK-5541
> URL: https://issues.apache.org/jira/browse/FLINK-5541
> Project: Flink
> Issue Type: Bug
> Components: Storm Compatibility
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> if (localJar == null) {
> try {
> for (final URL url : ((ContextEnvironment)
> ExecutionEnvironment.getExecutionEnvironment())
> .getJars()) {
> // TODO verify that there is only one jar
> localJar = new File(url.toURI()).getAbsolutePath();
> }
> } catch (final URISyntaxException e) {
> // ignore
> } catch (final ClassCastException e) {
> // ignore
> }
> }
> logger.info("Submitting topology " + name + " in distributed mode with
> conf " + serConf);
> client.submitTopologyWithOpts(name, localJar, topology);
> {code}
> Since the try block may encounter URISyntaxException / ClassCastException, we
> should check that localJar is not null before calling
> submitTopologyWithOpts().
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLINK-6359) Utilize Hierarchical Timing Wheels for performant timer
[
https://issues.apache.org/jira/browse/FLINK-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991975#comment-15991975
]
Ted Yu edited comment on FLINK-6359 at 7/4/17 9:01 PM:
---
Interesting observations, Ben.
was (Author: yuzhih...@gmail.com):
Interesting observations, Ben .
> Utilize Hierarchical Timing Wheels for performant timer
> ---
>
> Key: FLINK-6359
> URL: https://issues.apache.org/jira/browse/FLINK-6359
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, Local Runtime
>Reporter: Ted Yu
>
> In this thread on mailing list:
> http://search-hadoop.com/m/Flink/VkLeQPmRa31hd5cw
> Gyula Fóra mentioned that timer deletion becomes performance bottleneck due
> to the usage of priority queue.
> Benjamin has an implementation for Hierarchical Timing Wheels (Apache
> License) :
> https://github.com/ben-manes/caffeine/blob/master/caffeine/src/main/java/com/github/benmanes/caffeine/cache/TimerWheel.java
> {code}
> * A hierarchical timer wheel to add, remove, and fire expiration events in
> amortized O(1) time. The
> * expiration events are deferred until the timer is advanced, which is
> performed as part of the
> * cache's maintenance cycle.
> {code}
> We should consider porting the above over to facilitate performant timer.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLINK-6838) RescalingITCase fails in master branch
[
https://issues.apache.org/jira/browse/FLINK-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-6838:
--
Description:
{code}
Tests in error:
RescalingITCase.testSavepointRescalingInKeyedState[1] » JobExecution Job
execu...
RescalingITCase.testSavepointRescalingWithKeyedAndNonPartitionedState[1] »
JobExecution
{code}
Both failed with similar cause:
{code}
testSavepointRescalingInKeyedState[1](org.apache.flink.test.checkpointing.RescalingITCase)
Time elapsed: 4.813 sec <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.streaming.runtime.tasks.AsynchronousException:
java.lang.Exception: Could not materialize checkpoint 4 for operator Flat Map
-> Sink: Unnamed (1/2).
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:967)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not materialize checkpoint 4 for operator
Flat Map -> Sink: Unnamed (1/2).
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:967)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Cannot
register Closeable, registry is already closed. Closing argument.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot register Closeable, registry is already
closed. Closing argument.
at
org.apache.flink.util.AbstractCloseableRegistry.registerClosable(AbstractCloseableRegistry.java:66)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBFullSnapshotOperation.openCheckpointStream(RocksDBKeyedStateBackend.java:495)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$3.openIOHandle(RocksDBKeyedStateBackend.java:394)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$3.openIOHandle(RocksDBKeyedStateBackend.java:390)
at
org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
{code}
was:
{code}
Tests in error:
RescalingITCase.testSavepointRe
[jira] [Closed] (FLINK-4286) Have Kafka examples that use the Kafka 0.9 connector
[
https://issues.apache.org/jira/browse/FLINK-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-4286.
---
Resolution: Won't Fix
> Have Kafka examples that use the Kafka 0.9 connector
>
>
> Key: FLINK-4286
> URL: https://issues.apache.org/jira/browse/FLINK-4286
> Project: Flink
> Issue Type: Improvement
> Components: Examples
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Dmitrii Kniazev
>Priority: Minor
> Labels: starter
>
> The {{ReadFromKafka}} and {{WriteIntoKafka}} examples use the 0.8 connector,
> and the built example jar is named {{Kafka.jar}} under
> {{examples/streaming/}} in the distributed package.
> Since we have different connectors for different Kafka versions, it would be
> good to have examples for different versions, and package them as
> {{Kafka08.jar}} and {{Kafka09.jar}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074061#comment-16074061
]
ASF GitHub Bot commented on FLINK-7037:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4221
Thanks! That makes sense.
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4221: [FLINK-7037] Remove scala suffic from flink-examples modu...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4221 Thanks! That makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074055#comment-16074055 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 `mvn verify` now works as it should, execution tests and stuff. > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 `mvn verify` now works as it should, execution tests and stuff. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Comment Edited] (FLINK-4286) Have Kafka examples that use the Kafka 0.9 connector
[
https://issues.apache.org/jira/browse/FLINK-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073961#comment-16073961
]
Dmitrii Kniazev edited comment on FLINK-4286 at 7/4/17 8:04 PM:
Cool. I suppose, in this case we can close this issue with 'won't fix'
resolution.
was (Author: mylog00):
Cool. In this case we can close this issue with 'won't fix' resolution.
> Have Kafka examples that use the Kafka 0.9 connector
>
>
> Key: FLINK-4286
> URL: https://issues.apache.org/jira/browse/FLINK-4286
> Project: Flink
> Issue Type: Improvement
> Components: Examples
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Dmitrii Kniazev
>Priority: Minor
> Labels: starter
>
> The {{ReadFromKafka}} and {{WriteIntoKafka}} examples use the 0.8 connector,
> and the built example jar is named {{Kafka.jar}} under
> {{examples/streaming/}} in the distributed package.
> Since we have different connectors for different Kafka versions, it would be
> good to have examples for different versions, and package them as
> {{Kafka08.jar}} and {{Kafka09.jar}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @rmetzger I can fix the issue with verify by reverting the profile logic. Instead of activating the surefire plugin in each module, the plugin is enabled by default in the root pom but disabled in the profile of each module if the property is set --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074012#comment-16074012 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @rmetzger I can fix the issue with verify by reverting the profile logic. Instead of activating the surefire plugin in each module, the plugin is enabled by default in the root pom but disabled in the profile of each module if the property is set > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074010#comment-16074010 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @rmetzger Now i understand why the tests aren't run. I thought that I was activating the profiles with the properties in the root pom.xml, but this isn't the case. I was just setting the system properties in the travis scripts... back to the drawing board i guess... > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 @rmetzger Now i understand why the tests aren't run. I thought that I was activating the profiles with the properties in the root pom.xml, but this isn't the case. I was just setting the system properties in the travis scripts... back to the drawing board i guess... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073999#comment-16073999 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 WIth the existing surefire configuration we also cannot do this in 2 steps, one for compilation and one for tests. It is not possible to only run tests with any maven invocation that includes a life-cycle (i.e verify), which means that you have to recompile some modules twice which isn't an option because scala compilation is so utterly horrible. You _could_ theoretically cd into each project after compilation and directly invoke the surefire plugin, but this doesn't work as we have 2 executions (1 for unit tests, 1 for IT cases), and when invoking a plugin directly you can at most run 1 execution and it has to be named either "default-test" or "default-cli". This would be possible in maven 3.3.1, but we can't use that without messing with the shading, We could use maven 3.3.X if we split the repo and add a separate one for flink-dist, but that's not gonna happen anytime soon. We could directly invoke the surefire plugin if we merged our surefire executions, but then we're still left with the problem of having to keep the travis scripts in sync with the directory structure of the project. > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 WIth the existing surefire configuration we also cannot do this in 2 steps, one for compilation and one for tests. It is not possible to only run tests with any maven invocation that includes a life-cycle (i.e verify), which means that you have to recompile some modules twice which isn't an option because scala compilation is so utterly horrible. You _could_ theoretically cd into each project after compilation and directly invoke the surefire plugin, but this doesn't work as we have 2 executions (1 for unit tests, 1 for IT cases), and when invoking a plugin directly you can at most run 1 execution and it has to be named either "default-test" or "default-cli". This would be possible in maven 3.3.1, but we can't use that without messing with the shading, We could use maven 3.3.X if we split the repo and add a separate one for flink-dist, but that's not gonna happen anytime soon. We could directly invoke the surefire plugin if we merged our surefire executions, but then we're still left with the problem of having to keep the travis scripts in sync with the directory structure of the project. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073974#comment-16073974 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 another more sane option would be to add a new module for each group that has a dependency for each module we want tested. THEN we could use the `-am` option. > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 another more sane option would be to add a new module for each group that has a dependency for each module we want tested. THEN we could use the `-am` option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4286) Have Kafka examples that use the Kafka 0.9 connector
[
https://issues.apache.org/jira/browse/FLINK-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073961#comment-16073961
]
Dmitrii Kniazev commented on FLINK-4286:
Cool. In this case we can close this issue with 'won't fix' resolution.
> Have Kafka examples that use the Kafka 0.9 connector
>
>
> Key: FLINK-4286
> URL: https://issues.apache.org/jira/browse/FLINK-4286
> Project: Flink
> Issue Type: Improvement
> Components: Examples
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Dmitrii Kniazev
>Priority: Minor
> Labels: starter
>
> The {{ReadFromKafka}} and {{WriteIntoKafka}} examples use the 0.8 connector,
> and the built example jar is named {{Kafka.jar}} under
> {{examples/streaming/}} in the distributed package.
> Since we have different connectors for different Kafka versions, it would be
> good to have examples for different versions, and package them as
> {{Kafka08.jar}} and {{Kafka09.jar}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073956#comment-16073956 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 corrected my previous comment regarding the problems of `-am` and `-amd`. > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 corrected my previous comment regarding the problems of `-am` and `-amd`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4286) Have Kafka examples that use the Kafka 0.9 connector
[
https://issues.apache.org/jira/browse/FLINK-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073949#comment-16073949
]
ASF GitHub Bot commented on FLINK-4286:
---
Github user mylog00 closed the pull request at:
https://github.com/apache/flink/pull/4250
> Have Kafka examples that use the Kafka 0.9 connector
>
>
> Key: FLINK-4286
> URL: https://issues.apache.org/jira/browse/FLINK-4286
> Project: Flink
> Issue Type: Improvement
> Components: Examples
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Dmitrii Kniazev
>Priority: Minor
> Labels: starter
>
> The {{ReadFromKafka}} and {{WriteIntoKafka}} examples use the 0.8 connector,
> and the built example jar is named {{Kafka.jar}} under
> {{examples/streaming/}} in the distributed package.
> Since we have different connectors for different Kafka versions, it would be
> good to have examples for different versions, and package them as
> {{Kafka08.jar}} and {{Kafka09.jar}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4250: [FLINK-4286] Have Kafka examples that use the Kafka 0.9 c...
Github user mylog00 commented on the issue: https://github.com/apache/flink/pull/4250 wouldn't to fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4286) Have Kafka examples that use the Kafka 0.9 connector
[
https://issues.apache.org/jira/browse/FLINK-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073948#comment-16073948
]
ASF GitHub Bot commented on FLINK-4286:
---
Github user mylog00 commented on the issue:
https://github.com/apache/flink/pull/4250
wouldn't to fix
> Have Kafka examples that use the Kafka 0.9 connector
>
>
> Key: FLINK-4286
> URL: https://issues.apache.org/jira/browse/FLINK-4286
> Project: Flink
> Issue Type: Improvement
> Components: Examples
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Dmitrii Kniazev
>Priority: Minor
> Labels: starter
>
> The {{ReadFromKafka}} and {{WriteIntoKafka}} examples use the 0.8 connector,
> and the built example jar is named {{Kafka.jar}} under
> {{examples/streaming/}} in the distributed package.
> Since we have different connectors for different Kafka versions, it would be
> good to have examples for different versions, and package them as
> {{Kafka08.jar}} and {{Kafka09.jar}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4250: [FLINK-4286] Have Kafka examples that use the Kafk...
Github user mylog00 closed the pull request at: https://github.com/apache/flink/pull/4250 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[ https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073947#comment-16073947 ] ASF GitHub Bot commented on FLINK-7047: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 `mvn verify` _should_ run all tests since the default for all added properties is true. But yes, it currently doesn't. I don't quite understand this as the default for all groups is true (i.e. all test profiles should activate) and I'm not modifying the phases of these profiles. Maven's `-am` and `-amd` options don't work for us since these don't build dependencies marked as provided, i.e. most core flink dependencies. You can easily verify this yourself. Maven's `-pl` command is a bit brittle in general as we have to include every single module; we can't just point to flink-libraries and call it a day. This adds a maintainability problem; a new module will not be tested on travis if the contributor/reviewer isn't aware of the magic in the travis scripts. This is less likely with the approach in this PR as part of the magic is in the poms which are more likely to be seen by a developer. > Reorganize build profiles > - > > Key: FLINK-7047 > URL: https://issues.apache.org/jira/browse/FLINK-7047 > Project: Flink > Issue Type: Improvement > Components: Tests, Travis >Affects Versions: 1.4.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.4.0 > > > With the current build times once again hitting the timeout it is time to > revisit our approach. > The current approach of splitting all tests by name, while easy to maintain > or extend, has the big disadvantage that it's fairly binary in regards to the > timeout: either we're below the timeout and all builds pass, or we're above > and the entire merging process stalls. Furthermore, it requires all modules > to be compiled. > I propose a different approach by which we bundle several modules, only > execute the tests of these modules and skip the compilation of some modules > that are not required for these tests. > 5 groups are my current suggestion, which will result in 10 build profiles > total. > The groups are: > # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb > # *libraries* - flink-libraries and flink-storm > # *connectors* - flink-connectors, flink-connector-wikiedits, > flink-tweet-inputformat > # *tests* - flink-tests > # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples, > flink-dist > To not increase the total number of profiles to ridiculous numbers i also > propose to only test against 2 combinations of jdk+hadoop+scala: > # oraclejdk8 + hadoop 2.8.0 + scala 2.11 > # openjdk7 + hadoop 2.4.1 + scala 2.10 > My current estimate is that this will cause profiles to take at most 40 > minutes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4233 `mvn verify` _should_ run all tests since the default for all added properties is true. But yes, it currently doesn't. I don't quite understand this as the default for all groups is true (i.e. all test profiles should activate) and I'm not modifying the phases of these profiles. Maven's `-am` and `-amd` options don't work for us since these don't build dependencies marked as provided, i.e. most core flink dependencies. You can easily verify this yourself. Maven's `-pl` command is a bit brittle in general as we have to include every single module; we can't just point to flink-libraries and call it a day. This adds a maintainability problem; a new module will not be tested on travis if the contributor/reviewer isn't aware of the magic in the travis scripts. This is less likely with the approach in this PR as part of the magic is in the poms which are more likely to be seen by a developer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073933#comment-16073933
]
ASF GitHub Bot commented on FLINK-7037:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4221
@aljoscha Because it is the only parent module that has a scala suffix. All
other modules with a scala suffix don't have children.
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4221: [FLINK-7037] Remove scala suffic from flink-examples modu...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4221 @aljoscha Because it is the only parent module that has a scala suffix. All other modules with a scala suffix don't have children. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-6008.
---
Resolution: Fixed
Fix Version/s: 1.4.0
1.4: 28c3354e735e98657d8d90912234cf8e896e78d6
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.4.0
>
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * -promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}-
> * -extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]-
> * -remove {{NAME_ADDRESSABLE}} blobs after job/task termination-
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (FLINK-6043) Display time when exceptions/root cause of failure happened
[ https://issues.apache.org/jira/browse/FLINK-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-6043. --- Resolution: Fixed Fix Version/s: 1.4.0 1.4: 0f91083b2b497485b4375935ed7397583a03ca7c > Display time when exceptions/root cause of failure happened > --- > > Key: FLINK-6043 > URL: https://issues.apache.org/jira/browse/FLINK-6043 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Affects Versions: 1.3.0 >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Minor > Fix For: 1.4.0 > > > In order to better understand the behaviour of Flink jobs, it would be nice > to add timestamp information to exception causing the job to restart or to > fail. This information could then be displayed in the web UI making it easier > for the user to understand what happened when. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-7037.
---
Resolution: Fixed
1.4: bd858845a4ade89f78c06f72444563adfa5a974b
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (FLINK-6823) Activate checkstyle for runtime/broadcast
[ https://issues.apache.org/jira/browse/FLINK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-6823. --- Resolution: Fixed 1.4: 4d18afed80774c57766752f8a3cc45133a4a457a > Activate checkstyle for runtime/broadcast > - > > Key: FLINK-6823 > URL: https://issues.apache.org/jira/browse/FLINK-6823 > Project: Flink > Issue Type: Improvement > Components: Local Runtime >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6043) Display time when exceptions/root cause of failure happened
[ https://issues.apache.org/jira/browse/FLINK-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073919#comment-16073919 ] ASF GitHub Bot commented on FLINK-6043: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3583 > Display time when exceptions/root cause of failure happened > --- > > Key: FLINK-6043 > URL: https://issues.apache.org/jira/browse/FLINK-6043 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Affects Versions: 1.3.0 >Reporter: Till Rohrmann >Assignee: Chesnay Schepler >Priority: Minor > > In order to better understand the behaviour of Flink jobs, it would be nice > to add timestamp information to exception causing the job to restart or to > fail. This information could then be displayed in the web UI making it easier > for the user to understand what happened when. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6823) Activate checkstyle for runtime/broadcast
[ https://issues.apache.org/jira/browse/FLINK-6823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073921#comment-16073921 ] ASF GitHub Bot commented on FLINK-6823: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4068 > Activate checkstyle for runtime/broadcast > - > > Key: FLINK-6823 > URL: https://issues.apache.org/jira/browse/FLINK-6823 > Project: Flink > Issue Type: Improvement > Components: Local Runtime >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Trivial > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073922#comment-16073922
]
ASF GitHub Bot commented on FLINK-7037:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4221
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6008) collection of BlobServer improvements
[
https://issues.apache.org/jira/browse/FLINK-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073920#comment-16073920
]
ASF GitHub Bot commented on FLINK-6008:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4146
> collection of BlobServer improvements
> -
>
> Key: FLINK-6008
> URL: https://issues.apache.org/jira/browse/FLINK-6008
> Project: Flink
> Issue Type: Improvement
> Components: Network
>Affects Versions: 1.3.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> The following things should be improved around the BlobServer/BlobCache:
> * update config uptions with non-deprecated ones, e.g.
> {{high-availability.cluster-id}} and {{high-availability.storageDir}}
> * -promote {{BlobStore#deleteAll(JobID)}} to the {{BlobService}}-
> * -extend the {{BlobService}} to work with {{NAME_ADDRESSABLE}} blobs
> (prepares FLINK-4399]-
> * -remove {{NAME_ADDRESSABLE}} blobs after job/task termination-
> * do not fail the {{BlobServer}} when a delete operation fails
> * code style, like using {{Preconditions.checkArgument}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4146: [FLINK-6008][blob] collection of BlobServer improv...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4146 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4068: [FLINK-6823] Activate checkstyle for runtime/broad...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4068 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #4221: [FLINK-7037] Remove scala suffic from flink-exampl...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4221 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3583: [FLINK-6043] [web] Display exception timestamp
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3583 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL
[
https://issues.apache.org/jira/browse/FLINK-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073879#comment-16073879
]
ASF GitHub Bot commented on FLINK-6925:
---
Github user sunjincheng121 commented on the issue:
https://github.com/apache/flink/pull/4138
Thanks @wuchong ! From my point of view +1 to merge.
@twalthr do you have any other comments? :)
> Add CONCAT/CONCAT_WS supported in SQL
> -
>
> Key: FLINK-6925
> URL: https://issues.apache.org/jira/browse/FLINK-6925
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> CONCAT(str1,str2,...)Returns the string that results from concatenating the
> arguments. May have one or more arguments. If all arguments are nonbinary
> strings, the result is a nonbinary string. If the arguments include any
> binary strings, the result is a binary string. A numeric argument is
> converted to its equivalent nonbinary string form.
> CONCAT() returns NULL if any argument is NULL.
> * Syntax:
> CONCAT(str1,str2,...)
> * Arguments
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT('F', 'lin', 'k') -> 'Flink'
> CONCAT('M', NULL, 'L') -> NULL
> CONCAT(14.3) -> '14.3'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat]
> CONCAT_WS() stands for Concatenate With Separator and is a special form of
> CONCAT(). The first argument is the separator for the rest of the arguments.
> The separator is added between the strings to be concatenated. The separator
> can be a string, as can the rest of the arguments. If the separator is NULL,
> the result is NULL.
> * Syntax:
> CONCAT_WS(separator,str1,str2,...)
> * Arguments
> ** separator -
> ** str1,str2,... -
> * Return Types
> string
> * Example:
> CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second
> name,Last Name'
> * See more:
> ** [MySQL|
> https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4138: [FLINK-6925][table]Add CONCAT/CONCAT_WS supported in SQL
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/4138 Thanks @wuchong ! From my point of view +1 to merge. @twalthr do you have any other comments? :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
[
https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073873#comment-16073873
]
ASF GitHub Bot commented on FLINK-6649:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125504798
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/GroupAggProcessFunctionWithUpdateInterval.scala
---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.runtime.aggregate
+
+import java.lang.{Long => JLong}
+
+import org.apache.flink.api.common.ExecutionConfig
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeutils.TypeSerializer
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.{StreamQueryConfig, Types}
+import org.apache.flink.table.codegen.{Compiler,
GeneratedAggregationsFunction}
+import org.apache.flink.table.runtime.types.CRow
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.{Logger, LoggerFactory}
+
+/**
+ * Aggregate Function used for the groupby (without window) aggregate
+ * with update interval config.
+ *
+ * @param genAggregations Generated aggregate helper function
+ * @param aggregationStateType The row type info of aggregation
+ * @param outputRowType The row type info of output.
+ */
+class GroupAggProcessFunctionWithUpdateInterval(
--- End diff --
So, I think `NonWindowedAgg` and `IntervalUpdateNonWindowedAgg` is better.
And I am fine if you want add `ProcessFunction` at the end. Because many class
name end with xxxFunction.
> Improve Non-window group aggregate with configurable `earlyFire`.
> -
>
> Key: FLINK-6649
> URL: https://issues.apache.org/jira/browse/FLINK-6649
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently, Non-windowed group aggregate is earlyFiring at count(1), that is
> every row will emit a aggregate result. But some times user want config count
> number (`early firing with count[N]`) , to reduce the downstream pressure.
> This JIRA. will enable the config of e`earlyFiring` for Non-windowed group
> aggregate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4157: [FLINK-6649][table]Improve Non-window group aggreg...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125504798
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/GroupAggProcessFunctionWithUpdateInterval.scala
---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.runtime.aggregate
+
+import java.lang.{Long => JLong}
+
+import org.apache.flink.api.common.ExecutionConfig
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeutils.TypeSerializer
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.{StreamQueryConfig, Types}
+import org.apache.flink.table.codegen.{Compiler,
GeneratedAggregationsFunction}
+import org.apache.flink.table.runtime.types.CRow
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.{Logger, LoggerFactory}
+
+/**
+ * Aggregate Function used for the groupby (without window) aggregate
+ * with update interval config.
+ *
+ * @param genAggregations Generated aggregate helper function
+ * @param aggregationStateType The row type info of aggregation
+ * @param outputRowType The row type info of output.
+ */
+class GroupAggProcessFunctionWithUpdateInterval(
--- End diff --
So, I think `NonWindowedAgg` and `IntervalUpdateNonWindowedAgg` is better.
And I am fine if you want add `ProcessFunction` at the end. Because many class
name end with xxxFunction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Updated] (FLINK-7103) Implement skeletal structure of dispatcher component
[
https://issues.apache.org/jira/browse/FLINK-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-7103:
-
Summary: Implement skeletal structure of dispatcher component (was:
Implement dispatcher component)
> Implement skeletal structure of dispatcher component
>
>
> Key: FLINK-7103
> URL: https://issues.apache.org/jira/browse/FLINK-7103
> Project: Flink
> Issue Type: Sub-task
> Components: Cluster Management
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
>
> Implement the skeletal structure of the {{Dispatcher}} component. The initial
> functionality will support job submissions and listing of jobs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
[
https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073867#comment-16073867
]
ASF GitHub Bot commented on FLINK-6649:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125504087
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/GroupAggProcessFunctionWithUpdateInterval.scala
---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.runtime.aggregate
+
+import java.lang.{Long => JLong}
+
+import org.apache.flink.api.common.ExecutionConfig
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeutils.TypeSerializer
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.{StreamQueryConfig, Types}
+import org.apache.flink.table.codegen.{Compiler,
GeneratedAggregationsFunction}
+import org.apache.flink.table.runtime.types.CRow
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.{Logger, LoggerFactory}
+
+/**
+ * Aggregate Function used for the groupby (without window) aggregate
+ * with update interval config.
+ *
+ * @param genAggregations Generated aggregate helper function
+ * @param aggregationStateType The row type info of aggregation
+ * @param outputRowType The row type info of output.
+ */
+class GroupAggProcessFunctionWithUpdateInterval(
--- End diff --
Yes, that's true. :)
> Improve Non-window group aggregate with configurable `earlyFire`.
> -
>
> Key: FLINK-6649
> URL: https://issues.apache.org/jira/browse/FLINK-6649
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently, Non-windowed group aggregate is earlyFiring at count(1), that is
> every row will emit a aggregate result. But some times user want config count
> number (`early firing with count[N]`) , to reduce the downstream pressure.
> This JIRA. will enable the config of e`earlyFiring` for Non-windowed group
> aggregate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4157: [FLINK-6649][table]Improve Non-window group aggreg...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125504087
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/runtime/aggregate/GroupAggProcessFunctionWithUpdateInterval.scala
---
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.runtime.aggregate
+
+import java.lang.{Long => JLong}
+
+import org.apache.flink.api.common.ExecutionConfig
+import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
+import org.apache.flink.api.common.typeutils.TypeSerializer
+import org.apache.flink.api.java.typeutils.RowTypeInfo
+import org.apache.flink.configuration.Configuration
+import org.apache.flink.streaming.api.functions.ProcessFunction
+import org.apache.flink.table.api.{StreamQueryConfig, Types}
+import org.apache.flink.table.codegen.{Compiler,
GeneratedAggregationsFunction}
+import org.apache.flink.table.runtime.types.CRow
+import org.apache.flink.types.Row
+import org.apache.flink.util.Collector
+import org.slf4j.{Logger, LoggerFactory}
+
+/**
+ * Aggregate Function used for the groupby (without window) aggregate
+ * with update interval config.
+ *
+ * @param genAggregations Generated aggregate helper function
+ * @param aggregationStateType The row type info of aggregation
+ * @param outputRowType The row type info of output.
+ */
+class GroupAggProcessFunctionWithUpdateInterval(
--- End diff --
Yes, that's true. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (FLINK-7103) Implement dispatcher component
Till Rohrmann created FLINK-7103:
Summary: Implement dispatcher component
Key: FLINK-7103
URL: https://issues.apache.org/jira/browse/FLINK-7103
Project: Flink
Issue Type: Sub-task
Components: Cluster Management
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Implement the skeletal structure of the {{Dispatcher}} component. The initial
functionality will support job submissions and listing of jobs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4252: [FLINK-7074] [tm] Add entry point for the TaskMana...
Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/4252#discussion_r125503520 --- Diff: flink-runtime/pom.xml --- @@ -117,6 +117,16 @@ under the License. + org.slf4j + slf4j-log4j12 --- End diff -- The problem I had was that I couldn't start the `TaskManagerRunner#main` from within Intellij, because the class `org.apache.log4j.Level` could not be found. If I'm not mistaken, then we filter out the log4j dependencies when building the dist packages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-7074) Add entry point for TaskManagerRunner
[
https://issues.apache.org/jira/browse/FLINK-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073863#comment-16073863
]
ASF GitHub Bot commented on FLINK-7074:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4252#discussion_r125503520
--- Diff: flink-runtime/pom.xml ---
@@ -117,6 +117,16 @@ under the License.
+ org.slf4j
+ slf4j-log4j12
--- End diff --
The problem I had was that I couldn't start the `TaskManagerRunner#main`
from within Intellij, because the class `org.apache.log4j.Level` could not be
found. If I'm not mistaken, then we filter out the log4j dependencies when
building the dist packages.
> Add entry point for TaskManagerRunner
> -
>
> Key: FLINK-7074
> URL: https://issues.apache.org/jira/browse/FLINK-7074
> Project: Flink
> Issue Type: Sub-task
> Components: Distributed Coordination
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
>
> Adding a {{main}} method to the {{TaskManagerRunner}} will allow us to use
> this class as the entry point for the code running inside a Mesos task or in
> a Yarn container. In this method, we have to setup the components of the
> {{TaskManager}}. It might be possible to share common setup logic between the
> TaskManager's entry point and the cluster entry point.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7102) improve ClassLoaderITCase
[
https://issues.apache.org/jira/browse/FLINK-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073859#comment-16073859
]
ASF GitHub Bot commented on FLINK-7102:
---
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/4255
[FLINK-7102] improve ClassLoaderITCase
* ClassLoaderITCase unnecessarily runs multiple tests in a single test case
* ClassLoaderITCase#testDisposeSavepointWithCustomKvState() does not cancel
its
job (thus the order of execution of test cases defines the outcome)
* ClassLoaderITCase uses e.getCause().getCause() which may cause
{{NullPointerException}}s hiding the original error
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-7102
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4255
commit 87f28e5d972d33c82f44c05ab62743cb614e5239
Author: Nico Kruber
Date: 2017-07-04T11:49:55Z
[FLINK-7102] improve ClassLoaderITCase
* ClassLoaderITCase unnecessarily runs multiple tests in a single test case
* ClassLoaderITCase#testDisposeSavepointWithCustomKvState() does not cancel
its
job (thus the order of execution of test cases defines the outcome)
* ClassLoaderITCase uses e.getCause().getCause() which may cause
{{NullPointerException}}s hiding the original error
> improve ClassLoaderITCase
> -
>
> Key: FLINK-7102
> URL: https://issues.apache.org/jira/browse/FLINK-7102
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> {{ClassLoaderITCase}}...
> * unnecessarily runs multiple tests in a single test case
> * {{#testDisposeSavepointWithCustomKvState()}} does not cancel its job (thus
> the order of execution of test cases defines the outcome)
> * uses {{e.getCause().getCause()}} which may cause {{NullPointerException}}s
> hiding the original error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4255: [FLINK-7102] improve ClassLoaderITCase
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/4255
[FLINK-7102] improve ClassLoaderITCase
* ClassLoaderITCase unnecessarily runs multiple tests in a single test case
* ClassLoaderITCase#testDisposeSavepointWithCustomKvState() does not cancel
its
job (thus the order of execution of test cases defines the outcome)
* ClassLoaderITCase uses e.getCause().getCause() which may cause
{{NullPointerException}}s hiding the original error
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-7102
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4255.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4255
commit 87f28e5d972d33c82f44c05ab62743cb614e5239
Author: Nico Kruber
Date: 2017-07-04T11:49:55Z
[FLINK-7102] improve ClassLoaderITCase
* ClassLoaderITCase unnecessarily runs multiple tests in a single test case
* ClassLoaderITCase#testDisposeSavepointWithCustomKvState() does not cancel
its
job (thus the order of execution of test cases defines the outcome)
* ClassLoaderITCase uses e.getCause().getCause() which may cause
{{NullPointerException}}s hiding the original error
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-7097) Start TaskExecutor via start-up script
[
https://issues.apache.org/jira/browse/FLINK-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073856#comment-16073856
]
ASF GitHub Bot commented on FLINK-7097:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4253#discussion_r125502076
--- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
@@ -73,11 +78,11 @@ if [[ $STARTSTOP == "start" ]] || [[ $STARTSTOP ==
"start-foreground" ]]; then
fi
if [[ $STARTSTOP == "start-foreground" ]]; then
-exec "${FLINK_BIN_DIR}"/flink-console.sh taskmanager "${args[@]}"
+exec "${FLINK_BIN_DIR}"/flink-console.sh TYPE "${args[@]}"
--- End diff --
Yup, thanks for catching it :-)
> Start TaskExecutor via start-up script
> --
>
> Key: FLINK-7097
> URL: https://issues.apache.org/jira/browse/FLINK-7097
> Project: Flink
> Issue Type: Sub-task
> Components: Startup Shell Scripts
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Labels: flip-6
>
> Start the {{TaskExecutor}} via the start-up scripts.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4253: [FLINK-7097] [scripts] Enable Flip-6 TaskExecutor ...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4253#discussion_r125502076
--- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
@@ -73,11 +78,11 @@ if [[ $STARTSTOP == "start" ]] || [[ $STARTSTOP ==
"start-foreground" ]]; then
fi
if [[ $STARTSTOP == "start-foreground" ]]; then
-exec "${FLINK_BIN_DIR}"/flink-console.sh taskmanager "${args[@]}"
+exec "${FLINK_BIN_DIR}"/flink-console.sh TYPE "${args[@]}"
--- End diff --
Yup, thanks for catching it :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-7037) Remove scala suffix from flink-examples module
[
https://issues.apache.org/jira/browse/FLINK-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073855#comment-16073855
]
ASF GitHub Bot commented on FLINK-7037:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4221
@zentol why was this only a problem for the example module?
> Remove scala suffix from flink-examples module
> --
>
> Key: FLINK-7037
> URL: https://issues.apache.org/jira/browse/FLINK-7037
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> Exceptions such as this
> {code}
> [ERROR] Failed to execute goal on project flink-tests_2.11: Could not resolve
> dependencies for project org.apache.flink:flink-tests_2.11:jar:1.4-SNAPSHOT:
> Failed to collect dependencies at
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failed to read
> artifact descriptor for
> org.apache.flink:flink-examples-batch_2.11:jar:1.4-SNAPSHOT: Failure to find
> org.apache.flink:flink-examples_${scala.binary.version}:pom:1.4-SNAPSHOT in
> https://oss.sonatype.org/content/repositories/snapshots/ was cached in the
> local repository, resolution will not be reattempted until the update
> interval of sonatype-snapshots has elapsed or updates are forced -> [Help 1]
> {code}
> are caused by modules that have a parent module which have a scala suffix.
> The scala version is not properly resolved for parents. In retrospect this is
> rather obvious, as the scala version is defined in the parent; in order to
> resolve the scala version one would ahve to know who the parent is, but you
> need the scala version for that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
[
https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073854#comment-16073854
]
ASF GitHub Bot commented on FLINK-6649:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125501792
--- Diff:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/harness/NonWindowHarnessTest.scala
---
@@ -80,18 +80,79 @@ class NonWindowHarnessTest extends HarnessTestBase {
val expectedOutput = new ConcurrentLinkedQueue[Object]()
-expectedOutput.add(new StreamRecord(CRow(Row.of(1L: JLong, 1: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(2L: JLong, 1: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(3L: JLong, 3: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(4L: JLong, 6: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(5L: JLong, 10: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(6L: JLong, 3: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(7L: JLong, 5: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(8L: JLong, 11: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(9L: JLong, 18: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(10L: JLong, 3: JInt),
true), 1))
-
-verify(expectedOutput, result, new RowResultSortComparator(6))
+expectedOutput.add(new StreamRecord(CRow(Row.of(1L: JLong, "aaa", 1:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(2L: JLong, "bbb", 1:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(3L: JLong, "aaa", 3:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(4L: JLong, "aaa", 6:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(5L: JLong, "aaa", 10:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(6L: JLong, "bbb", 3:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(7L: JLong, "aaa", 5:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(8L: JLong, "aaa", 11:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(9L: JLong, "aaa", 18:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(10L: JLong, "bbb", 3:
JInt), true), 1))
+
+verifySorted(expectedOutput, result, new RowResultSortComparator)
+
+testHarness.close()
+ }
+
+ @Test
+ def testProcTimeNonWindowWithUpdateInterval(): Unit = {
+
+val processFunction = new KeyedProcessOperator[String, CRow, CRow](
+ new GroupAggProcessFunctionWithUpdateInterval(
+genSumAggFunction,
+sumAggregationStateType,
+sumAggregationRowType,
+false,
+queryConfig
+.withIdleStateRetentionTime(Time.seconds(4), Time.seconds(5))
+.withUnboundedAggregateUpdateInterval(Time.seconds(1
+
+val testHarness =
+ createHarnessTester(
+processFunction,
+new TupleRowKeySelector[String](2),
+BasicTypeInfo.STRING_TYPE_INFO)
+
+testHarness.open()
+
+testHarness.setProcessingTime(1)
+
+testHarness.processElement(new StreamRecord(CRow(Row.of(1L: JLong, 1:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(2L: JLong, 1:
JInt, "bbb"), true), 1))
+testHarness.setProcessingTime(1000)
+testHarness.processElement(new StreamRecord(CRow(Row.of(3L: JLong, 2:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(4L: JLong, 3:
JInt, "aaa"), true), 1))
+
+testHarness.setProcessingTime(1002)
+testHarness.processElement(new StreamRecord(CRow(Row.of(5L: JLong, 4:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(6L: JLong, 2:
JInt, "bbb"), true), 1))
+
+testHarness.setProcessingTime(4003)
+testHarness.processElement(new StreamRecord(CRow(Row.of(7L: JLong, 5:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(8L: JLong, 6:
JInt, "aaa"), true), 1))
+
+// clear all states
+testHarness.setProcessingTime(10003)
+testHarness.processElement(new StreamRecord(CRow(Row.of(9L: JLong, 7:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(10L: JLong, 3:
JInt, "bbb"), true), 1))
+
+testHarness.setProcessingTime(12003)
+
+val result = testHarness.getOutput
+
+val expectedOutput = new ConcurrentLinkedQueue
[GitHub] flink pull request #4157: [FLINK-6649][table]Improve Non-window group aggreg...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125501792
--- Diff:
flink-libraries/flink-table/src/test/scala/org/apache/flink/table/runtime/harness/NonWindowHarnessTest.scala
---
@@ -80,18 +80,79 @@ class NonWindowHarnessTest extends HarnessTestBase {
val expectedOutput = new ConcurrentLinkedQueue[Object]()
-expectedOutput.add(new StreamRecord(CRow(Row.of(1L: JLong, 1: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(2L: JLong, 1: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(3L: JLong, 3: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(4L: JLong, 6: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(5L: JLong, 10: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(6L: JLong, 3: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(7L: JLong, 5: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(8L: JLong, 11: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(9L: JLong, 18: JInt),
true), 1))
-expectedOutput.add(new StreamRecord(CRow(Row.of(10L: JLong, 3: JInt),
true), 1))
-
-verify(expectedOutput, result, new RowResultSortComparator(6))
+expectedOutput.add(new StreamRecord(CRow(Row.of(1L: JLong, "aaa", 1:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(2L: JLong, "bbb", 1:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(3L: JLong, "aaa", 3:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(4L: JLong, "aaa", 6:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(5L: JLong, "aaa", 10:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(6L: JLong, "bbb", 3:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(7L: JLong, "aaa", 5:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(8L: JLong, "aaa", 11:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(9L: JLong, "aaa", 18:
JInt), true), 1))
+expectedOutput.add(new StreamRecord(CRow(Row.of(10L: JLong, "bbb", 3:
JInt), true), 1))
+
+verifySorted(expectedOutput, result, new RowResultSortComparator)
+
+testHarness.close()
+ }
+
+ @Test
+ def testProcTimeNonWindowWithUpdateInterval(): Unit = {
+
+val processFunction = new KeyedProcessOperator[String, CRow, CRow](
+ new GroupAggProcessFunctionWithUpdateInterval(
+genSumAggFunction,
+sumAggregationStateType,
+sumAggregationRowType,
+false,
+queryConfig
+.withIdleStateRetentionTime(Time.seconds(4), Time.seconds(5))
+.withUnboundedAggregateUpdateInterval(Time.seconds(1
+
+val testHarness =
+ createHarnessTester(
+processFunction,
+new TupleRowKeySelector[String](2),
+BasicTypeInfo.STRING_TYPE_INFO)
+
+testHarness.open()
+
+testHarness.setProcessingTime(1)
+
+testHarness.processElement(new StreamRecord(CRow(Row.of(1L: JLong, 1:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(2L: JLong, 1:
JInt, "bbb"), true), 1))
+testHarness.setProcessingTime(1000)
+testHarness.processElement(new StreamRecord(CRow(Row.of(3L: JLong, 2:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(4L: JLong, 3:
JInt, "aaa"), true), 1))
+
+testHarness.setProcessingTime(1002)
+testHarness.processElement(new StreamRecord(CRow(Row.of(5L: JLong, 4:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(6L: JLong, 2:
JInt, "bbb"), true), 1))
+
+testHarness.setProcessingTime(4003)
+testHarness.processElement(new StreamRecord(CRow(Row.of(7L: JLong, 5:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(8L: JLong, 6:
JInt, "aaa"), true), 1))
+
+// clear all states
+testHarness.setProcessingTime(10003)
+testHarness.processElement(new StreamRecord(CRow(Row.of(9L: JLong, 7:
JInt, "aaa"), true), 1))
+testHarness.processElement(new StreamRecord(CRow(Row.of(10L: JLong, 3:
JInt, "bbb"), true), 1))
+
+testHarness.setProcessingTime(12003)
+
+val result = testHarness.getOutput
+
+val expectedOutput = new ConcurrentLinkedQueue[Object]()
+
+expectedOutput.add(new StreamRecord(CRow(Row.of(1L: JLong, "aaa", 6:
JInt), true), 1001))
--- End diff --
Yes, Agree with you. :)
---
If your project is set up for it, you can reply to this email and have your
repl
[GitHub] flink issue #4221: [FLINK-7037] Remove scala suffic from flink-examples modu...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4221 @zentol why was this only a problem for the example module? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-7102) improve ClassLoaderITCase
Nico Kruber created FLINK-7102:
--
Summary: improve ClassLoaderITCase
Key: FLINK-7102
URL: https://issues.apache.org/jira/browse/FLINK-7102
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.4.0
Reporter: Nico Kruber
Assignee: Nico Kruber
{{ClassLoaderITCase}}...
* unnecessarily runs multiple tests in a single test case
* {{#testDisposeSavepointWithCustomKvState()}} does not cancel its job (thus
the order of execution of test cases defines the outcome)
* uses {{e.getCause().getCause()}} which may cause {{NullPointerException}}s
hiding the original error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLINK-7082) Flip-6: Generic entry point for Flink sessions
[
https://issues.apache.org/jira/browse/FLINK-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-7082:
Assignee: Till Rohrmann
> Flip-6: Generic entry point for Flink sessions
> --
>
> Key: FLINK-7082
> URL: https://issues.apache.org/jira/browse/FLINK-7082
> Project: Flink
> Issue Type: Bug
> Components: Cluster Management
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
>
> Implement a generic entry point for Flink sessions. This
> {{ClusterEntryPoint}} has to start a {{ResourceManager}}, the {{Dispatcher}}
> component and the cluster's RESTful endpoint. This class could serve as the
> basis for a {{Mesos-}} and {{YarnEntryPoint}} to run Flink sessions.
> Maybe we can use a common base for the session and the per-job mode. The
> session has to start a dispatcher component and the per-job mode retrieves
> the {{JobGraph}} and directly starts a {{JobManager}} with this job.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4233: [FLINK-7047] [travis] Reorganize build profiles
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4233
@aljoscha this may be caused by publishing pom's with unresolved
`${scala.binary.version}`. I am planning to file a ticket after #4221 is
merged. The StackOverflow link looks promising.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-7047) Reorganize build profiles
[
https://issues.apache.org/jira/browse/FLINK-7047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073848#comment-16073848
]
ASF GitHub Bot commented on FLINK-7047:
---
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4233
@aljoscha this may be caused by publishing pom's with unresolved
`${scala.binary.version}`. I am planning to file a ticket after #4221 is
merged. The StackOverflow link looks promising.
> Reorganize build profiles
> -
>
> Key: FLINK-7047
> URL: https://issues.apache.org/jira/browse/FLINK-7047
> Project: Flink
> Issue Type: Improvement
> Components: Tests, Travis
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>
> With the current build times once again hitting the timeout it is time to
> revisit our approach.
> The current approach of splitting all tests by name, while easy to maintain
> or extend, has the big disadvantage that it's fairly binary in regards to the
> timeout: either we're below the timeout and all builds pass, or we're above
> and the entire merging process stalls. Furthermore, it requires all modules
> to be compiled.
> I propose a different approach by which we bundle several modules, only
> execute the tests of these modules and skip the compilation of some modules
> that are not required for these tests.
> 5 groups are my current suggestion, which will result in 10 build profiles
> total.
> The groups are:
> # *core* - core flink modules like core,runtime,streaming-java,metrics,rocksdb
> # *libraries* - flink-libraries and flink-storm
> # *connectors* - flink-connectors, flink-connector-wikiedits,
> flink-tweet-inputformat
> # *tests* - flink-tests
> # *misc* - flink-yarn, fink-yarn-tests, flink-mesos, flink-examples,
> flink-dist
> To not increase the total number of profiles to ridiculous numbers i also
> propose to only test against 2 combinations of jdk+hadoop+scala:
> # oraclejdk8 + hadoop 2.8.0 + scala 2.11
> # openjdk7 + hadoop 2.4.1 + scala 2.10
> My current estimate is that this will cause profiles to take at most 40
> minutes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.
[
https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073847#comment-16073847
]
ASF GitHub Bot commented on FLINK-6649:
---
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125500774
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/queryConfig.scala
---
@@ -37,6 +37,13 @@ class BatchQueryConfig private[table] extends QueryConfig
class StreamQueryConfig private[table] extends QueryConfig {
/**
+* The non-windowed groupby aggregate update the calculation result
according a configuration of
+* time interval. By default non-windowed groupby aggregate will update
calculation result each
+* row.
+*/
+ private var unboundedAggregateUpdateInterval: Time = Time.milliseconds(0)
--- End diff --
I think we need to achieve separately, i.e., proc-time non-windowed
aggregations using `process-time timer.` and event-time non-windowed
aggregations using event-time timer. `updateInterval` also make sense, because
`updateInterval` is relative to the last emit time. i.e., emitTime =
current(Proc/Row)Time+updateInterval. (Calculate emitTime when the data
arrives).
> Improve Non-window group aggregate with configurable `earlyFire`.
> -
>
> Key: FLINK-6649
> URL: https://issues.apache.org/jira/browse/FLINK-6649
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently, Non-windowed group aggregate is earlyFiring at count(1), that is
> every row will emit a aggregate result. But some times user want config count
> number (`early firing with count[N]`) , to reduce the downstream pressure.
> This JIRA. will enable the config of e`earlyFiring` for Non-windowed group
> aggregate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4157: [FLINK-6649][table]Improve Non-window group aggreg...
Github user sunjincheng121 commented on a diff in the pull request:
https://github.com/apache/flink/pull/4157#discussion_r125500774
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/table/api/queryConfig.scala
---
@@ -37,6 +37,13 @@ class BatchQueryConfig private[table] extends QueryConfig
class StreamQueryConfig private[table] extends QueryConfig {
/**
+* The non-windowed groupby aggregate update the calculation result
according a configuration of
+* time interval. By default non-windowed groupby aggregate will update
calculation result each
+* row.
+*/
+ private var unboundedAggregateUpdateInterval: Time = Time.milliseconds(0)
--- End diff --
I think we need to achieve separately, i.e., proc-time non-windowed
aggregations using `process-time timer.` and event-time non-windowed
aggregations using event-time timer. `updateInterval` also make sense, because
`updateInterval` is relative to the last emit time. i.e., emitTime =
current(Proc/Row)Time+updateInterval. (Calculate emitTime when the data
arrives).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
