[jira] [Commented] (FLINK-6868) Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra Connector`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042304#comment-16042304
 ] 

ASF GitHub Bot commented on FLINK-6868:
---

GitHub user asdf2014 opened a pull request:

https://github.com/apache/flink/pull/4087

[FLINK-6868][build] Using `scala.binary.version` for 
`flink-streaming-scala` in `Cassandra Connector`

Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
Connector`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/asdf2014/flink cassandra_scala

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4087.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4087


commit 61f31e83407aa458161109f9576daf4cfb86a228
Author: asdf2014 <1571805...@qq.com>
Date:   2017-06-08T06:55:25Z

[FLINK-6868][build] Using `scala.binary.version` for 
`flink-streaming-scala` in `Cassandra Connector`




> Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
> Connector`
> -
>
> Key: FLINK-6868
> URL: https://issues.apache.org/jira/browse/FLINK-6868
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 1.3.0
>Reporter: Benedict Jin
>  Labels: improvement
> Fix For: 1.4.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Shoud using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
> Connector`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4087: [FLINK-6868][build] Using `scala.binary.version` f...

[asdf2014]

GitHub user asdf2014 opened a pull request:

https://github.com/apache/flink/pull/4087

[FLINK-6868][build] Using `scala.binary.version` for 
`flink-streaming-scala` in `Cassandra Connector`

Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
Connector`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/asdf2014/flink cassandra_scala

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4087.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4087


commit 61f31e83407aa458161109f9576daf4cfb86a228
Author: asdf2014 <1571805...@qq.com>
Date:   2017-06-08T06:55:25Z

[FLINK-6868][build] Using `scala.binary.version` for 
`flink-streaming-scala` in `Cassandra Connector`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-6868) Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra Connector`

[Benedict Jin (JIRA)]

Benedict Jin created FLINK-6868:
---

 Summary: Using `scala.binary.version` for `flink-streaming-scala` 
in `Cassandra Connector`
 Key: FLINK-6868
 URL: https://issues.apache.org/jira/browse/FLINK-6868
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.3.0
Reporter: Benedict Jin
 Fix For: 1.4.0


Shoud using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
Connector`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6868) Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra Connector`

[Benedict Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benedict Jin updated FLINK-6868:

Issue Type: Improvement  (was: Bug)

> Using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
> Connector`
> -
>
> Key: FLINK-6868
> URL: https://issues.apache.org/jira/browse/FLINK-6868
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Affects Versions: 1.3.0
>Reporter: Benedict Jin
>  Labels: improvement
> Fix For: 1.4.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Shoud using `scala.binary.version` for `flink-streaming-scala` in `Cassandra 
> Connector`.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6867) Elasticsearch 1.x IT test still instable due to embedded node instability

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-6867:
---
Description: 
The integration tests for Elasticsearch 1.x seems to still be instable, being 
that the test is failing more frequently recently. One example is [1].

The last attempt to fix this was FLINK-5772, in which the test attempts was 
increased to 3. This doesn't seem to fix the issue. In the worst scenario, 
since the root cause is an instability with Elasticsearch 1.x's embedded node, 
and ES 1.x is a very old version that is usually recommended to be upgraded 
from, we could also consider removing the IT test for ES 1.x.

[1] https://travis-ci.org/apache/flink/jobs/240444523

  was:
The integration tests for Elasticsearch 1.x seems to still be instable, being 
that the test is failing more frequently recently. One example is [1].

The last attempt to fix this was FLINK-5772, in which the test attempts was 
increased to 3. This doesn't seem to fix the issue. In the worst scenario, 
since the root cause is an instability with Elasticsearch 1.x's embedded node, 
and ES 1.x is a very old version that is usually recommended to be upgraded 
from, we could also consider removing the IT test for ES 1.x.


> Elasticsearch 1.x IT test still instable due to embedded node instability
> -
>
> Key: FLINK-6867
> URL: https://issues.apache.org/jira/browse/FLINK-6867
> Project: Flink
>  Issue Type: Bug
>  Components: ElasticSearch Connector, Tests
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>
> The integration tests for Elasticsearch 1.x seems to still be instable, being 
> that the test is failing more frequently recently. One example is [1].
> The last attempt to fix this was FLINK-5772, in which the test attempts was 
> increased to 3. This doesn't seem to fix the issue. In the worst scenario, 
> since the root cause is an instability with Elasticsearch 1.x's embedded 
> node, and ES 1.x is a very old version that is usually recommended to be 
> upgraded from, we could also consider removing the IT test for ES 1.x.
> [1] https://travis-ci.org/apache/flink/jobs/240444523



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-6867) Elasticsearch 1.x ITCase still instable due to embedded node instability

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-6867:
---
Summary: Elasticsearch 1.x ITCase still instable due to embedded node 
instability  (was: Elasticsearch 1.x IT test still instable due to embedded 
node instability)

> Elasticsearch 1.x ITCase still instable due to embedded node instability
> 
>
> Key: FLINK-6867
> URL: https://issues.apache.org/jira/browse/FLINK-6867
> Project: Flink
>  Issue Type: Bug
>  Components: ElasticSearch Connector, Tests
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>
> The integration tests for Elasticsearch 1.x seems to still be instable, being 
> that the test is failing more frequently recently. One example is [1].
> The last attempt to fix this was FLINK-5772, in which the test attempts was 
> increased to 3. This doesn't seem to fix the issue. In the worst scenario, 
> since the root cause is an instability with Elasticsearch 1.x's embedded 
> node, and ES 1.x is a very old version that is usually recommended to be 
> upgraded from, we could also consider removing the IT test for ES 1.x.
> [1] https://travis-ci.org/apache/flink/jobs/240444523



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-6867) Elasticsearch 1.x IT test still instable due to embedded node instability

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-6867:
--

 Summary: Elasticsearch 1.x IT test still instable due to embedded 
node instability
 Key: FLINK-6867
 URL: https://issues.apache.org/jira/browse/FLINK-6867
 Project: Flink
  Issue Type: Bug
  Components: ElasticSearch Connector, Tests
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


The integration tests for Elasticsearch 1.x seems to still be instable, being 
that the test is failing more frequently recently. One example is [1].

The last attempt to fix this was FLINK-5772, in which the test attempts was 
increased to 3. This doesn't seem to fix the issue. In the worst scenario, 
since the root cause is an instability with Elasticsearch 1.x's embedded node, 
and ES 1.x is a very old version that is usually recommended to be upgraded 
from, we could also consider removing the IT test for ES 1.x.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6494) Migrate ResourceManager configuration options

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042149#comment-16042149
 ] 

ASF GitHub Bot commented on FLINK-6494:
---

GitHub user zjureel reopened a pull request:

https://github.com/apache/flink/pull/4075

[FLINK-6494] Migrate ResourceManager/Yarn/Mesos configuration options

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjureel/flink FLINK-6494

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4075


commit a33bee88929d1c98d4bd9c1ba175fb5bebd9aa96
Author: zjureel 
Date:   2017-06-08T03:38:56Z

[FLINK-6494] Migrate ResourceManager configuration options




> Migrate ResourceManager configuration options
> -
>
> Key: FLINK-6494
> URL: https://issues.apache.org/jira/browse/FLINK-6494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, ResourceManager
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4075: [FLINK-6494] Migrate ResourceManager/Yarn/Mesos co...

[zjureel]

GitHub user zjureel reopened a pull request:

https://github.com/apache/flink/pull/4075

[FLINK-6494] Migrate ResourceManager/Yarn/Mesos configuration options

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjureel/flink FLINK-6494

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4075


commit a33bee88929d1c98d4bd9c1ba175fb5bebd9aa96
Author: zjureel 
Date:   2017-06-08T03:38:56Z

[FLINK-6494] Migrate ResourceManager configuration options




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-6488) Mark deprecated for 'start-local.sh' script

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated FLINK-6488:

Summary: Mark deprecated for 'start-local.sh' script  (was: Remove 
'start-local.sh' script)

> Mark deprecated for 'start-local.sh' script
> ---
>
> Key: FLINK-6488
> URL: https://issues.apache.org/jira/browse/FLINK-6488
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> The {{start-cluster.sh}} scripts work locally now, without needing SSH setup.
> We can remove {{start-local.sh}} without any loss of functionality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6488) Remove 'start-local.sh' script

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042060#comment-16042060
 ] 

mingleizhang commented on FLINK-6488:
-

Hey, [~Zentol] Here is the sample code for this change in {{start-local.sh}}. 
Check out and what do you think ? :)

{code}STREAMING_MODE=$1

echo "Warning: this file is deprecated and will be remove in the future | You 
can call start-cluster.sh instead"

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`{code}

> Remove 'start-local.sh' script
> --
>
> Key: FLINK-6488
> URL: https://issues.apache.org/jira/browse/FLINK-6488
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Stephan Ewen
>Assignee: mingleizhang
>
> The {{start-cluster.sh}} scripts work locally now, without needing SSH setup.
> We can remove {{start-local.sh}} without any loss of functionality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (FLINK-6643) Flink restarts job in HA even if NoRestartStrategy is set

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042036#comment-16042036
 ] 

mingleizhang edited comment on FLINK-6643 at 6/8/17 1:06 AM:
-

Thanks for review [~till.rohrmann]. Just since the logging said {{Using restart 
strategy NoRestartStrategy for f94b1f7a0e9e3dbcb160c687e476ca77}} with the 
corresponding code {{log.info(s"Using restart strategy $restartStrategy for 
$jobId.")}} . It just seems confuse people who first see this kinda logging. 
Doesn't it consume performance if recover a job without really need to do it , 
or it probably for fault-tolerant or something like that. This issue and PR 
will be close this night here if there is no more question. 


was (Author: mingleizhang):
Thanks for review [~till.rohrmann]. Just since the logging said {{Using restart 
strategy NoRestartStrategy for f94b1f7a0e9e3dbcb160c687e476ca77}} with the 
corresponding code {{log.info(s"Using restart strategy $restartStrategy for 
$jobId.")}} . It just seems confuse people who first this kinda logging. 
Doesn't it consume performance if recover a job without really need to do it , 
or it probably for fault-tolerant or something like that. This issue and PR 
will be close this night here if there is no more question. 

> Flink restarts job in HA even if NoRestartStrategy is set
> -
>
> Key: FLINK-6643
> URL: https://issues.apache.org/jira/browse/FLINK-6643
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.0
>Reporter: Robert Metzger
>Assignee: mingleizhang
>Priority: Critical
>  Labels: flink-rel-1.3.1-blockers
>
> While testing Flink 1.3 RC1, I found that the JobManager is trying to recover 
> a job that had the {{NoRestartStrategy}} set.
> {code}
> 2017-05-19 15:09:04,038 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover all jobs.
> 2017-05-19 15:09:04,039 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Retrieving all stored job ids from ZooKeeper under 
> flink/application_149487096_0064/jobgraphs.
> 2017-05-19 15:09:04,041 INFO  org.apache.flink.yarn.YarnJobManager
>   - There are 1 jobs to recover. Starting the job recovery.
> 2017-05-19 15:09:04,043 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,043 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovering job graph f94b1f7a0e9e3dbcb160c687e476ca77 from 
> flink/application_149487096_0064/jobgraphs/f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,078 WARN  org.apache.hadoop.util.NativeCodeLoader 
>   - Unable to load native-hadoop library for your platform... 
> using builtin-java classes where applicable
> 2017-05-19 15:09:04,142 INFO  
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovered SubmittedJobGraph(f94b1f7a0e9e3dbcb160c687e476ca77, 
> JobInfo(clients: 
> Set((Actor[akka.tcp://flink@permanent-qa-cluster-master.c.astral-sorter-757.internal:40391/user/$a#-155566858],EXECUTION_RESULT_AND_STATE_CHANGES)),
>  start: 1495206476885)).
> 2017-05-19 15:09:04,142 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting recovered job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,143 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting job f94b1f7a0e9e3dbcb160c687e476ca77 
> (CarTopSpeedWindowingExample) (Recovery).
> 2017-05-19 15:09:04,151 INFO  org.apache.flink.yarn.YarnJobManager
>   - Using restart strategy NoRestartStrategy for 
> f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,163 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6643) Flink restarts job in HA even if NoRestartStrategy is set

[mingleizhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042036#comment-16042036
 ] 

mingleizhang commented on FLINK-6643:
-

Thanks for review [~till.rohrmann]. Just since the logging said {{Using restart 
strategy NoRestartStrategy for f94b1f7a0e9e3dbcb160c687e476ca77}} with the 
corresponding code {{log.info(s"Using restart strategy $restartStrategy for 
$jobId.")}} . It just seems confuse people who first this kinda logging. 
Doesn't it consume performance if recover a job without really need to do it , 
or it probably for fault-tolerant or something like that. This issue and PR 
will be close this night here if there is no more question. 

> Flink restarts job in HA even if NoRestartStrategy is set
> -
>
> Key: FLINK-6643
> URL: https://issues.apache.org/jira/browse/FLINK-6643
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.0
>Reporter: Robert Metzger
>Assignee: mingleizhang
>Priority: Critical
>  Labels: flink-rel-1.3.1-blockers
>
> While testing Flink 1.3 RC1, I found that the JobManager is trying to recover 
> a job that had the {{NoRestartStrategy}} set.
> {code}
> 2017-05-19 15:09:04,038 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover all jobs.
> 2017-05-19 15:09:04,039 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Retrieving all stored job ids from ZooKeeper under 
> flink/application_149487096_0064/jobgraphs.
> 2017-05-19 15:09:04,041 INFO  org.apache.flink.yarn.YarnJobManager
>   - There are 1 jobs to recover. Starting the job recovery.
> 2017-05-19 15:09:04,043 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,043 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovering job graph f94b1f7a0e9e3dbcb160c687e476ca77 from 
> flink/application_149487096_0064/jobgraphs/f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,078 WARN  org.apache.hadoop.util.NativeCodeLoader 
>   - Unable to load native-hadoop library for your platform... 
> using builtin-java classes where applicable
> 2017-05-19 15:09:04,142 INFO  
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovered SubmittedJobGraph(f94b1f7a0e9e3dbcb160c687e476ca77, 
> JobInfo(clients: 
> Set((Actor[akka.tcp://flink@permanent-qa-cluster-master.c.astral-sorter-757.internal:40391/user/$a#-155566858],EXECUTION_RESULT_AND_STATE_CHANGES)),
>  start: 1495206476885)).
> 2017-05-19 15:09:04,142 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting recovered job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,143 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting job f94b1f7a0e9e3dbcb160c687e476ca77 
> (CarTopSpeedWindowingExample) (Recovery).
> 2017-05-19 15:09:04,151 INFO  org.apache.flink.yarn.YarnJobManager
>   - Using restart strategy NoRestartStrategy for 
> f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,163 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6850) Links on "Powered by Flink" page not rendered correctly

[Hai Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hai Zhou closed FLINK-6850.
---
Resolution: Won't Do

> Links on "Powered by Flink" page  not rendered correctly
> 
>
> Key: FLINK-6850
> URL: https://issues.apache.org/jira/browse/FLINK-6850
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Hai Zhou
>Priority: Minor
>
> The "Powered by Flink" page on the Flink website, some links are broken:
> 1.***Bouygues Telecom*** 
> See Bouygues Telcom at Flink Forward 2016
> (I guess) the right link:  
> 
> 2. ***Otto Group*** 
> See Otto at Flink Forward 2016
> (I guess) the right link: 
> 
> 3. ***ResearchGate*** 
> See ResearchGate at Flink Forward 2016
> (I guess) the right link: 
> 
>  I cannot to update  the links.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6281) Create TableSink for JDBC

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041825#comment-16041825
 ] 

ASF GitHub Bot commented on FLINK-6281:
---

Github user haohui commented on a diff in the pull request:

https://github.com/apache/flink/pull/3712#discussion_r120765156
  
--- Diff: 
flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java
 ---
@@ -202,14 +202,20 @@ public void writeRecord(Row row) throws IOException {
upload.addBatch();
batchCount++;
if (batchCount >= batchInterval) {
-   upload.executeBatch();
-   batchCount = 0;
+   flush();
}
} catch (SQLException | IllegalArgumentException e) {
throw new IllegalArgumentException("writeRecord() 
failed", e);
}
}
 
+   void flush() throws SQLException {
+   if (upload != null) {
+   upload.executeBatch();
--- End diff --

It is a synchronous call. It will throw `SQLException` and abort the sink. 
The behavior has not been changed.


> Create TableSink for JDBC
> -
>
> Key: FLINK-6281
> URL: https://issues.apache.org/jira/browse/FLINK-6281
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> It would be nice to integrate the table APIs with the JDBC connectors so that 
> the rows in the tables can be directly pushed into JDBC.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3712: [FLINK-6281] Create TableSink for JDBC.

[haohui]

Github user haohui commented on a diff in the pull request:

https://github.com/apache/flink/pull/3712#discussion_r120765156
  
--- Diff: 
flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java
 ---
@@ -202,14 +202,20 @@ public void writeRecord(Row row) throws IOException {
upload.addBatch();
batchCount++;
if (batchCount >= batchInterval) {
-   upload.executeBatch();
-   batchCount = 0;
+   flush();
}
} catch (SQLException | IllegalArgumentException e) {
throw new IllegalArgumentException("writeRecord() 
failed", e);
}
}
 
+   void flush() throws SQLException {
+   if (upload != null) {
+   upload.executeBatch();
--- End diff --

It is a synchronous call. It will throw `SQLException` and abort the sink. 
The behavior has not been changed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6833) Race condition: Asynchronous checkpointing task can fail completed StreamTask

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041788#comment-16041788
 ] 

ASF GitHub Bot commented on FLINK-6833:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4058
  
I would like to merge this PR if I could clarify all concerns and there are 
no further objections.


> Race condition: Asynchronous checkpointing task can fail completed StreamTask
> -
>
> Key: FLINK-6833
> URL: https://issues.apache.org/jira/browse/FLINK-6833
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime, State Backends, Checkpointing
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> A {{StreamTask}} which is about to finish and thus transitioning its 
> containing {{Task}} into the {{ExecutionState.FINISHED}} state, can be failed 
> by a concurrent asynchronous checkpointing operation. The problem is that 
> upon termination the {{StreamTask}} cancels all concurrent operations 
> (amongst others ongoing asynchronous checkpoints). The cancellation of the 
> async checkpoint triggers the {{StreamTask#handleAsyncException}} call which 
> will fail the containing {{Task}}. If the {{handleAsyncException}} completes 
> before the {{StreamTask}} has been properly terminated, then the containing 
> {{Task}} will transition into {{ExecutionState.FAILED}} instead of 
> {{ExecutionState.FINISHED}}.
> In order to resolve this race condition, we should check in the 
> {{StreamTask#handleAsyncException}} whether the {{StreamTask}} is still 
> running or has already been terminated. Only in the former case, we should 
> fail the containing {{Task}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4058: [FLINK-6833] [task] Fail StreamTask only due to async exc...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4058
  
I would like to merge this PR if I could clarify all concerns and there are 
no further objections.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6643) Flink restarts job in HA even if NoRestartStrategy is set

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041778#comment-16041778
 ] 

ASF GitHub Bot commented on FLINK-6643:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4049
  
Thanks for your contribution @zhangminglei. I fear, though, that we cannot 
merge your PR like it is since it prevents the job submission for all jobs 
which have `NoRestartStrategy` set. Furthermore I think the issue FLINK-6643 is 
actually a non-issue. Therefore, you can close this PR.


> Flink restarts job in HA even if NoRestartStrategy is set
> -
>
> Key: FLINK-6643
> URL: https://issues.apache.org/jira/browse/FLINK-6643
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.0
>Reporter: Robert Metzger
>Assignee: mingleizhang
>Priority: Critical
>  Labels: flink-rel-1.3.1-blockers
>
> While testing Flink 1.3 RC1, I found that the JobManager is trying to recover 
> a job that had the {{NoRestartStrategy}} set.
> {code}
> 2017-05-19 15:09:04,038 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover all jobs.
> 2017-05-19 15:09:04,039 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Retrieving all stored job ids from ZooKeeper under 
> flink/application_149487096_0064/jobgraphs.
> 2017-05-19 15:09:04,041 INFO  org.apache.flink.yarn.YarnJobManager
>   - There are 1 jobs to recover. Starting the job recovery.
> 2017-05-19 15:09:04,043 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,043 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovering job graph f94b1f7a0e9e3dbcb160c687e476ca77 from 
> flink/application_149487096_0064/jobgraphs/f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,078 WARN  org.apache.hadoop.util.NativeCodeLoader 
>   - Unable to load native-hadoop library for your platform... 
> using builtin-java classes where applicable
> 2017-05-19 15:09:04,142 INFO  
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovered SubmittedJobGraph(f94b1f7a0e9e3dbcb160c687e476ca77, 
> JobInfo(clients: 
> Set((Actor[akka.tcp://flink@permanent-qa-cluster-master.c.astral-sorter-757.internal:40391/user/$a#-155566858],EXECUTION_RESULT_AND_STATE_CHANGES)),
>  start: 1495206476885)).
> 2017-05-19 15:09:04,142 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting recovered job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,143 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting job f94b1f7a0e9e3dbcb160c687e476ca77 
> (CarTopSpeedWindowingExample) (Recovery).
> 2017-05-19 15:09:04,151 INFO  org.apache.flink.yarn.YarnJobManager
>   - Using restart strategy NoRestartStrategy for 
> f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,163 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4049: [FLINK-6643] [JobManager] Flink restarts job in HA even i...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4049
  
Thanks for your contribution @zhangminglei. I fear, though, that we cannot 
merge your PR like it is since it prevents the job submission for all jobs 
which have `NoRestartStrategy` set. Furthermore I think the issue FLINK-6643 is 
actually a non-issue. Therefore, you can close this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6643) Flink restarts job in HA even if NoRestartStrategy is set

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041775#comment-16041775
 ] 

Till Rohrmann commented on FLINK-6643:
--

I'm not sure whether this is actually a valid issue. The logging excerpt does 
not show any restart attempts. What it shows is that a job is recovered. 
Furthermore it logs the restart strategy for this job. Since the restart 
strategy has nothing to do with the job recovery, it should be ok to recover a 
job even if it has {{NoRestartStrategy}} set. Only in case of a job failure it 
should not try to recover. Therefore, I think we can close this issue.

> Flink restarts job in HA even if NoRestartStrategy is set
> -
>
> Key: FLINK-6643
> URL: https://issues.apache.org/jira/browse/FLINK-6643
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.3.0
>Reporter: Robert Metzger
>Assignee: mingleizhang
>Priority: Critical
>  Labels: flink-rel-1.3.1-blockers
>
> While testing Flink 1.3 RC1, I found that the JobManager is trying to recover 
> a job that had the {{NoRestartStrategy}} set.
> {code}
> 2017-05-19 15:09:04,038 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover all jobs.
> 2017-05-19 15:09:04,039 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Retrieving all stored job ids from ZooKeeper under 
> flink/application_149487096_0064/jobgraphs.
> 2017-05-19 15:09:04,041 INFO  org.apache.flink.yarn.YarnJobManager
>   - There are 1 jobs to recover. Starting the job recovery.
> 2017-05-19 15:09:04,043 INFO  org.apache.flink.yarn.YarnJobManager
>   - Attempting to recover job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,043 DEBUG 
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovering job graph f94b1f7a0e9e3dbcb160c687e476ca77 from 
> flink/application_149487096_0064/jobgraphs/f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,078 WARN  org.apache.hadoop.util.NativeCodeLoader 
>   - Unable to load native-hadoop library for your platform... 
> using builtin-java classes where applicable
> 2017-05-19 15:09:04,142 INFO  
> org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - 
> Recovered SubmittedJobGraph(f94b1f7a0e9e3dbcb160c687e476ca77, 
> JobInfo(clients: 
> Set((Actor[akka.tcp://flink@permanent-qa-cluster-master.c.astral-sorter-757.internal:40391/user/$a#-155566858],EXECUTION_RESULT_AND_STATE_CHANGES)),
>  start: 1495206476885)).
> 2017-05-19 15:09:04,142 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting recovered job f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,143 INFO  org.apache.flink.yarn.YarnJobManager
>   - Submitting job f94b1f7a0e9e3dbcb160c687e476ca77 
> (CarTopSpeedWindowingExample) (Recovery).
> 2017-05-19 15:09:04,151 INFO  org.apache.flink.yarn.YarnJobManager
>   - Using restart strategy NoRestartStrategy for 
> f94b1f7a0e9e3dbcb160c687e476ca77.
> 2017-05-19 15:09:04,163 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph- Job recovers 
> via failover strategy: full graph restart
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6833) Race condition: Asynchronous checkpointing task can fail completed StreamTask

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041755#comment-16041755
 ] 

ASF GitHub Bot commented on FLINK-6833:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4058#discussion_r120757786
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
 ---
@@ -824,11 +824,14 @@ public ProcessingTimeService 
getProcessingTimeService() {
 * FAILED, and, if the invokable code is running, starts an 
asynchronous thread
 * that aborts that code.
 *
-* This method never blocks.
+* This method never blocks.
 */
@Override
public void handleAsyncException(String message, Throwable exception) {
-   getEnvironment().failExternally(exception);
+   if (isRunning) {
--- End diff --

If `isRunning == true` when entering the if branch, then depending on what 
happens before `failExternally`, we can assume that the `handleAsyncException` 
either happened atomically before `isRunning` was set to `false` or not. But 
what we don't want to happen is that if `isRunning == false`, that we can still 
fail the task. Thus, I think it solves a valid problem.


> Race condition: Asynchronous checkpointing task can fail completed StreamTask
> -
>
> Key: FLINK-6833
> URL: https://issues.apache.org/jira/browse/FLINK-6833
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime, State Backends, Checkpointing
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> A {{StreamTask}} which is about to finish and thus transitioning its 
> containing {{Task}} into the {{ExecutionState.FINISHED}} state, can be failed 
> by a concurrent asynchronous checkpointing operation. The problem is that 
> upon termination the {{StreamTask}} cancels all concurrent operations 
> (amongst others ongoing asynchronous checkpoints). The cancellation of the 
> async checkpoint triggers the {{StreamTask#handleAsyncException}} call which 
> will fail the containing {{Task}}. If the {{handleAsyncException}} completes 
> before the {{StreamTask}} has been properly terminated, then the containing 
> {{Task}} will transition into {{ExecutionState.FAILED}} instead of 
> {{ExecutionState.FINISHED}}.
> In order to resolve this race condition, we should check in the 
> {{StreamTask#handleAsyncException}} whether the {{StreamTask}} is still 
> running or has already been terminated. Only in the former case, we should 
> fail the containing {{Task}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4058: [FLINK-6833] [task] Fail StreamTask only due to as...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4058#discussion_r120757786
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
 ---
@@ -824,11 +824,14 @@ public ProcessingTimeService 
getProcessingTimeService() {
 * FAILED, and, if the invokable code is running, starts an 
asynchronous thread
 * that aborts that code.
 *
-* This method never blocks.
+* This method never blocks.
 */
@Override
public void handleAsyncException(String message, Throwable exception) {
-   getEnvironment().failExternally(exception);
+   if (isRunning) {
--- End diff --

If `isRunning == true` when entering the if branch, then depending on what 
happens before `failExternally`, we can assume that the `handleAsyncException` 
either happened atomically before `isRunning` was set to `false` or not. But 
what we don't want to happen is that if `isRunning == false`, that we can still 
fail the task. Thus, I think it solves a valid problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041742#comment-16041742
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120749760
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
--- End diff --

Add links to the relevant sections of the Overview page:
* http://localhost:4000/dev/table/common.html#mixing-table-api-and-sql
* 
http://localhost:4000/dev/table/common.html#integration-with-datastream-and-dataset-api
* http://localhost:4000/dev/table/common.html#emit-a-table


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041751#comment-16041751
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753703
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
--- End diff --

Add a space between `Distinct` and `(Batch only)`. Same for the following 
`(Batch only)` labels.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041752#comment-16041752
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120750248
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique 
table name under the `Table`'s `TableEnvironment` and return the table name. 
This allows SQL to be called directly on tables in a string concatenation (see 
examples below).
--- End diff --

`... will automatically register the table under a unique name in its 
TableEnvironment and ...`?


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041744#comment-16041744
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754480
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
--- End diff --

Add a comment that join must have at least one conjunctive equality 
predicate. CROSS or Theta joins are not supported.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041747#comment-16041747
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755459
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
--- End diff --

Add a line for `User defined aggregation function`?


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041734#comment-16041734
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120749951
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
--- End diff --

Add a link to 
http://localhost:4000/dev/table/common.html#register-an-external-catalog


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041737#comment-16041737
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753514
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
--- End diff --

indention. I think we should format most following queries (everything 
that's not `SELECT FROM`) in multiple lines, for example like:
```
SELECT ...
FROM ...
GROUP BY ...
```


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041750#comment-16041750
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754779
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
+  
+{% highlight sql %}
+SELECT * FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+SELECT * FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+  
+
+
+   Expanding arrays into a relation
+   
+{% highlight sql %}
+SELECT users, tag FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+  
+
+
+   User Defined Table Function (UDTF)
+   
+  SQL queries can refer to UDTFs to expand a value into a relation 
provided that they are registered in the TableEnvironment.
--- End diff --

`TableFunction` is not nicely rendered in my setup. Can you 
check on yours?


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: D

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041746#comment-16041746
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755990
  
--- Diff: docs/dev/table/sql.md ---
@@ -217,7 +444,7 @@ Group windows are defined in the `GROUP BY` clause of a 
SQL query. Just like que
   
 
 
-For SQL queries on streaming tables, the `time_attr` argument of the group 
window function must be one of the `rowtime()` or `proctime()` time-indicators, 
which distinguish between event or processing time, respectively. For SQL on 
batch tables, the `time_attr` argument of the group window function must be an 
attribute of type `TIMESTAMP`. 
+For SQL queries on streaming tables, the `time_attr` argument of the group 
window function must refer to the virtual column that specifies the processing 
time or the event time. For SQL on batch tables, the `time_attr` argument of 
the group window function must be an attribute of type `TIMESTAMP`.
--- End diff --

Add a link to the time attribute section on the Table API Streaming doc 
page.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041749#comment-16041749
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754991
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
+  
+{% highlight sql %}
+SELECT * FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+SELECT * FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+  
+
+
+   Expanding arrays into a relation
+   
+{% highlight sql %}
+SELECT users, tag FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+  
+
+
+   User Defined Table Function (UDTF)
+   
+  SQL queries can refer to UDTFs to expand a value into a relation 
provided that they are registered in the TableEnvironment.
--- End diff --

TableFunctions do also accept multiple values. Add a link to the UDF docs 
page.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Component

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041739#comment-16041739
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120751070
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
--- End diff --

I would not mention the translation into DataStream and DataSet programs 
(see another comment below).


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041745#comment-16041745
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752941
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
--- End diff --

User Defined Scalar Functions


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120749951
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
--- End diff --

Add a link to 
http://localhost:4000/dev/table/common.html#register-an-external-catalog


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041740#comment-16041740
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120756834
  
--- Diff: docs/dev/table/sql.md ---
@@ -340,18 +567,14 @@ val result4 = tableEnv.sql(
 
 {% top %}
 
-### Limitations
-
-**TODO: Integrate this with the examples**
-
  Batch
--- End diff --

I would remove the `Batch` and `Streaming` limitations sections. 
Instead, I would integrate the relevant information with the `Operations` 
sections, e.g., add a comments to UNNEST that WITH ORDINALITY is not supported, 
etc.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041736#comment-16041736
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752386
  
--- Diff: docs/dev/table/sql.md ---
@@ -82,14 +84,12 @@ val result2 = tableEnv.sql(
 
 
 
-**TODO: Add some intro.**
-
 {% top %}
 
 Supported Syntax
 
 
-Flink uses [Apache 
Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. 
Currently, Flink SQL only supports query-related SQL syntax and only a subset 
of the comprehensive SQL standard. The following BNF-grammar describes the 
supported SQL features:
+Flink parses SQL using [Apache 
Calcite](https://calcite.apache.org/docs/reference.html). Flink supports 
standard ANSI SQL, but it provides no supports for DML and DDL. The following 
BNF-grammar describes the supported SQL features:
--- End diff --

We should mention that not all features are supported by batch and 
streaming. The `Operations` section will show which features are supported by 
batch and streaming.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041738#comment-16041738
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752761
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
--- End diff --

Markdown highlighting does not seem to work here. We should also add a link 
to the UDF docs.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041748#comment-16041748
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754137
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
--- End diff --

Add a brief description that only all aggregates must be defined over the 
same window, i.e., we do not support aggregates over different windows.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041743#comment-16041743
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755168
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
--- End diff --

Add a comment that Flink does not optimize join order yet and joins in the 
same order as specified in the query.


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120750870
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique 
table name under the `Table`'s `TableEnvironment` and return the table name. 
This allows SQL to be called directly on tables in a string concatenation (see 
examples below).
 
-**TODO: Rework intro. Move some parts below. **
+*Note: Flink's SQL support is not yet feature complete. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
 
 * This will be replaced by the TOC
 {:toc}
 
 Specifying a Query
 ---
 
+Here are a few examples on specifying a DataStream / DataSet program using 
SQL:
--- End diff --

I would not mention that SQL queries are translated into DataStream / 
DataSet programs. This is useful information for some users and it should be 
mentioned in an internals section. I would assume that most users do not care 
about this and might be even confused.

What do you think @haohui ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041741#comment-16041741
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753745
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
--- End diff --

Over windows are only supported for streaming


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755990
  
--- Diff: docs/dev/table/sql.md ---
@@ -217,7 +444,7 @@ Group windows are defined in the `GROUP BY` clause of a 
SQL query. Just like que
   
 
 
-For SQL queries on streaming tables, the `time_attr` argument of the group 
window function must be one of the `rowtime()` or `proctime()` time-indicators, 
which distinguish between event or processing time, respectively. For SQL on 
batch tables, the `time_attr` argument of the group window function must be an 
attribute of type `TIMESTAMP`. 
+For SQL queries on streaming tables, the `time_attr` argument of the group 
window function must refer to the virtual column that specifies the processing 
time or the event time. For SQL on batch tables, the `time_attr` argument of 
the group window function must be an attribute of type `TIMESTAMP`.
--- End diff --

Add a link to the time attribute section on the Table API Streaming doc 
page.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754779
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
+  
+{% highlight sql %}
+SELECT * FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+SELECT * FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+  
+
+
+   Expanding arrays into a relation
+   
+{% highlight sql %}
+SELECT users, tag FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+  
+
+
+   User Defined Table Function (UDTF)
+   
+  SQL queries can refer to UDTFs to expand a value into a relation 
provided that they are registered in the TableEnvironment.
--- End diff --

`TableFunction` is not nicely rendered in my setup. Can you 
check on yours?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755168
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
--- End diff --

Add a comment that Flink does not optimize join order yet and joins in the 
same order as specified in the query.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120756834
  
--- Diff: docs/dev/table/sql.md ---
@@ -340,18 +567,14 @@ val result4 = tableEnv.sql(
 
 {% top %}
 
-### Limitations
-
-**TODO: Integrate this with the examples**
-
  Batch
--- End diff --

I would remove the `Batch` and `Streaming` limitations sections. 
Instead, I would integrate the relevant information with the `Operations` 
sections, e.g., add a comments to UNNEST that WITH ORDINALITY is not supported, 
etc.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754991
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
+  
+{% highlight sql %}
+SELECT * FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+SELECT * FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+  
+
+
+   Expanding arrays into a relation
+   
+{% highlight sql %}
+SELECT users, tag FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+  
+
+
+   User Defined Table Function (UDTF)
+   
+  SQL queries can refer to UDTFs to expand a value into a relation 
provided that they are registered in the TableEnvironment.
--- End diff --

TableFunctions do also accept multiple values. Add a link to the UDF docs 
page.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753514
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
--- End diff --

indention. I think we should format most following queries (everything 
that's not `SELECT FROM`) in multiple lines, for example like:
```
SELECT ...
FROM ...
GROUP BY ...
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6749) Table API / SQL Docs: SQL Page

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041735#comment-16041735
 ] 

ASF GitHub Bot commented on FLINK-6749:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120750870
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique 
table name under the `Table`'s `TableEnvironment` and return the table name. 
This allows SQL to be called directly on tables in a string concatenation (see 
examples below).
 
-**TODO: Rework intro. Move some parts below. **
+*Note: Flink's SQL support is not yet feature complete. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
 
 * This will be replaced by the TOC
 {:toc}
 
 Specifying a Query
 ---
 
+Here are a few examples on specifying a DataStream / DataSet program using 
SQL:
--- End diff --

I would not mention that SQL queries are translated into DataStream / 
DataSet programs. This is useful information for some users and it should be 
mentioned in an internals section. I would assume that most users do not care 
about this and might be even confused.

What do you think @haohui ?


> Table API / SQL Docs: SQL Page
> --
>
> Key: FLINK-6749
> URL: https://issues.apache.org/jira/browse/FLINK-6749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table API & SQL
>Affects Versions: 1.3.0
>Reporter: Fabian Hueske
>Assignee: Haohui Mai
>
> Update and refine {{./docs/dev/table/sql.md}} in feature branch 
> https://github.com/apache/flink/tree/tableDocs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120755459
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
--- End diff --

Add a line for `User defined aggregation function`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120749760
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
--- End diff --

Add links to the relevant sections of the Overview page:
* http://localhost:4000/dev/table/common.html#mixing-table-api-and-sql
* 
http://localhost:4000/dev/table/common.html#integration-with-datastream-and-dataset-api
* http://localhost:4000/dev/table/common.html#emit-a-table


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752386
  
--- Diff: docs/dev/table/sql.md ---
@@ -82,14 +84,12 @@ val result2 = tableEnv.sql(
 
 
 
-**TODO: Add some intro.**
-
 {% top %}
 
 Supported Syntax
 
 
-Flink uses [Apache 
Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. 
Currently, Flink SQL only supports query-related SQL syntax and only a subset 
of the comprehensive SQL standard. The following BNF-grammar describes the 
supported SQL features:
+Flink parses SQL using [Apache 
Calcite](https://calcite.apache.org/docs/reference.html). Flink supports 
standard ANSI SQL, but it provides no supports for DML and DDL. The following 
BNF-grammar describes the supported SQL features:
--- End diff --

We should mention that not all features are supported by batch and 
streaming. The `Operations` section will show which features are supported by 
batch and streaming.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752941
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
--- End diff --

User Defined Scalar Functions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754137
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
--- End diff --

Add a brief description that only all aggregates must be defined over the 
same window, i.e., we do not support aggregates over different windows.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120750248
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
 
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be 
registered in the `TableEnvironment` in order to be accessible by a SQL query 
(see [Registering Tables](#registering-tables)). For convenience 
`Table.toString()` will automatically register an unique table name under the 
`Table`'s `TableEnvironment` and return the table name. So it allows to call 
SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be 
registered in the `TableEnvironment` (see see [Registering 
Tables](common.html#register-a-table-in-the-catalog)). Possible data sources 
include Tables, DataSets, DataStreams, and external TableSources. 
Alternatively, users can also register external catalogs in the 
`TableEnvironment` to specify the location of the data sources.
 
-*Note: Flink's SQL support is not feature complete, yet. Queries that 
include unsupported SQL features will cause a `TableException`. The limitations 
of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique 
table name under the `Table`'s `TableEnvironment` and return the table name. 
This allows SQL to be called directly on tables in a string concatenation (see 
examples below).
--- End diff --

`... will automatically register the table under a unique name in its 
TableEnvironment and ...`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753703
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
--- End diff --

Add a space between `Distinct` and `(Batch only)`. Same for the following 
`(Batch only)` labels.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120753745
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
--- End diff --

Over windows are only supported for streaming


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120754480
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+  
+
+  
+
+
 
 {% top %}
 
-### GroupBy Windows
+### Aggregations
 
-**TODO: Integrate this with the examples**
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+
+  GroupBy
+  
+{% highlight sql %}
+SELECT a, SUM(b) as d FROM Orders GROUP BY a
+{% endhighlight %}
+  
+
+
+   GroupBy Window
+   
+Use a group window to compute a single result row per group. 
(See Group Windows for more details.)
+{% highlight sql %}
+SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(rowtime, 
INTERVAL '1' DAY), user
+{% endhighlight %}
+  
+
+
+   Over Window
+   
+{% highlight sql %}
+SELECT COUNT(amount) OVER (PARTITION BY user ORDER BY proctime ROWS 
BETWEEN 2 PRECEDING AND CURRENT ROW) FROM Orders
+{% endhighlight %}
+  
+
+
+  Distinct(Batch only)
+  
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+  
+
+
+  Grouping sets, rollup, cube(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+  
+
+
+  Having(Batch only)
+  
+{% highlight sql %}
+SELECT SUM(amount) FROM Orders GROUP BY users HAVING SUM(amount) > 50
+{% endhighlight %}
+  
+
+  
+
+
+
+{% top %}
+
+### Joins
+
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+  Inner Equi-join / Outer Equi-join(Batch 
only)
--- End diff --

Add a comment that join must have at least one conjunctive equality 
predicate. CROSS or Theta joins are not supported.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120752761
  
--- Diff: docs/dev/table/sql.md ---
@@ -163,31 +164,257 @@ For a better definition of SQL queries within a Java 
String, Flink SQL uses a le
 
 {% top %}
 
-Example Queries

+Operations
+
+
+### Scan, Projection, and Filter
 
-**TODO: Add a examples for different operations with similar structure as 
for the Table API. Add highlighted tags if an operation is not supported by 
stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
-  * GroupBy
-  * GroupBy Windows (TUMBLE, HOP, SESSION)
-  * OVER windows (Only Stream)
-  * Grouping sets, rollup, cube (only batch)
-  * Having (only batch?)
-* Joins
-  * Inner equi joins (only batch)
-  * Outer equi joins (only batch)
-  * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+
+
+  
+
+  Operators
+  Description
+
+  
+  
+   
+   Scan / Select / As
+   
+{% highlight sql %}
+SELECT * FROM Orders
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+  
+   
+
+  Where / Filter
+  
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+  
+
+
+  User Defined Functions (UDF)
+  
+  SQL queries can refer to UDFs provided that they are registered 
in the `TableEnvironment`.
--- End diff --

Markdown highlighting does not seem to work here. We should also add a link 
to the UDF docs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4046: [FLINK-6749] [table] Table API / SQL Docs: SQL Pag...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/4046#discussion_r120751070
  
--- Diff: docs/dev/table/sql.md ---
@@ -22,20 +22,22 @@ specific language governing permissions and limitations
 under the License.
 -->
 
-SQL queries are specified using the `sql()` method of the 
`TableEnvironment`. The method returns the result of the SQL query as a `Table` 
which can be converted into a `DataSet` or `DataStream`, used in subsequent 
Table API queries, or written to a `TableSink` (see [Writing Tables to External 
Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries 
using the `sql()` method of the `TableEnvironment`. The method returns the 
result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / 
Table API queries, be converted into a `DataSet` or `DataStream`, or written to 
a `TableSink` (see [Writing Tables to External 
Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can 
seamlessly mixed and are holistically optimized and translated into a single 
program.
--- End diff --

I would not mention the translation into DataStream and DataSet programs 
(see another comment below).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-5486) Lack of synchronization in BucketingSink#handleRestoredBucketState()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822346#comment-15822346
 ] 

Ted Yu edited comment on FLINK-5486 at 6/7/17 9:47 PM:
---

Lock on State.bucketStates should be held in the following method:
{code}
  private void handleRestoredBucketState(State restoredState) {
Preconditions.checkNotNull(restoredState);

for (BucketState bucketState : restoredState.bucketStates.values()) {
{code}


was (Author: yuzhih...@gmail.com):
Lock on State.bucketStates should be held in the following method:

{code}
  private void handleRestoredBucketState(State restoredState) {
Preconditions.checkNotNull(restoredState);

for (BucketState bucketState : restoredState.bucketStates.values()) {
{code}

> Lack of synchronization in BucketingSink#handleRestoredBucketState()
> 
>
> Key: FLINK-5486
> URL: https://issues.apache.org/jira/browse/FLINK-5486
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Ted Yu
>
> Here is related code:
> {code}
>   
> handlePendingFilesForPreviousCheckpoints(bucketState.pendingFilesPerCheckpoint);
>   synchronized (bucketState.pendingFilesPerCheckpoint) {
> bucketState.pendingFilesPerCheckpoint.clear();
>   }
> {code}
> The handlePendingFilesForPreviousCheckpoints() call should be enclosed inside 
> the synchronization block. Otherwise during the processing of 
> handlePendingFilesForPreviousCheckpoints(), some entries of the map may be 
> cleared.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-4534) Lack of synchronization in BucketingSink#restoreState()

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-4534:
--
Description: 
Iteration over state.bucketStates is protected by synchronization in other 
methods, except for the following in restoreState():
{code}
for (BucketState bucketState : state.bucketStates.values()) {
{code}
and following in close():
{code}
for (Map.Entry> entry : 
state.bucketStates.entrySet()) {
  closeCurrentPartFile(entry.getValue());
{code}
w.r.t. bucketState.pendingFilesPerCheckpoint , there is similar issue starting 
line 752:
{code}
  Set pastCheckpointIds = 
bucketState.pendingFilesPerCheckpoint.keySet();
  LOG.debug("Moving pending files to final location on restore.");
  for (Long pastCheckpointId : pastCheckpointIds) {
{code}

  was:
Iteration over state.bucketStates is protected by synchronization in other 
methods, except for the following in restoreState():
{code}
for (BucketState bucketState : state.bucketStates.values()) {
{code}

and following in close():
{code}
for (Map.Entry> entry : 
state.bucketStates.entrySet()) {
  closeCurrentPartFile(entry.getValue());
{code}
w.r.t. bucketState.pendingFilesPerCheckpoint , there is similar issue starting 
line 752:
{code}
  Set pastCheckpointIds = 
bucketState.pendingFilesPerCheckpoint.keySet();
  LOG.debug("Moving pending files to final location on restore.");
  for (Long pastCheckpointId : pastCheckpointIds) {
{code}


> Lack of synchronization in BucketingSink#restoreState()
> ---
>
> Key: FLINK-4534
> URL: https://issues.apache.org/jira/browse/FLINK-4534
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming Connectors
>Reporter: Ted Yu
>
> Iteration over state.bucketStates is protected by synchronization in other 
> methods, except for the following in restoreState():
> {code}
> for (BucketState bucketState : state.bucketStates.values()) {
> {code}
> and following in close():
> {code}
> for (Map.Entry> entry : 
> state.bucketStates.entrySet()) {
>   closeCurrentPartFile(entry.getValue());
> {code}
> w.r.t. bucketState.pendingFilesPerCheckpoint , there is similar issue 
> starting line 752:
> {code}
>   Set pastCheckpointIds = 
> bucketState.pendingFilesPerCheckpoint.keySet();
>   LOG.debug("Moving pending files to final location on restore.");
>   for (Long pastCheckpointId : pastCheckpointIds) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5488) yarnClient should be closed in AbstractYarnClusterDescriptor for error conditions

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041664#comment-16041664
 ] 

Ted Yu commented on FLINK-5488:
---

Please also fix the following in deployInternal():
{code}
ClusterResourceDescription freeClusterMem = 
getCurrentFreeClusterResources(yarnClient);
{code}

> yarnClient should be closed in AbstractYarnClusterDescriptor for error 
> conditions
> -
>
> Key: FLINK-5488
> URL: https://issues.apache.org/jira/browse/FLINK-5488
> Project: Flink
>  Issue Type: Bug
>  Components: YARN
>Reporter: Ted Yu
>Assignee: Fang Yong
>
> Here is one example:
> {code}
> if(jobManagerMemoryMb > maxRes.getMemory() ) {
>   failSessionDuringDeployment(yarnClient, yarnApplication);
>   throw new YarnDeploymentException("The cluster does not have the 
> requested resources for the JobManager available!\n"
> + "Maximum Memory: " + maxRes.getMemory() + "MB Requested: " + 
> jobManagerMemoryMb + "MB. " + NOTE);
> }
> {code}
> yarnClient implements Closeable.
> It should be closed in situations where exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6848) Extend the managed state docs with a Scala example

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041663#comment-16041663
 ] 

ASF GitHub Bot commented on FLINK-6848:
---

Github user Fokko commented on the issue:

https://github.com/apache/flink/pull/4072
  
Hi @tzulitai,

I fully agree. Give me some time to work on the other Scala examples, I 
need to make sure that they are working properly. I'll wrap it up this week. 
I'll update the commit and rebase with master.

Cheers, Fokko


> Extend the managed state docs with a Scala example
> --
>
> Key: FLINK-6848
> URL: https://issues.apache.org/jira/browse/FLINK-6848
> Project: Flink
>  Issue Type: Bug
>Reporter: Fokko Driesprong
>
> Hi all,
> It would be nice to add a Scala example code snippet in the Managed state 
> docs. This makes it a bit easier to start using managed state in Scala. The 
> code is tested and works.
> Kind regards,
> Fokko



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4072: [FLINK-6848] Update managed state docs

[Fokko]

Github user Fokko commented on the issue:

https://github.com/apache/flink/pull/4072
  
Hi @tzulitai,

I fully agree. Give me some time to work on the other Scala examples, I 
need to make sure that they are working properly. I'll wrap it up this week. 
I'll update the commit and rebase with master.

Cheers, Fokko


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041657#comment-16041657
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Apart from the ImportOrder and AvoidStartImport modules everything that we 
could configure via IntelliJ Code Style seems to be covered.


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Apart from the ImportOrder and AvoidStartImport modules everything that we 
could configure via IntelliJ Code Style seems to be covered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041558#comment-16041558
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Here's a list of the currently supported modules btw.:
```
EmptyLineSeparator
FileTabCharacter
Indentation
LeftCurly
LineLength
NeedBraces
NoWhitespacesBefore
WhitespaceAfter
WhitespaceAround
```


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Here's a list of the currently supported modules btw.:
```
EmptyLineSeparator
FileTabCharacter
Indentation
LeftCurly
LineLength
NeedBraces
NoWhitespacesBefore
WhitespaceAfter
WhitespaceAround
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041551#comment-16041551
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
I don't know a lot about the intellij code style config; can we only define 
subset that is imported without affecting the rest?

Btw., I'm looking into contributing to the checkstyle plugin to add support 
for the rules that we need.


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
I don't know a lot about the intellij code style config; can we only define 
subset that is imported without affecting the rest?

Btw., I'm looking into contributing to the checkstyle plugin to add support 
for the rules that we need.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041533#comment-16041533
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/4086
  
Okay, second best may be to create an IntelliJ Code Style configuration for 
developers to import.


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/4086
  
Okay, second best may be to create an IntelliJ Code Style configuration for 
developers to import.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041521#comment-16041521
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Only a (small) subset of checkstyle modules can be moduled, which sadly 
doesn't include the `ImportOrder` module.


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Only a (small) subset of checkstyle modules can be moduled, which sadly 
doesn't include the `ImportOrder` module.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-6866) ClosureCleaner.clean fails for scala's JavaConverters wrapper classes

[SmedbergM (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SmedbergM updated FLINK-6866:
-
Description: 
MWE:

```
import scala.collection.JavaConverters._
import org.apache.flink.api.java.ClosureCleaner

object SerializationFailureMWE extends App {
  val m4j: java.util.Map[String,String] = new java.util.HashMap
  m4j.put("key1", "value1")

  val m: java.util.Map[String,String] = Map(
"key1" -> "value1"
  ).asJava

  println("Cleaning native Java map")
  ClosureCleaner.clean(m4j, true)

  println("Cleaning map converted by JavaConverters")
  ClosureCleaner.clean(m, true)
}
```

Program output:
```
Cleaning native Java map
Cleaning map converted by JavaConverters
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: 
{key1=value1} is not serializable. The object probably contains or references 
non serializable fields.
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
at 
SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at 
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at 
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at SerializationFailureMWE$.main(SerializationFailureMWE.scala:5)
at SerializationFailureMWE.main(SerializationFailureMWE.scala)
Caused by: java.io.NotSerializableException: 
scala.collection.convert.Wrappers$MapWrapper
...
```

  was:
MWE:

```
import scala.collection.JavaConverters._
import org.apache.flink.api.java.ClosureCleaner

object SerializationFailureMWE extends App {
  val m4j: java.util.Map[String,String] = new java.util.HashMap
  m4j.put("key1", "value1")

  val m: java.util.Map[String,String] = Map(
"key1" -> "value1"
  ).asJava

  println("Cleaning native Java map")

  ClosureCleaner.clean(m4j, true)

  println("Cleaning map converted by JavaConverters")

  ClosureCleaner.clean(m, true)
}
```

Program output:
```
Cleaning native Java map
Cleaning map converted by JavaConverters
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: 
{key1=value1} is not serializable. The object probably contains or references 
non serializable fields.
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
at 
SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at 
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at 
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at SerializationFailureMWE$.main(SerializationFailureMWE.scala:5)
at SerializationFailureMWE.main(SerializationFailureMWE.scala)
Caused by: java.io.NotSerializableException: 
scala.collection.convert.Wrappers$MapWrapper
...
```


> ClosureCleaner.clean fails for scala's JavaConverters wrapper classes
> -
>
> Key: FLINK-6866
> URL: https://issues.apache.org/jira/browse/FLINK-6866
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.2.0
>Reporter: SmedbergM
>
> MWE:
> ```
> import scala.collection.JavaConverters._
> import org.apache.flink.api.java.ClosureCleaner
> object SerializationFailureMWE extends App {
>   val m4j: java.util.Map[String,String] = new java.util.HashMap
>   m4j.put("key1", "value1")
>   val m: java.util.Map[String,String] = Map(
> "key1" -> "value1"
>   ).asJava
>   println("Cleaning native Java map")
>   ClosureCleaner.clean(m4j, true)
>   println("Cleaning map converted by JavaConverters")
>   ClosureCleaner.clean(m, true)
> }
> ```
> Program output:
> ```
> Cleaning native Java map
> Cleaning map converted by JavaConverters
> Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: {key1=value1} is not 
> serializable. The object probably contains or references non serializable 
> fields.
>   at 
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
>   at 
> SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
>   at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>   at 

[jira] [Updated] (FLINK-6866) ClosureCleaner.clean fails for scala's JavaConverters wrapper classes

[SmedbergM (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SmedbergM updated FLINK-6866:
-
Description: 
MWE:

```
import scala.collection.JavaConverters._
import org.apache.flink.api.java.ClosureCleaner

object SerializationFailureMWE extends App {
  val m4j: java.util.Map[String,String] = new java.util.HashMap
  m4j.put("key1", "value1")

  val m: java.util.Map[String,String] = Map(
"key1" -> "value1"
  ).asJava

  println("Cleaning native Java map")

  ClosureCleaner.clean(m4j, true)

  println("Cleaning map converted by JavaConverters")

  ClosureCleaner.clean(m, true)
}
```

Program output:
```
Cleaning native Java map
Cleaning map converted by JavaConverters
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: 
{key1=value1} is not serializable. The object probably contains or references 
non serializable fields.
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
at 
SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at 
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at 
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at SerializationFailureMWE$.main(SerializationFailureMWE.scala:5)
at SerializationFailureMWE.main(SerializationFailureMWE.scala)
Caused by: java.io.NotSerializableException: 
scala.collection.convert.Wrappers$MapWrapper
...
```

  was:
MWE:

```
import scala.collection.JavaConverters._
import org.apache.flink.api.java.ClosureCleaner

object SerializationFailureMWE extends App {
  val m4j: java.util.Map[String,String] = new java.util.HashMap
  m4j.put("key1", "value1")

  val m: java.util.Map[String,String] = Map(
"key1" -> "value1"
  ).asJava

  println("Cleaning native Java map")
  ClosureCleaner.clean(m4j, true)

  println("Cleaning map converted by JavaConverters")
  ClosureCleaner.clean(m, true)
}
```

Program output:
```
Cleaning native Java map
Cleaning map converted by JavaConverters
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: 
{key1=value1} is not serializable. The object probably contains or references 
non serializable fields.
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
at 
SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at 
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at 
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at SerializationFailureMWE$.main(SerializationFailureMWE.scala:5)
at SerializationFailureMWE.main(SerializationFailureMWE.scala)
Caused by: java.io.NotSerializableException: 
scala.collection.convert.Wrappers$MapWrapper
...
```


> ClosureCleaner.clean fails for scala's JavaConverters wrapper classes
> -
>
> Key: FLINK-6866
> URL: https://issues.apache.org/jira/browse/FLINK-6866
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.2.0
>Reporter: SmedbergM
>
> MWE:
> ```
> import scala.collection.JavaConverters._
> import org.apache.flink.api.java.ClosureCleaner
> object SerializationFailureMWE extends App {
>   val m4j: java.util.Map[String,String] = new java.util.HashMap
>   m4j.put("key1", "value1")
>   val m: java.util.Map[String,String] = Map(
> "key1" -> "value1"
>   ).asJava
>   println("Cleaning native Java map")
>   ClosureCleaner.clean(m4j, true)
>   println("Cleaning map converted by JavaConverters")
>   ClosureCleaner.clean(m, true)
> }
> ```
> Program output:
> ```
> Cleaning native Java map
> Cleaning map converted by JavaConverters
> Exception in thread "main" 
> org.apache.flink.api.common.InvalidProgramException: {key1=value1} is not 
> serializable. The object probably contains or references non serializable 
> fields.
>   at 
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
>   at 
> SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
>   at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
>   at 

[jira] [Created] (FLINK-6866) ClosureCleaner.clean fails for scala's JavaConverters wrapper classes

[SmedbergM (JIRA)]

SmedbergM created FLINK-6866:


 Summary: ClosureCleaner.clean fails for scala's JavaConverters 
wrapper classes
 Key: FLINK-6866
 URL: https://issues.apache.org/jira/browse/FLINK-6866
 Project: Flink
  Issue Type: Bug
  Components: DataStream API
Affects Versions: 1.2.0
Reporter: SmedbergM


MWE:

```
import scala.collection.JavaConverters._
import org.apache.flink.api.java.ClosureCleaner

object SerializationFailureMWE extends App {
  val m4j: java.util.Map[String,String] = new java.util.HashMap
  m4j.put("key1", "value1")

  val m: java.util.Map[String,String] = Map(
"key1" -> "value1"
  ).asJava

  println("Cleaning native Java map")
  ClosureCleaner.clean(m4j, true)

  println("Cleaning map converted by JavaConverters")
  ClosureCleaner.clean(m, true)
}
```

Program output:
```
Cleaning native Java map
Cleaning map converted by JavaConverters
Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: 
{key1=value1} is not serializable. The object probably contains or references 
non serializable fields.
at 
org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:100)
at 
SerializationFailureMWE$delayedInit$body.apply(SerializationFailureMWE.scala:17)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at 
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at 
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at SerializationFailureMWE$.main(SerializationFailureMWE.scala:5)
at SerializationFailureMWE.main(SerializationFailureMWE.scala)
Caused by: java.io.NotSerializableException: 
scala.collection.convert.Wrappers$MapWrapper
...
```



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6815) Javadocs don't work anymore in Flink 1.4-SNAPSHOT

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-6815.
---
   Resolution: Fixed
Fix Version/s: 1.4.0
   1.3.1

1.4: http://git-wip-us.apache.org/repos/asf/flink/commit/e13a7f80
1.3: http://git-wip-us.apache.org/repos/asf/flink/commit/f72eff7f

> Javadocs don't work anymore in Flink 1.4-SNAPSHOT
> -
>
> Key: FLINK-6815
> URL: https://issues.apache.org/jira/browse/FLINK-6815
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Affects Versions: 1.4.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
> Fix For: 1.3.1, 1.4.0
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/scala/KeyedStream.html
>  results in a 404 error.
> The problem 
> (https://ci.apache.org/builders/flink-docs-master/builds/731/steps/Java%20&%20Scala%20docs/logs/stdio)
>  is the following:
> {code}
> [ERROR] Failed to execute goal 
> net.alchim31.maven:scala-maven-plugin:3.2.2:compile (doc) on project 
> flink-annotations: wrap: 
> org.apache.maven.artifact.resolver.ArtifactNotFoundException: Could not find 
> artifact com.typesafe.genjavadoc:genjavadoc-plugin_2.10.6:jar:0.8 in central 
> (https://repo.maven.apache.org/maven2)
> {code}
> I think the problem is that we upgraded the scala version to 2.10.6, but the 
> plugin doesn't have version 0.8 for that scala version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041329#comment-16041329
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Something the import did in fact set was spaces around operators. It may be 
that not all checkstyle rules are properly imported.


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
Something the import did in fact set was spaces around operators. It may be 
that not all checkstyle rules are properly imported.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041302#comment-16041302
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
hmmi just retried it and it didn't work this time. Let me investigate a 
bit.


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
hmmi just retried it and it didn't work this time. Let me investigate a 
bit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4086: [FLINK-6865] Update checkstyle documentation

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
yes, it's a good idea to "flip" the checkstyle names.

I only checked a few things, but importing the checkstyle configuration, 
with the Checkstyle-IDEA plugin enabled, at the very least correctly configured 
the import order. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6865) Expand checkstyle docs to include import in intellij

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041263#comment-16041263
 ] 

ASF GitHub Bot commented on FLINK-6865:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4086
  
yes, it's a good idea to "flip" the checkstyle names.

I only checked a few things, but importing the checkstyle configuration, 
with the Checkstyle-IDEA plugin enabled, at the very least correctly configured 
the import order. 


> Expand checkstyle docs to include import in intellij
> 
>
> Key: FLINK-6865
> URL: https://issues.apache.org/jira/browse/FLINK-6865
> Project: Flink
>  Issue Type: Improvement
>  Components: Checkstyle
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.4.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6830) Add ITTests for savepoint migration from 1.3

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai resolved FLINK-6830.

   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed for master via 3792be4b5f80826d5dbf51c0517e8c00847472ca.
Fixed for 1.3.1 via d4a646a035366918a100f64428c471464870b8d0.

> Add ITTests for savepoint migration from 1.3
> 
>
> Key: FLINK-6830
> URL: https://issues.apache.org/jira/browse/FLINK-6830
> Project: Flink
>  Issue Type: Test
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.3.1, 1.4.0
>
>
> Already with FLINK-6763 and FLINK-6764 we'll need to change the serialization 
> formats between 1.3.0 and 1.3.x.
> We probably should add the stateful job migration ITCases for restoring from 
> Flink 1.3.x now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6830) Add ITTests for savepoint migration from 1.3

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai closed FLINK-6830.
--

> Add ITTests for savepoint migration from 1.3
> 
>
> Key: FLINK-6830
> URL: https://issues.apache.org/jira/browse/FLINK-6830
> Project: Flink
>  Issue Type: Test
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.3.1, 1.4.0
>
>
> Already with FLINK-6763 and FLINK-6764 we'll need to change the serialization 
> formats between 1.3.0 and 1.3.x.
> We probably should add the stateful job migration ITCases for restoring from 
> Flink 1.3.x now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6853) Migrating from Flink 1.1 fails for FlinkCEP

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai resolved FLINK-6853.

   Resolution: Fixed
Fix Version/s: 1.4.0
   1.3.1

Fixed for master via 4895472ba2279d2982a45279d7be76bf3dfd8768.
Fixed for 1.3.1 via 1d89dd06c1f9b09420ad3ff095d0842b4a951938.

> Migrating from Flink 1.1 fails for FlinkCEP
> ---
>
> Key: FLINK-6853
> URL: https://issues.apache.org/jira/browse/FLINK-6853
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.3.1, 1.4.0
>
>
> Migrating from Flink 1.1 fails for CEP, since in 1.1, the legacy 
> {{MultiplexingStreamRecordSerializer}} is used for stream  elements in the 
> serialized priority queue (via the {{PriorityQueueSerializer}}).
> In newer versions, the {{StreamElementSerializer}} is used instead. For this 
> to work, we need to implement the compatibility methods for 
> {{StreamElementSerializer}} such that it is also compatible with 
> configuration snapshots taken from the {{MultiplexingStreamRecordSerializer}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6808) Stream join fails when checkpointing is enabled

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai resolved FLINK-6808.

Resolution: Fixed

Fixed for 1.3.1 via f74caf7062b1cc23a704f8f8b8171be430b60807.
Fixed for master via 539787b21822eb839d0408a989cd541450bd08d2.

> Stream join fails when checkpointing is enabled
> ---
>
> Key: FLINK-6808
> URL: https://issues.apache.org/jira/browse/FLINK-6808
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.3.0
>Reporter: Francisco Rosa
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
>  Labels: flink-rel-1.3.1-blockers
> Fix For: 1.3.1
>
>
> The combination of joining streams and checkpointing fails in 1.3.0. It used 
> to work with the previous 1.2 version. Code example for failure:
> {code:title=Example|borderStyle=solid}
> public static void main(String[] args) throws Exception {
> final StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
> // enable checkpoints
> env.enableCheckpointing(5000);
> // create two streams
> DataStreamSource one = env.generateSequence(0, 5000);
> DataStreamSource two = env.generateSequence(2000, 15000);
> // process both, provide a delay to make sure checkpoint will happen
> DataStream oneProcessed = one.
> map(oneValue -> {
> Thread.sleep(1000);
> return "val-" + oneValue;
> });
> DataStream twoProcessed = two.
> map(oneValue -> {
> Thread.sleep(1000);
> return "val-" + oneValue;
> });
> // join the two streams, join on string match
> DataStream joinedStreams = oneProcessed.
> join(twoProcessed).
> where(String::toString).
> equalTo(String::toString).
> window(TumblingProcessingTimeWindows.of(Time.seconds(5))).
> apply(new JoinFunction() {
> @Override
> public String join(String oneValue, String twoValue) {
> // nothing really relevant, just concatenate string
> return oneValue + "+" + twoValue;
> }
> });
> // output results
> joinedStreams.print();
> env.execute("Issue with stream join and checkpoints");
> }
> {code}
> Stack trace:
> {noformat}
> java.lang.Exception: Could not perform checkpoint 1 for operator 
> TriggerWindow(TumblingProcessingTimeWindows(5000), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@3769cce0},
>  ProcessingTimeTrigger(), WindowedStream.apply(CoGroupedStreams.java:300)) -> 
> Sink: Unnamed (1/1).
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:550)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:378)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:281)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:183)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:213)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:262)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Could not complete snapshot 1 for operator 
> TriggerWindow(TumblingProcessingTimeWindows(5000), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@3769cce0},
>  ProcessingTimeTrigger(), WindowedStream.apply(CoGroupedStreams.java:300)) -> 
> Sink: Unnamed (1/1).
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1089)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:653)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:589)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.ja

[jira] [Closed] (FLINK-6853) Migrating from Flink 1.1 fails for FlinkCEP

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai closed FLINK-6853.
--

> Migrating from Flink 1.1 fails for FlinkCEP
> ---
>
> Key: FLINK-6853
> URL: https://issues.apache.org/jira/browse/FLINK-6853
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.3.1, 1.4.0
>
>
> Migrating from Flink 1.1 fails for CEP, since in 1.1, the legacy 
> {{MultiplexingStreamRecordSerializer}} is used for stream  elements in the 
> serialized priority queue (via the {{PriorityQueueSerializer}}).
> In newer versions, the {{StreamElementSerializer}} is used instead. For this 
> to work, we need to implement the compatibility methods for 
> {{StreamElementSerializer}} such that it is also compatible with 
> configuration snapshots taken from the {{MultiplexingStreamRecordSerializer}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-6844) TraversableSerializer should implement compatibility methods

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai closed FLINK-6844.
--

> TraversableSerializer should implement compatibility methods
> 
>
> Key: FLINK-6844
> URL: https://issues.apache.org/jira/browse/FLINK-6844
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
>  Labels: flink-rel-1.3.1-blockers
> Fix For: 1.3.1, 1.4.0
>
>
> The {{TraversableSerializer}} may be used as a serializer for managed state 
> and takes part in checkpointing, therefore should implement the compatibility 
> methods.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-6844) TraversableSerializer should implement compatibility methods

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai resolved FLINK-6844.

   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed for 1.3 via e1e207c898ed436df656d01364cf0e5fa818b730.
Fixed for master via c11d5ed5388a5a30ca4ea0c5ac68e22e5989cb54.

> TraversableSerializer should implement compatibility methods
> 
>
> Key: FLINK-6844
> URL: https://issues.apache.org/jira/browse/FLINK-6844
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
>  Labels: flink-rel-1.3.1-blockers
> Fix For: 1.3.1, 1.4.0
>
>
> The {{TraversableSerializer}} may be used as a serializer for managed state 
> and takes part in checkpointing, therefore should implement the compatibility 
> methods.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6841) using TableSourceTable for both Stream and Batch OR remove useless import

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041179#comment-16041179
 ] 

ASF GitHub Bot commented on FLINK-6841:
---

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/4061
  
Hi @twalthr I think `DefinedProctimeAttribute` can be implemented by a 
batch table source. If we do not want a batch table source can implements 
`DefinedProctimeAttribute `, we should add move `DefinedProctimeAttribute` to 
`StreamTableSourceTable`. Otherwise, I feel that `StreamTableSourceTable` can 
be deleted. What do you think?


> using TableSourceTable for both Stream and Batch OR remove useless import
> -
>
> Key: FLINK-6841
> URL: https://issues.apache.org/jira/browse/FLINK-6841
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.4.0
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> 1. {{StreamTableSourceTable}} exist useless import of {{TableException}}
> 2. {{StreamTableSourceTable}} only override {{getRowType}} of  
> {{FlinkTable}}, I think we can override the method in {{TableSourceTable}}, 
> If so we can using {{TableSourceTable}} for both {{Stream}} and {{Batch}}.
> What do you think? [~fhueske] [~twalthr]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #4061: [FLINK-6841][table]Using TableSourceTable for both Stream...

[sunjincheng121]

Github user sunjincheng121 commented on the issue:

https://github.com/apache/flink/pull/4061
  
Hi @twalthr I think `DefinedProctimeAttribute` can be implemented by a 
batch table source. If we do not want a batch table source can implements 
`DefinedProctimeAttribute `, we should add move `DefinedProctimeAttribute` to 
`StreamTableSourceTable`. Otherwise, I feel that `StreamTableSourceTable` can 
be deleted. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6830) Add ITTests for savepoint migration from 1.3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041169#comment-16041169
 ] 

ASF GitHub Bot commented on FLINK-6830:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4059


> Add ITTests for savepoint migration from 1.3
> 
>
> Key: FLINK-6830
> URL: https://issues.apache.org/jira/browse/FLINK-6830
> Project: Flink
>  Issue Type: Test
>  Components: State Backends, Checkpointing
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.3.1
>
>
> Already with FLINK-6763 and FLINK-6764 we'll need to change the serialization 
> formats between 1.3.0 and 1.3.x.
> We probably should add the stateful job migration ITCases for restoring from 
> Flink 1.3.x now.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6808) Stream join fails when checkpointing is enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041168#comment-16041168
 ] 

ASF GitHub Bot commented on FLINK-6808:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4052


> Stream join fails when checkpointing is enabled
> ---
>
> Key: FLINK-6808
> URL: https://issues.apache.org/jira/browse/FLINK-6808
> Project: Flink
>  Issue Type: Bug
>  Components: DataStream API
>Affects Versions: 1.3.0
>Reporter: Francisco Rosa
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
>  Labels: flink-rel-1.3.1-blockers
> Fix For: 1.3.1
>
>
> The combination of joining streams and checkpointing fails in 1.3.0. It used 
> to work with the previous 1.2 version. Code example for failure:
> {code:title=Example|borderStyle=solid}
> public static void main(String[] args) throws Exception {
> final StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
> // enable checkpoints
> env.enableCheckpointing(5000);
> // create two streams
> DataStreamSource one = env.generateSequence(0, 5000);
> DataStreamSource two = env.generateSequence(2000, 15000);
> // process both, provide a delay to make sure checkpoint will happen
> DataStream oneProcessed = one.
> map(oneValue -> {
> Thread.sleep(1000);
> return "val-" + oneValue;
> });
> DataStream twoProcessed = two.
> map(oneValue -> {
> Thread.sleep(1000);
> return "val-" + oneValue;
> });
> // join the two streams, join on string match
> DataStream joinedStreams = oneProcessed.
> join(twoProcessed).
> where(String::toString).
> equalTo(String::toString).
> window(TumblingProcessingTimeWindows.of(Time.seconds(5))).
> apply(new JoinFunction() {
> @Override
> public String join(String oneValue, String twoValue) {
> // nothing really relevant, just concatenate string
> return oneValue + "+" + twoValue;
> }
> });
> // output results
> joinedStreams.print();
> env.execute("Issue with stream join and checkpoints");
> }
> {code}
> Stack trace:
> {noformat}
> java.lang.Exception: Could not perform checkpoint 1 for operator 
> TriggerWindow(TumblingProcessingTimeWindows(5000), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@3769cce0},
>  ProcessingTimeTrigger(), WindowedStream.apply(CoGroupedStreams.java:300)) -> 
> Sink: Unnamed (1/1).
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:550)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(BarrierBuffer.java:378)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:281)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:183)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:213)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:262)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Could not complete snapshot 1 for operator 
> TriggerWindow(TumblingProcessingTimeWindows(5000), 
> ListStateDescriptor{serializer=org.apache.flink.api.common.typeutils.base.ListSerializer@3769cce0},
>  ProcessingTimeTrigger(), WindowedStream.apply(CoGroupedStreams.java:300)) -> 
> Sink: Unnamed (1/1).
>   at 
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:406)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1157)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1089)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:653)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:589)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:542)

[jira] [Commented] (FLINK-6853) Migrating from Flink 1.1 fails for FlinkCEP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041171#comment-16041171
 ] 

ASF GitHub Bot commented on FLINK-6853:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4079


> Migrating from Flink 1.1 fails for FlinkCEP
> ---
>
> Key: FLINK-6853
> URL: https://issues.apache.org/jira/browse/FLINK-6853
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>
> Migrating from Flink 1.1 fails for CEP, since in 1.1, the legacy 
> {{MultiplexingStreamRecordSerializer}} is used for stream  elements in the 
> serialized priority queue (via the {{PriorityQueueSerializer}}).
> In newer versions, the {{StreamElementSerializer}} is used instead. For this 
> to work, we need to implement the compatibility methods for 
> {{StreamElementSerializer}} such that it is also compatible with 
> configuration snapshots taken from the {{MultiplexingStreamRecordSerializer}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-6844) TraversableSerializer should implement compatibility methods

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041170#comment-16041170
 ] 

ASF GitHub Bot commented on FLINK-6844:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4081


> TraversableSerializer should implement compatibility methods
> 
>
> Key: FLINK-6844
> URL: https://issues.apache.org/jira/browse/FLINK-6844
> Project: Flink
>  Issue Type: Bug
>  Components: Type Serialization System
>Affects Versions: 1.3.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Blocker
>  Labels: flink-rel-1.3.1-blockers
> Fix For: 1.3.1
>
>
> The {{TraversableSerializer}} may be used as a serializer for managed state 
> and takes part in checkpointing, therefore should implement the compatibility 
> methods.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #4081: [FLINK-6844] [scala] Implement compatibility metho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4081


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---