[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981831#comment-15981831
 ] 

Ismaël Mejía commented on BEAM-1886:


Yes it is I removed the support for sinks from flink so this is ready.

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981579#comment-15981579
 ] 

Davor Bonaci commented on BEAM-1886:


Is this done? (I think so, but wanted to double-check before closing.)

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976963#comment-15976963
 ] 

ASF GitHub Bot commented on BEAM-1886:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2537


> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976964#comment-15976964
 ] 

ASF GitHub Bot commented on BEAM-1886:
--

Github user aljoscha closed the pull request at:

https://github.com/apache/beam/pull/2611


> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976722#comment-15976722
 ] 

ASF GitHub Bot commented on BEAM-1886:
--

GitHub user aljoscha opened a pull request:

https://github.com/apache/beam/pull/2611

[BEAM-1886] Remove TextIO override in Flink runner

This is a duplicate of #2537. The tests work locally but not on Jenkins, 
I'm trying to figure out why that might be.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/beam 
finish-pr-2537-remove-flink-source

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2611.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2611


commit 8f8ce303eb443bb8996ea76d5b979cb4b1000787
Author: JingsongLi 
Date:   2017-04-14T03:22:43Z

[BEAM-1886] Remove TextIO override in Flink runner




> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973158#comment-15973158
 ] 

Kenneth Knowles commented on BEAM-1886:
---

Yes, it does. That is the recent work from [~reuvenlax]. There is still a 
little churn around it, but if the override is removed here, then the Flink 
runner can just automatically receive any refactors that occur.

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969240#comment-15969240
 ] 

Aljoscha Krettek commented on BEAM-1886:


[~kenn] Does regular write now support unbounded inputs? If yes, we can also 
remove the special support for unbounded Flink sinks.

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968552#comment-15968552
 ] 

ASF GitHub Bot commented on BEAM-1886:
--

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2537

[BEAM-1886] Remove TextIO override in Flink runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1886

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2537


commit d2a037cabcde65dd3aced6ad1e09dbfa6909dc35
Author: JingsongLi 
Date:   2017-04-14T03:22:43Z

[BEAM-1886] Remove TextIO override in Flink runner




> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963232#comment-15963232
 ] 

Kenneth Knowles commented on BEAM-1886:
---

CC: [~reuvenlax] who is making changes in that area of the core SDK right now.

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963221#comment-15963221
 ] 

Aljoscha Krettek commented on BEAM-1886:


+1 I think we should remove these overrides. They are leftovers from earlier 
lack of support in Beam itself.

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)