[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/534 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user gyfora commented on the pull request: https://github.com/apache/flink/pull/534#issuecomment-86555612 Ok I will add that one --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/534#issuecomment-86554956 I think we need to refactor some parts, but this is a crucial fix. Tests are passing. +1 to merge after throwing the Exception Robert mentioned. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user gyfora commented on the pull request: https://github.com/apache/flink/pull/534#issuecomment-86554780 Now it seems that all these issues are fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user gyfora commented on the pull request: https://github.com/apache/flink/pull/534#issuecomment-86140841 Should we merge this anyway? The recordwriter issue seems to be independent from this, and this commit solves some other bugs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user gyfora commented on a diff in the pull request:
https://github.com/apache/flink/pull/534#discussion_r27115550
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/streamvertex/StreamVertex.java
---
@@ -283,16 +288,18 @@ public StreamConfig getConfig() {
*
* @param id
*/
- private void actOnBarrier(long id) {
- try {
- outputHandler.broadcastBarrier(id);
- // TODO checkpoint state here
- confirmBarrier(id);
- if (LOG.isDebugEnabled()) {
- LOG.debug("Superstep " + id + " processed: " +
StreamVertex.this);
+ private synchronized void actOnBarrier(long id) {
+ if (this.isRunning) {
+ try {
+ outputHandler.broadcastBarrier(id);
+ // TODO checkpoint state here
+ confirmBarrier(id);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Superstep " + id + "
processed: " + StreamVertex.this);
+ }
+ } catch (Exception e) {
+ // TODO:Figure this out properly
--- End diff --
but I am trying to sort all these out with ufuk
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user gyfora commented on a diff in the pull request:
https://github.com/apache/flink/pull/534#discussion_r27115257
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/streamvertex/StreamVertex.java
---
@@ -283,16 +288,18 @@ public StreamConfig getConfig() {
*
* @param id
*/
- private void actOnBarrier(long id) {
- try {
- outputHandler.broadcastBarrier(id);
- // TODO checkpoint state here
- confirmBarrier(id);
- if (LOG.isDebugEnabled()) {
- LOG.debug("Superstep " + id + " processed: " +
StreamVertex.this);
+ private synchronized void actOnBarrier(long id) {
+ if (this.isRunning) {
+ try {
+ outputHandler.broadcastBarrier(id);
+ // TODO checkpoint state here
+ confirmBarrier(id);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Superstep " + id + "
processed: " + StreamVertex.this);
+ }
+ } catch (Exception e) {
+ // TODO:Figure this out properly
--- End diff --
I had some weird exceptions after the streams finished, and in any case
this shouldnt affect the recovery logic
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user uce commented on the pull request: https://github.com/apache/flink/pull/534#issuecomment-85909585 Hey gyula, What is the problem with the stream writer? And how can I reproduce the record writer broadcast problem? If you give me some instructions to reproduce the problem, I could also look into it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/534#discussion_r27100141
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/streamvertex/StreamVertex.java
---
@@ -283,16 +288,18 @@ public StreamConfig getConfig() {
*
* @param id
*/
- private void actOnBarrier(long id) {
- try {
- outputHandler.broadcastBarrier(id);
- // TODO checkpoint state here
- confirmBarrier(id);
- if (LOG.isDebugEnabled()) {
- LOG.debug("Superstep " + id + " processed: " +
StreamVertex.this);
+ private synchronized void actOnBarrier(long id) {
+ if (this.isRunning) {
+ try {
+ outputHandler.broadcastBarrier(id);
+ // TODO checkpoint state here
+ confirmBarrier(id);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Superstep " + id + "
processed: " + StreamVertex.this);
+ }
+ } catch (Exception e) {
+ // TODO:Figure this out properly
--- End diff --
Why are you swallowing the exception here silently?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/534 [FLINK-1775] BarrierBuffer fix to avoid end of stream deadlock This fixes several corner-cases in the barrier buffer logic. Still two problems remain: - If the StreamRecordWriter is active (a thread is running to flush the ouputs every so many milliseconds) - On travis the RecordWriters seem to deadlock at the sources for some reason on broadcastEvent calls. (I cannot reproduce this locally) You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink FLINK-1775 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/534.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #534 commit 35d71c8b8461e689ec02e64ef1089adfe1f4d7cb Author: Gyula Fora Date: 2015-03-24T23:14:21Z [FLINK-1775] BarrierBuffer fix to avoid end of stream deadlock --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
Github user gyfora closed the pull request at: https://github.com/apache/flink/pull/526 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1775] BarrierBuffer fix to avoid end of...
GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/526 [FLINK-1775] BarrierBuffer fix to avoid end of stream deadlock You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink FLINK-1775 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #526 commit d5b5e1a1f87306e5a94258c58418ac8129615111 Author: Gyula Fora Date: 2015-03-23T22:29:55Z [FLINK-1775] BarrierBuffer fix to avoid end of stream deadlock --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
