date:20170809

[jira] [Commented] (FLINK-7372) Remove ActorGateway from JobGraph

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119651#comment-16119651
 ] 

ASF GitHub Bot commented on FLINK-7372:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4483#discussion_r132140409
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/client/JobClientActorTest.java
 ---
@@ -388,7 +391,9 @@ else if (message instanceof RegisterJobClient) {
testFuture.tell(Acknowledge.get(), 
getSelf());
}
}
-   else if (message instanceof RegisterTest) {
+   else if (message instanceof RequestBlobManagerPort$) {
--- End diff --

Yes because `RequestBlobManagerPort` is a case object.


> Remove ActorGateway from JobGraph
> -
>
> Key: FLINK-7372
> URL: https://issues.apache.org/jira/browse/FLINK-7372
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> As a preliminary step for easier Flip-6 integration we should try to decouple 
> as many components from the underlying RPC abstraction as possible. One of 
> these components is the {{JobGraph}} which has a dependency on 
> {{ActorGateway}} via its {{JobGraph#uploadUserJars}} method.
> I propose to get rid of the {{ActorGateway}} parameter and passing instead 
> the BlobServer's address as an {{InetSocketAddress}} instance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4483: [FLINK-7372] [JobGraph] Remove ActorGateway from JobGraph

2017-08-09 Thread tillrohrmann

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4483
  
Rebased this PR onto the latest master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4483: [FLINK-7372] [JobGraph] Remove ActorGateway from J...

2017-08-09 Thread tillrohrmann

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4483#discussion_r132140409
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/client/JobClientActorTest.java
 ---
@@ -388,7 +391,9 @@ else if (message instanceof RegisterJobClient) {
testFuture.tell(Acknowledge.get(), 
getSelf());
}
}
-   else if (message instanceof RegisterTest) {
+   else if (message instanceof RequestBlobManagerPort$) {
--- End diff --

Yes because `RequestBlobManagerPort` is a case object.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7383) Remove ConfigurationUtil

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119649#comment-16119649
 ] 

ASF GitHub Bot commented on FLINK-7383:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4489


> Remove ConfigurationUtil
> 
>
> Key: FLINK-7383
> URL: https://issues.apache.org/jira/browse/FLINK-7383
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Trivial
> Fix For: 1.4.0
>
>
> The {{ConfigurationUtil}} can be removed because it is no longer used and is 
> basically subsumed by Flink's {{ConfigOption}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7383) Remove ConfigurationUtil

2017-08-09 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-7383.

   Resolution: Fixed
Fix Version/s: 1.4.0

Removed via ff70cc3afd004aad4016f94485b95a3ced1639c9

> Remove ConfigurationUtil
> 
>
> Key: FLINK-7383
> URL: https://issues.apache.org/jira/browse/FLINK-7383
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Trivial
> Fix For: 1.4.0
>
>
> The {{ConfigurationUtil}} can be removed because it is no longer used and is 
> basically subsumed by Flink's {{ConfigOption}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7107) Start Yarn session via start-up script

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119648#comment-16119648
 ] 

ASF GitHub Bot commented on FLINK-7107:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4465


> Start Yarn session via start-up script
> --
>
> Key: FLINK-7107
> URL: https://issues.apache.org/jira/browse/FLINK-7107
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.4.0
>
>
> The {{yarn-session.sh}} shell script should be adapted to be able to also 
> start a Flip-6 Yarn session cluster ({{YarnSessionClusterEntrypoint}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7107) Start Yarn session via start-up script

2017-08-09 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-7107.

   Resolution: Fixed
Fix Version/s: 1.4.0

Added via a570aa5c0c7d9a18e6cdcb689dcad9ff173dc2ac

> Start Yarn session via start-up script
> --
>
> Key: FLINK-7107
> URL: https://issues.apache.org/jira/browse/FLINK-7107
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.4.0
>
>
> The {{yarn-session.sh}} shell script should be adapted to be able to also 
> start a Flip-6 Yarn session cluster ({{YarnSessionClusterEntrypoint}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4489: [FLINK-7383] Remove ConfigurationUtil

2017-08-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4489


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4465: [FLINK-7107] [flip6] Add option to start a Flip-6 ...

2017-08-09 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4465


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7383) Remove ConfigurationUtil

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119642#comment-16119642
 ] 

ASF GitHub Bot commented on FLINK-7383:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4489
  
Thanks for the review @aljoscha. Merging this PR.


> Remove ConfigurationUtil
> 
>
> Key: FLINK-7383
> URL: https://issues.apache.org/jira/browse/FLINK-7383
> Project: Flink
>  Issue Type: Improvement
>  Components: Configuration
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Trivial
>
> The {{ConfigurationUtil}} can be removed because it is no longer used and is 
> basically subsumed by Flink's {{ConfigOption}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4489: [FLINK-7383] Remove ConfigurationUtil

2017-08-09 Thread tillrohrmann

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4489
  
Thanks for the review @aljoscha. Merging this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7107) Start Yarn session via start-up script

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119640#comment-16119640
 ] 

ASF GitHub Bot commented on FLINK-7107:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4465
  
Thanks for the review @zentol. Merging this PR.


> Start Yarn session via start-up script
> --
>
> Key: FLINK-7107
> URL: https://issues.apache.org/jira/browse/FLINK-7107
> Project: Flink
>  Issue Type: Sub-task
>  Components: Startup Shell Scripts
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> The {{yarn-session.sh}} shell script should be adapted to be able to also 
> start a Flip-6 Yarn session cluster ({{YarnSessionClusterEntrypoint}}).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4465: [FLINK-7107] [flip6] Add option to start a Flip-6 Yarn se...

2017-08-09 Thread tillrohrmann

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4465
  
Thanks for the review @zentol. Merging this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-7396) Don't put multiple directories in HADOOP_CONF_DIR in config.sh

2017-08-09 Thread Aljoscha Krettek (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-7396:

Priority: Blocker  (was: Major)

> Don't put multiple directories in HADOOP_CONF_DIR in config.sh
> --
>
> Key: FLINK-7396
> URL: https://issues.apache.org/jira/browse/FLINK-7396
> Project: Flink
>  Issue Type: Bug
>  Components: Startup Shell Scripts
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> In config.sh we do this:
> {code}
> # Check if deprecated HADOOP_HOME is set.
> if [ -n "$HADOOP_HOME" ]; then
> # HADOOP_HOME is set. Check if its a Hadoop 1.x or 2.x HADOOP_HOME path
> if [ -d "$HADOOP_HOME/conf" ]; then
> # its a Hadoop 1.x
> HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/conf"
> fi
> if [ -d "$HADOOP_HOME/etc/hadoop" ]; then
> # Its Hadoop 2.2+
> HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
> fi
> fi
> {code}
> while our {{HadoopFileSystem}} actually only treats this paths as a single 
> path, not a colon-separated path: 
> https://github.com/apache/flink/blob/854b05376a459a6197e41e141bb28a9befe481ad/flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java#L236
> I also think that other tools don't assume multiple paths in there and at 
> least one user ran into the problem on their setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7396) Don't put multiple directories in HADOOP_CONF_DIR in config.sh

2017-08-09 Thread Aljoscha Krettek (JIRA)

Aljoscha Krettek created FLINK-7396:
---

 Summary: Don't put multiple directories in HADOOP_CONF_DIR in 
config.sh
 Key: FLINK-7396
 URL: https://issues.apache.org/jira/browse/FLINK-7396
 Project: Flink
  Issue Type: Bug
  Components: Startup Shell Scripts
Affects Versions: 1.3.2, 1.4.0
Reporter: Aljoscha Krettek
 Fix For: 1.4.0, 1.3.3


In config.sh we do this:
{code}
# Check if deprecated HADOOP_HOME is set.
if [ -n "$HADOOP_HOME" ]; then
# HADOOP_HOME is set. Check if its a Hadoop 1.x or 2.x HADOOP_HOME path
if [ -d "$HADOOP_HOME/conf" ]; then
# its a Hadoop 1.x
HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/conf"
fi
if [ -d "$HADOOP_HOME/etc/hadoop" ]; then
# Its Hadoop 2.2+
HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
fi
fi
{code}
while our {{HadoopFileSystem}} actually only treats this paths as a single 
path, not a colon-separated path: 
https://github.com/apache/flink/blob/854b05376a459a6197e41e141bb28a9befe481ad/flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java#L236

I also think that other tools don't assume multiple paths in there and at least 
one user ran into the problem on their setup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7026) Add shaded asm dependency

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119633#comment-16119633
 ] 

ASF GitHub Bot commented on FLINK-7026:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4494
  
@zentol I think it should be safe to remove that asm relocation (for 
Kinesis, as far as I'm aware of).


> Add shaded asm dependency
> -
>
> Key: FLINK-7026
> URL: https://issues.apache.org/jira/browse/FLINK-7026
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4494: [FLINK-7026] Introduce flink-shaded-asm-5

2017-08-09 Thread tzulitai

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4494
  
@zentol I think it should be safe to remove that asm relocation (for 
Kinesis, as far as I'm aware of).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7026) Add shaded asm dependency

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119631#comment-16119631
 ] 

ASF GitHub Bot commented on FLINK-7026:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4494
  
Thank your for the review @greghogan . I made 2 more changes to the 
kinesis/shaded-curator poms; they were including/relocating asm in their 
shade-plugin configuration even though neither appears to have a dependency on 
asm.


> Add shaded asm dependency
> -
>
> Key: FLINK-7026
> URL: https://issues.apache.org/jira/browse/FLINK-7026
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4494: [FLINK-7026] Introduce flink-shaded-asm-5

2017-08-09 Thread zentol

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/4494
  
Thank your for the review @greghogan . I made 2 more changes to the 
kinesis/shaded-curator poms; they were including/relocating asm in their 
shade-plugin configuration even though neither appears to have a dependency on 
asm.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-7363) add hashes and signatures to the download page

2017-08-09 Thread Nico Kruber (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nico Kruber resolved FLINK-7363.

Resolution: Fixed

Fixed by 
[ea50989ae5289fc386fdcc445d4c256be8111697|https://github.com/apache/flink-web/commit/ea50989ae5289fc386fdcc445d4c256be8111697]

> add hashes and signatures to the download page
> --
>
> Key: FLINK-7363
> URL: https://issues.apache.org/jira/browse/FLINK-7363
> Project: Flink
>  Issue Type: Improvement
>  Components: Project Website
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> As part of the releases, we also generate MD5 hashes and cryptographic 
> signatures but neither link to those nor do we explain which keys are valid 
> release-signing keys. This should be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-4565) Support for SQL IN operator

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119582#comment-16119582
 ] 

ASF GitHub Bot commented on FLINK-4565:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4493
  
Thanks @tedyu. I will merge this in my next batch.


> Support for SQL IN operator
> ---
>
> Key: FLINK-4565
> URL: https://issues.apache.org/jira/browse/FLINK-4565
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.4.0
>
>
> It seems that Flink SQL supports the uncorrelated sub-query IN operator. But 
> it should also be available in the Table API and tested.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4493: [FLINK-4565] [table] Support for SQL IN operator - move c...

2017-08-09 Thread twalthr

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4493
  
Thanks @tedyu. I will merge this in my next batch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4565) Support for SQL IN operator

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119576#comment-16119576
 ] 

ASF GitHub Bot commented on FLINK-4565:
---

Github user tedyu commented on the issue:

https://github.com/apache/flink/pull/4404
  
@fhueske
Can you take a look at #4493 ?


> Support for SQL IN operator
> ---
>
> Key: FLINK-4565
> URL: https://issues.apache.org/jira/browse/FLINK-4565
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.4.0
>
>
> It seems that Flink SQL supports the uncorrelated sub-query IN operator. But 
> it should also be available in the Table API and tested.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4404: [FLINK-4565] [table] Support for SQL IN operator

2017-08-09 Thread tedyu

Github user tedyu commented on the issue:

https://github.com/apache/flink/pull/4404
  
@fhueske
Can you take a look at #4493 ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7367) Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119575#comment-16119575
 ] 

ASF GitHub Bot commented on FLINK-7367:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4473
  
@bowenli86 no, they're provided via a `Properties` when creating a 
`FlinkKinesisProducer`


> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, 
> MaxConnections, RequestTimeout, etc)
> ---
>
> Key: FLINK-7367
> URL: https://issues.apache.org/jira/browse/FLINK-7367
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
> Fix For: 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying 
> KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS 
> doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html] 
> and [their sample on 
> github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties],
>  developers can set more to make the max use of KinesisProducer, and make it 
> fault-tolerant (e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> We need to parameterize FlinkKinesisProducer to pass in the above params, in 
> order to cater to our need



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4473: [FLINK-7367][kinesis connector] Parameterize more configs...

2017-08-09 Thread tzulitai

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/4473
  
@bowenli86 no, they're provided via a `Properties` when creating a 
`FlinkKinesisProducer`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6988) Add Apache Kafka 0.11 connector

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119556#comment-16119556
 ] 

ASF GitHub Bot commented on FLINK-6988:
---

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4239
  
I think there is no way we can to handle it in any different way then to 
increase the timeout to some very large value. Or is it?


> Add Apache Kafka 0.11 connector
> ---
>
> Key: FLINK-6988
> URL: https://issues.apache.org/jira/browse/FLINK-6988
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Affects Versions: 1.3.1
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>
> Kafka 0.11 (it will be released very soon) add supports for transactions. 
> Thanks to that, Flink might be able to implement Kafka sink supporting 
> "exactly-once" semantic. API changes and whole transactions support is 
> described in 
> [KIP-98|https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging].
> The goal is to mimic implementation of existing BucketingSink. New 
> FlinkKafkaProducer011 would 
> * upon creation begin transaction, store transaction identifiers into the 
> state and would write all incoming data to an output Kafka topic using that 
> transaction
> * on `snapshotState` call, it would flush the data and write in state 
> information that current transaction is pending to be committed
> * on `notifyCheckpointComplete` we would commit this pending transaction
> * in case of crash between `snapshotState` and `notifyCheckpointComplete` we 
> either abort this pending transaction (if not every participant successfully 
> saved the snapshot) or restore and commit it. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4239: [FLINK-6988] flink-connector-kafka-0.11 with exactly-once...

2017-08-09 Thread pnowojski

Github user pnowojski commented on the issue:

https://github.com/apache/flink/pull/4239
  
I think there is no way we can to handle it in any different way then to 
increase the timeout to some very large value. Or is it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7352) ExecutionGraphRestartTest timeouts

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119545#comment-16119545
 ] 

ASF GitHub Bot commented on FLINK-7352:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4501

[FLINK-7352] [tests] Stabilize ExecutionGraphRestartTest

## What is the purpose of the change

Introduce an explicit waiting for the deployment of tasks. This replaces 
the loose
ordering induced by Thread.sleep and fixes the race conditions caused by it.

## Brief change log

- Introduce `WaitForTasks` consumer which is given to the 
`SimpleAckingTaskManagerGateway`
- Using a single `SimpleAckingTaskManagerGateway` to receive all task 
submission calls

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
fixExecutionGraphRestartTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4501.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4501


commit 40cd0c860dd600ce2baa69b0f0ba8cf7a787ff63
Author: Till Rohrmann 
Date:   2017-08-09T07:57:56Z

[FLINK-7352] [tests] Stabilize ExecutionGraphRestartTest

Introduce an explicit waiting for the deployment of tasks. This replaces 
the loose
ordering induced by Thread.sleep and fixes the race conditions caused by it.




> ExecutionGraphRestartTest timeouts
> --
>
> Key: FLINK-7352
> URL: https://issues.apache.org/jira/browse/FLINK-7352
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, Tests
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
>
> Recently, I received timeouts from some tests in 
> {{ExecutionGraphRestartTest}} like this
> {code}
> Tests in error: 
>   ExecutionGraphRestartTest.testConcurrentLocalFailAndRestart:638 » Timeout
> {code}
> This particular instance is from 1.3.2 RC2 and stuck in 
> {{ExecutionGraphTestUtils#waitUntilDeployedAndSwitchToRunning()}} but I also 
> had instances stuck in {{ExecutionGraphTestUtils#waitUntilJobStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4501: [FLINK-7352] [tests] Stabilize ExecutionGraphResta...

2017-08-09 Thread tillrohrmann

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4501

[FLINK-7352] [tests] Stabilize ExecutionGraphRestartTest

## What is the purpose of the change

Introduce an explicit waiting for the deployment of tasks. This replaces 
the loose
ordering induced by Thread.sleep and fixes the race conditions caused by it.

## Brief change log

- Introduce `WaitForTasks` consumer which is given to the 
`SimpleAckingTaskManagerGateway`
- Using a single `SimpleAckingTaskManagerGateway` to receive all task 
submission calls

## Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
fixExecutionGraphRestartTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4501.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4501


commit 40cd0c860dd600ce2baa69b0f0ba8cf7a787ff63
Author: Till Rohrmann 
Date:   2017-08-09T07:57:56Z

[FLINK-7352] [tests] Stabilize ExecutionGraphRestartTest

Introduce an explicit waiting for the deployment of tasks. This replaces 
the loose
ordering induced by Thread.sleep and fixes the race conditions caused by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-7352) ExecutionGraphRestartTest timeouts

2017-08-09 Thread Till Rohrmann (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-7352:


Assignee: Till Rohrmann

> ExecutionGraphRestartTest timeouts
> --
>
> Key: FLINK-7352
> URL: https://issues.apache.org/jira/browse/FLINK-7352
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, Tests
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Assignee: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
>
> Recently, I received timeouts from some tests in 
> {{ExecutionGraphRestartTest}} like this
> {code}
> Tests in error: 
>   ExecutionGraphRestartTest.testConcurrentLocalFailAndRestart:638 » Timeout
> {code}
> This particular instance is from 1.3.2 RC2 and stuck in 
> {{ExecutionGraphTestUtils#waitUntilDeployedAndSwitchToRunning()}} but I also 
> had instances stuck in {{ExecutionGraphTestUtils#waitUntilJobStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4454: [hotfix][docs] Add section in docs about writing u...

2017-08-09 Thread pnowojski

Github user pnowojski commented on a diff in the pull request:

https://github.com/apache/flink/pull/4454#discussion_r132112682
  
--- Diff: docs/dev/testing.md ---
@@ -0,0 +1,189 @@
+---
+title: "Testing"
+nav-parent_id: dev
+nav-id: testing
+nav-pos: 99
+---
+
+
+This page briefly discusses how to test Flink application in the local 
environment.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Unit testing
+
+It is encouraged to test your classes with unit tests as much as possible. 
For example if one implement following `ReduceFunction`:
+
+~~~java
+public class SumReduce implements ReduceFunction {
+@Override
+public Long reduce(Long value1, Long value2) throws Exception {
+return value1 + value2;
+}
+}
+~~~
+
+it is very easy to unit test it with your favorite framework:
+
+~~~java
+public class SumReduceTest {
+@Test
+public void testSum() throws Exception {
+SumReduce sumReduce = new SumReduce();
+
+assertEquals(42L, sumReduce.reduce(40L, 2L));
+}
+}
+~~~
+
+Or in scala:
+
+~~~scala
+class SumReduce extends ReduceFunction[Long] {
+override def reduce(value1: java.lang.Long,
+value2: java.lang.Long): java.lang.Long = value1 + 
value2
+}
+~~~
+
+~~~scala
+class SumReduceTest extends FlatSpec with Matchers {
+"SumReduce" should "add values" in {
+val sumReduce: SumReduce = new SumReduce()
+sumReduce.reduce(40L, 2L) should be (42L)
+}
+}
+~~~
+
+## Integration testing
+
+You also can write integration tests that are executed against local Flink 
mini cluster.
+In order to do so add a test dependency `flink-test-utils`.
+
+~~~ xml
+
+  org.apache.flink
+  flink-test-utils{{site.scala_version_suffix}}
+  {{site.version}}
+
+~~~
--- End diff --

Ok, got it. Now it makes sense :)

I have changed this to `{% highlight xml %}` version.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-7395) NumBytesOut metric in RecordWriter call synchronized method

2017-08-09 Thread Chesnay Schepler (JIRA)

Chesnay Schepler created FLINK-7395:
---

 Summary: NumBytesOut metric in RecordWriter call synchronized 
method
 Key: FLINK-7395
 URL: https://issues.apache.org/jira/browse/FLINK-7395
 Project: Flink
  Issue Type: Bug
  Components: Metrics
Affects Versions: 1.3.2, 1.4.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Priority: Blocker
 Fix For: 1.4.0, 1.3.3






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6787) Job-/StoppableException should extend FlinkException

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119510#comment-16119510
 ] 

ASF GitHub Bot commented on FLINK-6787:
---

GitHub user zjureel opened a pull request:

https://github.com/apache/flink/pull/4500

[FLINK-6787] Fix Job-/StoppingException extend FlinkException

## What is the purpose of the change

Fix Job-/StoppingException extend FlinkException

## Brief change log

  - *Job-/StoppingException extend FlinkException instead of Exception*

## Verifying this change

*(Please pick either of the following options)*
This change is already covered by existing tests

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable / docs / 
JavaDocs / not documented)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjureel/flink FLINK-6787

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4500


commit dce23138931c48b494bacd6335573cb34fbea3c6
Author: zjureel 
Date:   2017-08-09T05:49:02Z

[FLINK-6787] Fix Job-/StoppableException extend FlinkException




> Job-/StoppableException should extend FlinkException
> 
>
> Key: FLINK-6787
> URL: https://issues.apache.org/jira/browse/FLINK-6787
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-6787) Job-/StoppableException should extend FlinkException

2017-08-09 Thread Fang Yong (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fang Yong reassigned FLINK-6787:


Assignee: Fang Yong

> Job-/StoppableException should extend FlinkException
> 
>
> Key: FLINK-6787
> URL: https://issues.apache.org/jira/browse/FLINK-6787
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Affects Versions: 1.4.0
>Reporter: Chesnay Schepler
>Assignee: Fang Yong
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4500: [FLINK-6787] Fix Job-/StoppingException extend Fli...

2017-08-09 Thread zjureel

GitHub user zjureel opened a pull request:

https://github.com/apache/flink/pull/4500

[FLINK-6787] Fix Job-/StoppingException extend FlinkException

## What is the purpose of the change

Fix Job-/StoppingException extend FlinkException

## Brief change log

  - *Job-/StoppingException extend FlinkException instead of Exception*

## Verifying this change

*(Please pick either of the following options)*
This change is already covered by existing tests

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable / docs / 
JavaDocs / not documented)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjureel/flink FLINK-6787

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4500


commit dce23138931c48b494bacd6335573cb34fbea3c6
Author: zjureel 
Date:   2017-08-09T05:49:02Z

[FLINK-6787] Fix Job-/StoppableException extend FlinkException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7394) Implement basic InputChannel for credit-based logic

2017-08-09 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119498#comment-16119498
 ] 

ASF GitHub Bot commented on FLINK-7394:
---

GitHub user zhijiangW opened a pull request:

https://github.com/apache/flink/pull/4499

[FLINK-7394][core] Implement basic InputChannel for credit-based logic

## What is the purpose of the change

This pull request prepares the basic works for credit-based interaction 
with netty pipeline which will be submitted in next pull request.

This pull request depends on previous [FixedBufferPool 
PR](https://github.com/apache/flink/pull/4485) whose commit is also included 
for passing travis, so review the second commit for this pull request change.

In order not to affect the current process and existing cases, a temporary 
method called `testRegisterTask` is added in `NetworkEnvironment` for verifying 
partial behaviors until the whole feature codes are submitted.

## Brief change log

  - *`NetworkEnvironment` creates the `FixedBufferPool` for 
`SingleInputGate`*
  - *Assigns the exclusive buffers directly to `RemoteInputChannel` during 
registering task*
  - *`RemoteInputChannel` implements `BufferRecycler` interface to manage 
the exclusive buffers itself*
  - *`RemoteInputChannel` implements `BufferPoolListener` interface to be 
notified available floating buffers from buffer pool*
  - *`RemoteInputChannel` maintains unannounced credit and current sender 
backlog*

## Verifying this change

This change added tests and can be verified as follows:

  - *Added test for registering task with FixedBufferPool creation and 
exclusive buffers assignment*
  - *The credit and backlog logics will be verified after implementing the 
netty pipeline part in the next pull request*

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (yes)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhijiangW/flink FLINK-7394

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4499


commit 5c7a27f6fdd215150174c7827cc87b5ea08e01bc
Author: Zhijiang 
Date:   2017-08-07T09:31:17Z

[FLINK-7378][core]Implement the FixedBufferPool for floating buffers of 
SingleInputGate

commit b3cabd92051b2682c869742a5a346193b3d4ca33
Author: Zhijiang 
Date:   2017-08-09T05:43:56Z

[FLINK-7394][core]Implement basic InputChannel for credit-based logic




> Implement basic InputChannel for credit-based logic
> ---
>
> Key: FLINK-7394
> URL: https://issues.apache.org/jira/browse/FLINK-7394
> Project: Flink
>  Issue Type: Sub-task
>  Components: Core
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control. 
> The basic works are:
> * The exclusive buffers per channel are assigned to {{RemoteInputChannel}} 
> directly during registering task.
> * {{RemoteInputChannel}} implements {{BufferRecycler}} interface to manage 
> the exclusive buffers itself.
> * {{RemoteInputChannel}} implements {{BufferPoolListener}} interface to be 
> notified available floating buffers from buffer pool.
> * {{RemoteInputChannel}} maintains unannounced credit and current sender 
> backlog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4499: [FLINK-7394][core] Implement basic InputChannel fo...

2017-08-09 Thread zhijiangW

GitHub user zhijiangW opened a pull request:

https://github.com/apache/flink/pull/4499

[FLINK-7394][core] Implement basic InputChannel for credit-based logic

## What is the purpose of the change

This pull request prepares the basic works for credit-based interaction 
with netty pipeline which will be submitted in next pull request.

This pull request depends on previous [FixedBufferPool 
PR](https://github.com/apache/flink/pull/4485) whose commit is also included 
for passing travis, so review the second commit for this pull request change.

In order not to affect the current process and existing cases, a temporary 
method called `testRegisterTask` is added in `NetworkEnvironment` for verifying 
partial behaviors until the whole feature codes are submitted.

## Brief change log

  - *`NetworkEnvironment` creates the `FixedBufferPool` for 
`SingleInputGate`*
  - *Assigns the exclusive buffers directly to `RemoteInputChannel` during 
registering task*
  - *`RemoteInputChannel` implements `BufferRecycler` interface to manage 
the exclusive buffers itself*
  - *`RemoteInputChannel` implements `BufferPoolListener` interface to be 
notified available floating buffers from buffer pool*
  - *`RemoteInputChannel` maintains unannounced credit and current sender 
backlog*

## Verifying this change

This change added tests and can be verified as follows:

  - *Added test for registering task with FixedBufferPool creation and 
exclusive buffers assignment*
  - *The credit and backlog logics will be verified after implementing the 
netty pipeline part in the next pull request*

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (yes)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhijiangW/flink FLINK-7394

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4499


commit 5c7a27f6fdd215150174c7827cc87b5ea08e01bc
Author: Zhijiang 
Date:   2017-08-07T09:31:17Z

[FLINK-7378][core]Implement the FixedBufferPool for floating buffers of 
SingleInputGate

commit b3cabd92051b2682c869742a5a346193b3d4ca33
Author: Zhijiang 
Date:   2017-08-09T05:43:56Z

[FLINK-7394][core]Implement basic InputChannel for credit-based logic




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7352) ExecutionGraphRestartTest timeouts

2017-08-09 Thread Till Rohrmann (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119492#comment-16119492
 ] 

Till Rohrmann commented on FLINK-7352:
--

I think [~StephanEwen] is right and the problem is 
https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphTestUtils.java#L203.
 You can simulate it by removing the sleep and introducing a small sleep in 
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java#L401.

I think the solution would be to wait on the {{SimpleAckingTaskManagerGateway}} 
until it has received all task submissions before switching the {{Executions}} 
to running.

> ExecutionGraphRestartTest timeouts
> --
>
> Key: FLINK-7352
> URL: https://issues.apache.org/jira/browse/FLINK-7352
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, Tests
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Nico Kruber
>Priority: Critical
>  Labels: test-stability
>
> Recently, I received timeouts from some tests in 
> {{ExecutionGraphRestartTest}} like this
> {code}
> Tests in error: 
>   ExecutionGraphRestartTest.testConcurrentLocalFailAndRestart:638 » Timeout
> {code}
> This particular instance is from 1.3.2 RC2 and stuck in 
> {{ExecutionGraphTestUtils#waitUntilDeployedAndSwitchToRunning()}} but I also 
> had instances stuck in {{ExecutionGraphTestUtils#waitUntilJobStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

< 1 2 3

201 - 239 of 239 matches

Mail list logo