[jira] [Updated] (FLINK-12342) Yarn Resource Manager Acquires Too Many Containers

[Zhenqiu Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenqiu Huang updated FLINK-12342:
--
Attachment: flink-1.6.png
flink-1.4.png

> Yarn Resource Manager Acquires Too Many Containers
> --
>
> Key: FLINK-12342
> URL: https://issues.apache.org/jira/browse/FLINK-12342
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / YARN
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Environment: We runs job in Flink release 1.6.3. 
>Reporter: Zhenqiu Huang
>Assignee: Zhenqiu Huang
>Priority: Major
>  Labels: pull-request-available
> Attachments: Screen Shot 2019-04-29 at 12.06.23 AM.png, 
> container.log, flink-1.4.png, flink-1.6.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In currently implementation of YarnFlinkResourceManager, it starts to acquire 
> new container one by one when get request from SlotManager. The mechanism 
> works when job is still, say less than 32 containers. If the job has 256 
> container, containers can't be immediately allocated and appending requests 
> in AMRMClient will be not removed accordingly. We observe the situation that 
> AMRMClient ask for current pending request + 1 (the new request from slot 
> manager) containers. In this way, during the start time of such job, it asked 
> for 4000+ containers. If there is an external dependency issue happens, for 
> example hdfs access is slow. Then, the whole job will be blocked without 
> getting enough resource and finally killed with SlotManager request timeout.
> Thus, we should use the total number of container asked rather than pending 
> request in AMRMClient as threshold to make decision whether we need to add 
> one more resource request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12342) Yarn Resource Manager Acquires Too Many Containers

[Zhenqiu Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenqiu Huang updated FLINK-12342:
--
Attachment: Screen Shot 2019-04-29 at 12.06.23 AM.png

> Yarn Resource Manager Acquires Too Many Containers
> --
>
> Key: FLINK-12342
> URL: https://issues.apache.org/jira/browse/FLINK-12342
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / YARN
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
> Environment: We runs job in Flink release 1.6.3. 
>Reporter: Zhenqiu Huang
>Assignee: Zhenqiu Huang
>Priority: Major
>  Labels: pull-request-available
> Attachments: Screen Shot 2019-04-29 at 12.06.23 AM.png, container.log
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In currently implementation of YarnFlinkResourceManager, it starts to acquire 
> new container one by one when get request from SlotManager. The mechanism 
> works when job is still, say less than 32 containers. If the job has 256 
> container, containers can't be immediately allocated and appending requests 
> in AMRMClient will be not removed accordingly. We observe the situation that 
> AMRMClient ask for current pending request + 1 (the new request from slot 
> manager) containers. In this way, during the start time of such job, it asked 
> for 4000+ containers. If there is an external dependency issue happens, for 
> example hdfs access is slow. Then, the whole job will be blocked without 
> getting enough resource and finally killed with SlotManager request timeout.
> Thus, we should use the total number of container asked rather than pending 
> request in AMRMClient as threshold to make decision whether we need to add 
> one more resource request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12279) Create source to allow streaming data from websocket.

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829603#comment-16829603
 ] 

Dominik Wosiński commented on FLINK-12279:
--

AFAIK, it will provide same delivery guarantees as standard Java sockets, which 
are supported in Flink currently.  Maybe, you are right though  and possibly 
this issue should be moved to BAHIR.

 

> Create source to allow streaming data from websocket.
> -
>
> Key: FLINK-12279
> URL: https://issues.apache.org/jira/browse/FLINK-12279
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common
>Reporter: Dominik Wosiński
>Priority: Major
>
> Currently, there exists an API that allows user to read data from regular 
> Java socket. I think we should also create an API that will allow reading and 
> streaming data from websockets too. Java does have the `javax.websocket-api` 
> that allows asynchronous reading from webockets and I think it could be used 
> for this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-12367) Create rich versions of functions for Scala

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLINK-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dominik Wosiński closed FLINK-12367.

Resolution: Invalid

> Create rich versions of functions for Scala
> ---
>
> Key: FLINK-12367
> URL: https://issues.apache.org/jira/browse/FLINK-12367
> Project: Flink
>  Issue Type: Improvement
>Reporter: Dominik Wosiński
>Assignee: Dominik Wosiński
>Priority: Major
>
> Currently for some of the functions like for example _AsyncFunctions_ have 
> their rich versions available only for Java, which makes using them more 
> complicated to use in Scala. For example using _RichAsyncFunction_ that 
> allows state for example requires a lot of changes, since AFAIK it will not 
> normally work with _DataStream_ from scala API. This will force user to make 
> various changes in the code to be compatible with the API_._ This also 
> includes the fact that the variance is different for Scala and Java 
> collections, since the former ones are covariant by default and the latter 
> ones are invariant by default.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] flinkbot commented on issue #8314: [FLINK-12365][table] Add stats related catalog APIs

[GitBox]

flinkbot commented on issue #8314: [FLINK-12365][table] Add stats related 
catalog APIs
URL: https://github.com/apache/flink/pull/8314#issuecomment-487671263
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (FLINK-12312) Remove CLI command for rescaling

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao closed FLINK-12312.

Resolution: Fixed

1.9: 566d10d958d71fd113f8f7ecc08fa9b63a072919

> Remove CLI command for rescaling
> 
>
> Key: FLINK-12312
> URL: https://issues.apache.org/jira/browse/FLINK-12312
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Temporarily remove support to rescale job via CLI. See this thread for more 
> details: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Temporarily-remove-support-for-job-rescaling-via-CLI-action-quot-modify-quot-td27447.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] xuefuz opened a new pull request #8314: [FLINK-12365][table] Add stats related catalog APIs

[GitBox]

xuefuz opened a new pull request #8314: [FLINK-12365][table] Add stats related 
catalog APIs
URL: https://github.com/apache/flink/pull/8314
 
 
   
   
   ## What is the purpose of the change
   
   Added stats related catalog APIs. Made table/column stats first-level meta 
object in catalog. Removed stats related info from table definition.
   
   ## Brief change log
   
   *(for example:)*
 - Added stats related catalog APIs
 - Added stats related classes
 - Implemented these APIs in GenericInMemoryCatalog
 - Adapted CatalogTable definition for the new changes
 - Added tests
   
   ## Verifying this change
   
   Added unit test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (Yes)
 - If yes, how is the feature documented? (JavaDocs)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] asfgit closed pull request #8260: [FLINK-12312][runtime] Remove CLI command for rescaling

[GitBox]

asfgit closed pull request #8260: [FLINK-12312][runtime] Remove CLI command for 
rescaling
URL: https://github.com/apache/flink/pull/8260
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-12312) Remove CLI command for rescaling

[Gary Yao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Yao updated FLINK-12312:
-
Fix Version/s: 1.9.0

> Remove CLI command for rescaling
> 
>
> Key: FLINK-12312
> URL: https://issues.apache.org/jira/browse/FLINK-12312
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination
>Reporter: Gary Yao
>Assignee: Gary Yao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Temporarily remove support to rescale job via CLI. See this thread for more 
> details: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Temporarily-remove-support-for-job-rescaling-via-CLI-action-quot-modify-quot-td27447.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] bowenli86 commented on issue #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on issue #8205: [FLINK-12238] [hive] Support database 
related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#issuecomment-487665834
 
 
   @flinkbot attention @zentol 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] bowenli86 commented on issue #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on issue #8205: [FLINK-12238] [hive] Support database 
related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#issuecomment-487665669
 
 
   > Stephan pointed you in a good direction, we will have to check out what 
Hive is bundling. Looks like only hive-exec is bundling anything, with the 
following include patterns:
   > 
   > ```
   > org.apache.hive:hive-common
   >   org.apache.hive:hive-exec
   >   org.apache.hive:hive-serde
   >   org.apache.hive:hive-llap-common
   >   org.apache.hive:hive-llap-client
   >   org.apache.hive:hive-metastore
   >   org.apache.hive:hive-service-rpc
   >   com.esotericsoftware:kryo-shaded
   >  com.esotericsoftware:minlog
   >  org.objenesis:objenesis
   >   
org.apache.parquet:parquet-hadoop-bundle
   >   org.apache.thrift:libthrift
   >   org.apache.thrift:libfb303
   >   javax.jdo:jdo-api
   >   commons-lang:commons-lang
   >   org.apache.commons:commons-lang3
   >   org.jodd:jodd-core
   >   com.tdunning:json
   >   org.apache.avro:avro
   >   org.apache.avro:avro-mapred
   >   org.apache.hive.shims:hive-shims-0.23
   >   org.apache.hive.shims:hive-shims-0.23
   >   
org.apache.hive.shims:hive-shims-common
   >   com.googlecode.javaewah:JavaEWAH
   >   javolution:javolution
   >   com.google.protobuf:protobuf-java
   >   io.airlift:aircompressor
   >   org.codehaus.jackson:jackson-core-asl
   >   
org.codehaus.jackson:jackson-mapper-asl
   >   com.google.guava:guava
   >   net.sf.opencsv:opencsv
   >   org.apache.hive:spark-client
   >   org.apache.hive:hive-storage-api
   >   org.apache.orc:orc-core
   >   org.apache.orc:orc-tools
   >   joda-time:joda-time
   > ```
   > 
   > At a quick glance I haven't seen anything problematic; nevertheless we'll 
have to handle them like non-transitive dependencies and check their individual 
licenses, list them and bundle the licenses.
   
   I've changed hive-exec to be a test dependency now to not block us for so 
long since our work on hive table source/sink depends on this PR. I thought it 
is a not-so-hard problem to solve and include hive-exec when setting up 
flink-connector-hive but seems I was wrong. So let's leave it as test 
dependency now and change it to compile dependency whenever necessary later. 
What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] 
Support database related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#discussion_r279454577
 
 

 ##
 File path: 
flink-connectors/flink-connector-hive/src/main/resources/META-INF/NOTICE
 ##
 @@ -0,0 +1,34 @@
+flink-connector-hive
 
 Review comment:
   I've changed hive-exec to be a test dependency now to not block us for so 
long since our work on hive table source/sink depends on this PR. I thought it 
is a not-so-hard problem to solve and include hive-exec when setting up 
flink-connector-hive but seems I was wrong. So let's leave it as test 
dependency now and change it to compile dependency whenever necessary later. 
What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] 
Support database related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#discussion_r279454577
 
 

 ##
 File path: 
flink-connectors/flink-connector-hive/src/main/resources/META-INF/NOTICE
 ##
 @@ -0,0 +1,34 @@
+flink-connector-hive
 
 Review comment:
   I've changed hive-exec to be a test dependency now to not block us. I 
thought it is a not-so-hard problem to solve and include it when setting up 
flink-connector-hive but seems I was wrong. So let's leave it as test 
dependency now and change it to compile dependency whenever necessary later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] 
Support database related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#discussion_r279450999
 
 

 ##
 File path: flink-connectors/flink-connector-hive/pom.xml
 ##
 @@ -0,0 +1,425 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+   4.0.0
+
+   
+   org.apache.flink
+   flink-connectors
+   1.9-SNAPSHOT
+   ..
+   
+
+   flink-connector-hive_${scala.binary.version}
+   flink-connector-hive
+
+   jar
+
+   
+   2.3.4
+   
2.7.2
+   
+
+   
+
+   
+
+   
+   org.apache.flink
+   flink-table-common
+   ${project.version}
+   provided
+   
+
+   
+   org.apache.flink
+   flink-table-api-java
+   ${project.version}
+   provided
+   
+
+   
+   
+
+   
+   org.apache.hadoop
+   hadoop-common
+   ${hivemetastore.hadoop.version}
+   provided
+   
+
+   
+   org.apache.hadoop
+   hadoop-mapreduce-client-core
+   ${hivemetastore.hadoop.version}
+   provided
+   
+
+   
+   
+
+   
+   org.apache.hive
+   hive-metastore
+   ${hive.version}
+   
+   
+   org.apache.hive
+   hive-shims
+   
+   
+   javolution
+   javolution
+   
+   
+   com.google.guava
+   guava
+   
+   
+   com.google.protobuf
+   protobuf-java
+   
+   
+   org.apache.derby
+   derby
+   
+   
+   org.apache.hbase
+   hbase-client
+   
+   
+   commons-lang
+   commons-lang
+   
+   
+   com.zaxxer
+   HikariCP
+   
+   
+   javax.jdo
+   jdo-api
+   
+   
+   co.cask.tephra
+   tephra-api
+   
+   
+   co.cask.tephra
+   tephra-core
+   
+   
+   co.cask.tephra
+   
tephra-hbase-compat-1.0
+   
+   
+   commons-cli
+   commons-cli
+   
+   
+   org.apache.thrift
+   libfb303
+   
+   
+   javax.transaction
+   transaction-api
+   
+   
+   org.apache.orc
+   orc-core
+   
+   
+   joda-time
+   joda-time
+   
+   
+   

[GitHub] [flink] zhijiangW commented on issue #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

zhijiangW commented on issue #8310: [FLINK-12331][network] Introduce 
partition/gate setup to decouple task registration with NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#issuecomment-487657881
 
 
   Thanks for reviews @azagrebin !
   I think your suggestion of migrating the creation of partition/gate to 
`TaskExecutor` is great and I would further consider it in a separate commit or 
PR. :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279447428
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java
 ##
 @@ -131,6 +140,9 @@ public ResultPartition(
ResultPartitionType partitionType,
int numberOfSubpartitions,
int numTargetKeyGroups,
+   int networkBuffersPerChannel,
 
 Review comment:
   That might be a good idea from the aspect of minimal changes especially for 
tests. I would consider and have a try. It actually takes much effort to 
refactor the existing tests. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279447428
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java
 ##
 @@ -131,6 +140,9 @@ public ResultPartition(
ResultPartitionType partitionType,
int numberOfSubpartitions,
int numTargetKeyGroups,
+   int networkBuffersPerChannel,
 
 Review comment:
   That might be a good idea from the aspect of minimal changes especially for 
tests. I would consider and have a try.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279446413
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
 ##
 @@ -290,27 +304,38 @@ public String getOwningTaskName() {
// Setup/Life-cycle
// 

 
-   public void setBufferPool(BufferPool bufferPool) {
+   @Override
+   public void setup() throws IOException {
checkState(this.bufferPool == null, "Bug in input gate setup 
logic: buffer pool has" +
"already been set for this input gate.");
 
-   this.bufferPool = checkNotNull(bufferPool);
+   int maxNumberOfMemorySegments;
+   try {
+   if (isCreditBased) {
+   // assign exclusive buffers to input channels 
directly and use the rest for floating buffers
+   assignExclusiveSegments();
+
+   maxNumberOfMemorySegments = 
consumedPartitionType.isBounded() ? floatingNetworkBuffersPerGate : 
Integer.MAX_VALUE;
+   bufferPool = 
networkBufferPool.createBufferPool(0, maxNumberOfMemorySegments);
+   } else {
+   maxNumberOfMemorySegments = 
consumedPartitionType.isBounded() ?
+   numberOfInputChannels * 
networkBuffersPerChannel + floatingNetworkBuffersPerGate : Integer.MAX_VALUE;
+   bufferPool = 
networkBufferPool.createBufferPool(numberOfInputChannels, 
maxNumberOfMemorySegments);
+   }
+   } catch (Throwable t) {
+   if (bufferPool != null) {
+   bufferPool.lazyDestroy();
+   }
+
+   ExceptionUtils.rethrowIOException(t);
+   }
}
 
/**
 * Assign the exclusive buffers to all remote input channels directly 
for credit-based mode.
-*
-* @param networkBufferPool The global pool to request and recycle 
exclusive buffers
-* @param networkBuffersPerChannel The number of exclusive buffers for 
each channel
 */
-   public void assignExclusiveSegments(NetworkBufferPool 
networkBufferPool, int networkBuffersPerChannel) throws IOException {
-   checkState(this.isCreditBased, "Bug in input gate setup logic: 
exclusive buffers only exist with credit-based flow control.");
-   checkState(this.networkBufferPool == null, "Bug in input gate 
setup logic: global buffer pool has" +
 
 Review comment:
   In the constructor we already check not null for `NetworkBufferPool`. 
Actually `assignExclusiveSegments` should be called only from `setup` which 
also checks `networkBufferPool == null` already, so I removed this check here 
and it have not changed anything in practice.
   
   Considering not changing existing tests so much atm, so I retain this method 
public now. Do you mean if this method is used in tests separately, this 
removed check should be valid to be restored?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] 
Support database related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#discussion_r279444906
 
 

 ##
 File path: 
flink-connectors/flink-connector-hive/src/main/resources/META-INF/NOTICE
 ##
 @@ -0,0 +1,34 @@
+flink-connector-hive
+Copyright 2014-2019 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+This project bundles the following dependencies under the Apache Software 
License 2.0. (http://www.apache.org/licenses/LICENSE-2.0.txt)
+
+- commons-io:commons-io:2.4
+- commons-io:commons-dbcp:1.4
+- commons-io:commons-pool:1.5.4
+- commons-logging:commons-logging:1.1.3
+- commons-configuration:commons-configuration:1.7
+- commons-beanutils:commons-beanutils:1.8.3
+- com.jolbox:bonecp:0.8.0.RELEASE
+- org.apache.hive:hive-common:2.3.4
+- org.apache.hive:hive-exec:2.3.4
+- org.apache.hive:hive-metastore:2.3.4
+- org.apache.hive:hive-serde:2.3.4
+- org.apache.hive:hive-service-rpc:2.3.4
+- org.apache.hive:hive-storage-api:2.4.0
+- org.apache.thrift:libthrift:0.9.3
+- org.datanucleus:datanucleus-api-jdo:4.2.4
+- org.datanucleus:datanucleus-core:4.1.17
+- org.datanucleus:datanucleus-rdbms:4.1.19
+- org.datanucleus:javax.jdo:3.2.0-m3
+
+This project bundles the following dependencies under the CDDL and GPL 2.0 
license.
+
+- javax.servlet:servlet-api:2.5
+
+This project bundles the following dependencies under the BSD license. 
(https://www.antlr.org/license.html)
+
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module

[GitBox]

bowenli86 commented on a change in pull request #8205: [FLINK-12238] [hive] 
Support database related operations in GenericHiveMetastoreCatalog and setup 
flink-connector-hive module
URL: https://github.com/apache/flink/pull/8205#discussion_r279444868
 
 

 ##
 File path: 
flink-connectors/flink-connector-hive/src/main/resources/META-INF/NOTICE
 ##
 @@ -0,0 +1,34 @@
+flink-connector-hive
+Copyright 2014-2019 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+This project bundles the following dependencies under the Apache Software 
License 2.0. (http://www.apache.org/licenses/LICENSE-2.0.txt)
+
+- commons-io:commons-io:2.4
+- commons-io:commons-dbcp:1.4
+- commons-io:commons-pool:1.5.4
+- commons-logging:commons-logging:1.1.3
+- commons-configuration:commons-configuration:1.7
+- commons-beanutils:commons-beanutils:1.8.3
+- com.jolbox:bonecp:0.8.0.RELEASE
+- org.apache.hive:hive-common:2.3.4
+- org.apache.hive:hive-exec:2.3.4
+- org.apache.hive:hive-metastore:2.3.4
+- org.apache.hive:hive-serde:2.3.4
+- org.apache.hive:hive-service-rpc:2.3.4
+- org.apache.hive:hive-storage-api:2.4.0
+- org.apache.thrift:libthrift:0.9.3
+- org.datanucleus:datanucleus-api-jdo:4.2.4
+- org.datanucleus:datanucleus-core:4.1.17
+- org.datanucleus:datanucleus-rdbms:4.1.19
+- org.datanucleus:javax.jdo:3.2.0-m3
+
+This project bundles the following dependencies under the CDDL and GPL 2.0 
license.
+
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

zhijiangW commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279442622
 
 

 ##
 File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
 ##
 @@ -266,13 +281,14 @@ public void testFairConsumptionRemoteChannels() throws 
Exception {
//  Utilities
// 

 
-   private SingleInputGate createFairnessVerifyingInputGate(int 
numberOfChannels) {
+   private SingleInputGate createFairnessVerifyingInputGate(int 
numberOfChannels, NetworkBufferPool bufferPool) {
 
 Review comment:
   Not need this field actually.  In my first version for this test it was not 
a class field. After refactoring to class field I forgot to change this part.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-11921) Upgrade Calcite dependency to 1.19

[Rong Rong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829427#comment-16829427
 ] 

Rong Rong commented on FLINK-11921:
---

Closing to the end of this issue. Found out that one of the current rules we 
used: {{PushFilterIntoTableSourceScanRule}} is broken due to CALCITE-2454. The 
digest-to-relational mapping is simply not allowing this type of changes on the 
table source. I will investigate and come up with a solution. any 
suggestions/comments will be highly appreciated [~twalthr] [~ykt836] :-)

> Upgrade Calcite dependency to 1.19
> --
>
> Key: FLINK-11921
> URL: https://issues.apache.org/jira/browse/FLINK-11921
> Project: Flink
>  Issue Type: Improvement
>  Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Rong Rong
>Priority: Major
>
> Umbrella issue for all tasks related to the next Calcite upgrade.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] hequn8128 edited a comment on issue #8230: [FLINK-10977][table] Add streaming non-window FlatAggregate to Table API

[GitBox]

hequn8128 edited a comment on issue #8230: [FLINK-10977][table] Add streaming 
non-window FlatAggregate to Table API
URL: https://github.com/apache/flink/pull/8230#issuecomment-487645008
 
 
   @sunjincheng121 Hi, Thank you very much for your review and great 
suggestions. I think the pr has been improved a lot with your help. I have 
addressed your comments and updated the PR. Would be great if you can take 
another look. 
   
   Best, Hequn


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] hequn8128 commented on issue #8230: [FLINK-10977][table] Add streaming non-window FlatAggregate to Table API

[GitBox]

hequn8128 commented on issue #8230: [FLINK-10977][table] Add streaming 
non-window FlatAggregate to Table API
URL: https://github.com/apache/flink/pull/8230#issuecomment-487645008
 
 
   @sunjincheng121 Hi, Thank you very much for your review and great 
suggestions. I think the pr improved a lot with your help. I have addressed 
your comments and updated the PR. Would be great if you can take another look. 
   
   Best, Hequn


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] hequn8128 commented on a change in pull request #8230: [FLINK-10977][table] Add streaming non-window FlatAggregate to Table API

[GitBox]

hequn8128 commented on a change in pull request #8230: [FLINK-10977][table] Add 
streaming non-window FlatAggregate to Table API
URL: https://github.com/apache/flink/pull/8230#discussion_r279434307
 
 

 ##
 File path: 
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/operations/AggregateOperationFactory.java
 ##
 @@ -465,4 +535,98 @@ protected Void defaultMethod(Expression expression) {
return null;
}
}
+
+   /**
+* Extract a table aggregate Expression and it's aliases.
+*/
+   public Tuple2> 
extractTableAggFunctionAndAliases(Expression callExpr) {
+   TableAggFunctionCallVisitor visitor = new 
TableAggFunctionCallVisitor();
+   return Tuple2.of(callExpr.accept(visitor), visitor.getAlias());
+   }
+
+   private class TableAggFunctionCallVisitor extends 
ApiExpressionDefaultVisitor {
+
+   private List alias = new LinkedList<>();
+
+   public List getAlias() {
+   return alias;
+   }
+
+   @Override
+   public Expression visitCall(CallExpression call) {
+   FunctionDefinition definition = 
call.getFunctionDefinition();
+   if (definition.equals(AS)) {
+   return unwrapFromAlias(call);
+   } else if (definition instanceof 
AggregateFunctionDefinition) {
+   if (!isTableAggFunctionCall(call)) {
+   throw fail();
+   }
+   return call;
+   } else {
+   return defaultMethod(call);
+   }
+   }
+
+   private Expression unwrapFromAlias(CallExpression call) {
+   List children = call.getChildren();
+   List aliases = children.subList(1, 
children.size())
+   .stream()
+   .map(alias -> 
ExpressionUtils.extractValue(alias, Types.STRING)
+   .orElseThrow(() -> new 
ValidationException("Unexpected alias: " + alias)))
+   .collect(toList());
+
+   if (!isTableAggFunctionCall(children.get(0))) {
+   throw fail();
+   }
+
+   validateAlias(aliases, (AggregateFunctionDefinition) 
((CallExpression) children.get(0)).getFunctionDefinition());
+   alias = aliases;
+   return children.get(0);
+   }
+
+   private void validateAlias(
+   List aliases,
+   AggregateFunctionDefinition aggFunctionDefinition) {
+
+   TypeInformation resultType = 
aggFunctionDefinition.getResultTypeInfo();
+
+   int callArity = resultType.getTotalFields();
+   int aliasesSize = aliases.size();
+
+   if (aliasesSize > 0 && aliasesSize != callArity) {
+   throw new ValidationException(String.format(
+   "List of column aliases must have same 
degree as table; " +
+   "the returned table of function 
'%s' has " +
+   "%d columns, whereas alias list 
has %d columns",
+   aggFunctionDefinition.getName(),
+   callArity,
+   aliasesSize));
+   }
+   }
+
+   @Override
+   protected AggFunctionCall defaultMethod(Expression expression) {
+   throw fail();
+   }
+
+   private ValidationException fail() {
+   return new ValidationException(
+   "A flatAggregate only accepts an expression 
which defines a table aggregate " +
+   "function that might be followed by 
some alias.");
+   }
+   }
+
+   /**
+* Return true if the input {@link Expression} is a {@link 
CallExpression} of table aggregate function.
+*/
+   public static boolean isTableAggFunctionCall(Expression expression) {
+   return Collections.singletonList(expression).stream()
+   .filter(p -> p instanceof CallExpression)
+   .map(p -> (CallExpression) p)
+   .filter(p -> p.getFunctionDefinition().getType() == 
AGGREGATE_FUNCTION)
+   .filter(p -> p.getFunctionDefinition() instanceof 
AggregateFunctionDefinition)
+   .map(p -> (AggregateFunctionDefinition) 

[jira] [Commented] (FLINK-9787) Change ExecutionConfig#getGlobalJobParameters to return an instance of GlobalJobParameters instead of null if no custom globalJobParameters are set yet

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829396#comment-16829396
 ] 

Robert Metzger commented on FLINK-9787:
---

+1 to address this.

> Change ExecutionConfig#getGlobalJobParameters to return an instance of 
> GlobalJobParameters instead of null if no custom globalJobParameters are set 
> yet
> ---
>
> Key: FLINK-9787
> URL: https://issues.apache.org/jira/browse/FLINK-9787
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Florian Schmidt
>Assignee: Ji Liu
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 2.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently when accessing ExecutionConfig#getGlobalJobParameters this will 
> return `null` if no globalJobParameters are set. This can easily lead to 
> NullPointerExceptions when used with getGlobalJobParameters.toMap()
> An easy improvement for this would be to just return a new instance of 
> GlobalJobParameters if none is set with the empty map as the parameters
> This would be a breaking change since we expose this via 
> RuntimeContext#getExecutionConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12358) Verify whether rest documenation needs to be updated when building pull request

[Yun Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829395#comment-16829395
 ] 

Yun Tang commented on FLINK-12358:
--

[~Zentol] Thanks for your suggestion, I would try to fix this problem in this 
way.

> Verify whether rest documenation needs to be updated when building pull 
> request
> ---
>
> Key: FLINK-12358
> URL: https://issues.apache.org/jira/browse/FLINK-12358
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
>
> Currently, unlike configuration docs, rest-API docs have no any methods to 
> check whether updated to latest code. This is really annoying and not easy to 
> track if only checked by developers.
> I plan to check this in travis to verify whether any files have been updated 
> by using `git status`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-10446.

Resolution: Won't Fix

> Use the "guava beta checker" plugin to keep off of @Beta API
> 
>
> Key: FLINK-10446
> URL: https://issues.apache.org/jira/browse/FLINK-10446
> Project: Flink
>  Issue Type: Task
>  Components: Build System
>Reporter: Ted Yu
>Assignee: Ji Liu
>Priority: Major
>
> The Guava people publish an Error Prone plugin to detect when stuff that's 
> annotated with @Beta gets used. Those things shouldn't be used because the 
> project gives no promises about deprecating before removal.
> plugin:
> https://github.com/google/guava-beta-checker



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829390#comment-16829390
 ] 

Robert Metzger commented on FLINK-10446:


I'm proposing to close this Jira ticket, as it makes our build more complicated 
and slower for a problem Flink doesn't have.

> Use the "guava beta checker" plugin to keep off of @Beta API
> 
>
> Key: FLINK-10446
> URL: https://issues.apache.org/jira/browse/FLINK-10446
> Project: Flink
>  Issue Type: Task
>  Components: Build System
>Reporter: Ted Yu
>Assignee: Ji Liu
>Priority: Major
>
> The Guava people publish an Error Prone plugin to detect when stuff that's 
> annotated with @Beta gets used. Those things shouldn't be used because the 
> project gives no promises about deprecating before removal.
> plugin:
> https://github.com/google/guava-beta-checker



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12353) Add missing module to collect_japicmp_reports.sh

[Yun Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829387#comment-16829387
 ] 

Yun Tang commented on FLINK-12353:
--

[~Zentol] This sounds more reasonable, I would like to try to modify 
japicmp-plugin usage instead of modify this script to achieve the goal. And 
I'll create related PR if this works well.

> Add missing module to collect_japicmp_reports.sh
> 
>
> Key: FLINK-12353
> URL: https://issues.apache.org/jira/browse/FLINK-12353
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
>
> Currently, there are eight modules using japicmp plugin. However, only four 
> of them would collect japicmp reports in {{collect_japicmp_reports.sh}}. I 
> have to modify the shell script to collect all reports and therefore I plan 
> to contribute this change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-11935) Remove DateTimeUtils pull-in and fix datetime casting problem

[Rong Rong (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-11935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829386#comment-16829386
 ] 

Rong Rong commented on FLINK-11935:
---

I think this issue should not block calcite upgrade, we can still move forward 
with the master task and follow up on the next release. 
I am working on bumping the version on the main thread and update the comments 
/ pull-in updates, please kindly take a look once the PR is created on the 
master issue.

> Remove DateTimeUtils pull-in and fix datetime casting problem
> -
>
> Key: FLINK-11935
> URL: https://issues.apache.org/jira/browse/FLINK-11935
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / API
>Reporter: Rong Rong
>Assignee: vinoyang
>Priority: Major
>
> This {{DateTimeUtils}} was pulled in in FLINK-7235.
> Originally the time operation was not correctly done via the {{ymdToJulian}} 
> function before the date {{1970-01-01}} thus we need the fix. similar to 
> addressing this problem:
> {code:java}
>  Optimized :1017-12-05 22:58:58.998 
>  Expected :1017-11-29 22:58:58.998
>  Actual :1017-12-05 22:58:58.998
> {code}
>  
> However, after pulling in avatica 1.13, I found out that the optimized plans 
> of the time operations are actually correct. it is in fact the casting part 
> that creates problem:
> For example, the following:
> *{{(plus(-12000.months, cast('2017-11-29 22:58:58.998', TIMESTAMP))}}*
> result in a StringTestExpression of:
> *{{CAST(1017-11-29 22:58:58.998):VARCHAR(65536) CHARACTER SET "UTF-16LE" 
> COLLATE "ISO-8859-1$en_US$primary" NOT NULL}}*
> but the testing results are:
> {code:java}
>  Optimized :1017-11-29 22:58:58.998
>  Expected :1017-11-29 22:58:58.998
>  Actual :1017-11-23 22:58:58.998
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-12293) Fix some comment typos in flink-streaming-java.

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger closed FLINK-12293.
--
Resolution: Cannot Reproduce

I'm closing this Jira ticket.
It doesn't explain the problem detailed enough.

 

Typos can be fixed through a [hotfix] PR.

> Fix some comment typos in flink-streaming-java.
> ---
>
> Key: FLINK-12293
> URL: https://issues.apache.org/jira/browse/FLINK-12293
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Ji Liu
>Assignee: Ji Liu
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Some comments typos in flink-streaming-java should be fixed. I will provide a 
> PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] flinkbot commented on issue #8313: [FLINK-12184][coordinator]HistoryServerArchieFetcher incompatible with old version

[GitBox]

flinkbot commented on issue #8313: 
[FLINK-12184][coordinator]HistoryServerArchieFetcher incompatible with old 
version
URL: https://github.com/apache/flink/pull/8313#issuecomment-487639936
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] klion26 opened a new pull request #8313: [FLINK-12184][coordinator]HistoryServerArchieFetcher incompatible with old version

[GitBox]

klion26 opened a new pull request #8313: 
[FLINK-12184][coordinator]HistoryServerArchieFetcher incompatible with old 
version
URL: https://github.com/apache/flink/pull/8313
 
 
   ## What is the purpose of the change
   
   If we have old verison json files in history server, the 
HistoryServerArchiveFetcher can't convert legacy job overview. It will throw 
the NullPointerException when trying to convert legacy job overview because the 
tasks don't have the "pending" field.
   
   There are actually two paths for the legacy job overview to consider, one is 
coming from Flink < 1.4 where we actually wrote the pending field and the other 
is for Flink >= 1.4 (legacy mode) which split the pending field up into 
scheduled, deployed and created.
   
   
   ## Brief change log
   Add a condition check and add the right state.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   `HistoryServerTest#testHistoryServerIntegration`
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (**no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (**no**)
 - The serializers: (**no**)
 - The runtime per-record code paths (performance sensitive): (**no**)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (**no**)
 - The S3 file system connector: (**no**)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (**no**)
 - If yes, how is the feature documented? (**not applicable**)
   @tillrohrmann @yumengz5
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8240: [FLINK-12293][docs, datastream]Fix some comment typos in flink-streaming-java.

[GitBox]

rmetzger commented on issue #8240: [FLINK-12293][docs, datastream]Fix some 
comment typos in flink-streaming-java.
URL: https://github.com/apache/flink/pull/8240#issuecomment-487639163
 
 
   Thank you for opening this PR to Flink. I would like to close this pull 
request, without merging it, as it does not substantially improve Flink.
   
   Some of the fixed typos are indeed valid, while others are questionable. 
Still, these typos do not hurt any user experience (as the APIs are not user 
facing), nor do they make it harder to understand the APIs itself.
   I'm sure we would converge to a state (after a few rounds of reviews) where 
this change is acceptable, but then we would have spend a lot of valuable 
committer/contributor time on a change not raising the bar for Flink.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-10900) Mark Kafka 2.0 connector as beta feature

[Seth Wiesman (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829381#comment-16829381
 ] 

Seth Wiesman commented on FLINK-10900:
--

[~till.rohrmann] Is the universal Kafka connector still showing instabilities? 
Are there any tickets I can follow? 

> Mark Kafka 2.0 connector as beta feature
> 
>
> Key: FLINK-10900
> URL: https://issues.apache.org/jira/browse/FLINK-10900
> Project: Flink
>  Issue Type: Task
>  Components: Connectors / Kafka
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Piotr Nowojski
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.7.0
>
>
> Given the test problems with the Kafka 2.0 connector we should mark this 
> connector as a beta feature until we have fully understood why so many tests 
> deadlock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] azagrebin commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

azagrebin commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279417805
 
 

 ##
 File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
 ##
 @@ -266,13 +281,14 @@ public void testFairConsumptionRemoteChannels() throws 
Exception {
//  Utilities
// 

 
-   private SingleInputGate createFairnessVerifyingInputGate(int 
numberOfChannels) {
+   private SingleInputGate createFairnessVerifyingInputGate(int 
numberOfChannels, NetworkBufferPool bufferPool) {
 
 Review comment:
   do we need to pass `bufferPool` here always as argument if it is a class 
field?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

azagrebin commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279426797
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
 ##
 @@ -290,27 +304,38 @@ public String getOwningTaskName() {
// Setup/Life-cycle
// 

 
-   public void setBufferPool(BufferPool bufferPool) {
+   @Override
+   public void setup() throws IOException {
checkState(this.bufferPool == null, "Bug in input gate setup 
logic: buffer pool has" +
"already been set for this input gate.");
 
-   this.bufferPool = checkNotNull(bufferPool);
+   int maxNumberOfMemorySegments;
+   try {
+   if (isCreditBased) {
+   // assign exclusive buffers to input channels 
directly and use the rest for floating buffers
+   assignExclusiveSegments();
+
+   maxNumberOfMemorySegments = 
consumedPartitionType.isBounded() ? floatingNetworkBuffersPerGate : 
Integer.MAX_VALUE;
+   bufferPool = 
networkBufferPool.createBufferPool(0, maxNumberOfMemorySegments);
+   } else {
+   maxNumberOfMemorySegments = 
consumedPartitionType.isBounded() ?
+   numberOfInputChannels * 
networkBuffersPerChannel + floatingNetworkBuffersPerGate : Integer.MAX_VALUE;
+   bufferPool = 
networkBufferPool.createBufferPool(numberOfInputChannels, 
maxNumberOfMemorySegments);
+   }
+   } catch (Throwable t) {
+   if (bufferPool != null) {
+   bufferPool.lazyDestroy();
+   }
+
+   ExceptionUtils.rethrowIOException(t);
+   }
}
 
/**
 * Assign the exclusive buffers to all remote input channels directly 
for credit-based mode.
-*
-* @param networkBufferPool The global pool to request and recycle 
exclusive buffers
-* @param networkBuffersPerChannel The number of exclusive buffers for 
each channel
 */
-   public void assignExclusiveSegments(NetworkBufferPool 
networkBufferPool, int networkBuffersPerChannel) throws IOException {
-   checkState(this.isCreditBased, "Bug in input gate setup logic: 
exclusive buffers only exist with credit-based flow control.");
-   checkState(this.networkBufferPool == null, "Bug in input gate 
setup logic: global buffer pool has" +
 
 Review comment:
   Maybe, we could keep this check `networkBufferPool == null` here where it is 
really relevant and allow nulls in the constructor? I would also keep the 
existing methods used in tests as they are now to avoid so many changes in 
tests at the moment because it looks like another refactoring which could be 
done at least in another commit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

azagrebin commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279410838
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
 ##
 @@ -599,20 +599,23 @@ else if (current == ExecutionState.CANCELING) {
}
 
// 

-   // register the task with the network stack
+   // setup partitions and gates before task running
 
 Review comment:
   Move to separate method `Task.setupPartitionsAndGates()`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

azagrebin commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279410097
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/UnionInputGate.java
 ##
 @@ -146,6 +146,13 @@ public boolean isFinished() {
return true;
}
 
+   @Override
+   public void setup() throws IOException {
+   for (InputGate inputGate : inputGates) {
 
 Review comment:
I would keep this method empty if it is not really supposed to be called.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8310: [FLINK-12331][network] Introduce partition/gate setup to decouple task registration with NetworkEnvironment

[GitBox]

azagrebin commented on a change in pull request #8310: [FLINK-12331][network] 
Introduce partition/gate setup to decouple task registration with 
NetworkEnvironment
URL: https://github.com/apache/flink/pull/8310#discussion_r279398673
 
 

 ##
 File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java
 ##
 @@ -131,6 +140,9 @@ public ResultPartition(
ResultPartitionType partitionType,
int numberOfSubpartitions,
int numTargetKeyGroups,
+   int networkBuffersPerChannel,
 
 Review comment:
   Maybe, we could simplify it by passing a `Supplier 
bufferPoolToSetup` from `NetworkEnvironment` to `ResultPartition`? Code to 
create `ResultPartition` could still stay in `NetworkEnvironment`. 
`registerBufferPool ` could also stay in `ResultPartition`, become private and 
called in `setup`: `registerBufferPool(bufferPoolToSetup.get())`. The builder 
in tests could have some dummy or null supplier by default to avoid changes in 
tests where setting of BufferPool is not relevant.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8303: [FLINK-12343]add file replication config for yarn configuration

[GitBox]

flinkbot edited a comment on issue #8303: [FLINK-12343]add file replication 
config for yarn configuration
URL: https://github.com/apache/flink/pull/8303#issuecomment-487385120
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8303: [FLINK-12343]add file replication config for yarn configuration

[GitBox]

rmetzger commented on issue #8303: [FLINK-12343]add file replication config for 
yarn configuration
URL: https://github.com/apache/flink/pull/8303#issuecomment-487632182
 
 
   I tend to disagree. I think the problem is well defined, the configuration 
parameter name is okay (not great) and the idea of how to solve the problem is 
also good.
   However, there are some issues with the implementation approach: The 
replication factor is set with each call of the utility. I would rather set it 
once globally.
   Secondly, the code seems to use a default value (3), potentially ignoring a 
global configuration param set by HDFS.
   
   @flinkbot approve description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (FLINK-8640) Japicmp fails on java 9

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-8640.
---
   Resolution: Fixed
Fix Version/s: 1.9.0

master:
da1aa85b31f84a7c105e6a103f52fdc520454d90
7619cf31164b780ec803352e8c0c3d7a845a1932

> Japicmp fails on java 9
> ---
>
> Key: FLINK-8640
> URL: https://issues.apache.org/jira/browse/FLINK-8640
> Project: Flink
>  Issue Type: Sub-task
>  Components: Build System
>Affects Versions: 1.5.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{japicmp}} plugin does not work out-of-the-box with java 9 as per 
> [https://github.com/siom79/japicmp/issues/177|https://github.com/siom79/japicmp/issues/177].
> We have to add the following to the plugins dependency section:
> {code}
> 
>   
> javax.xml.bind
> jaxb-api
> 2.3.0
>   
>   
> com.sun.xml.bind
> jaxb-core
> 2.3.0
>   
>   
> com.sun.xml.bind
> jaxb-impl
> 2.3.0
>   
>   
> javax.activation
> javax.activation-api
> 1.2.0
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] zentol merged pull request #8271: [FLINK-8640][build][travis] Enable japicmp on Java 9

[GitBox]

zentol merged pull request #8271:  [FLINK-8640][build][travis] Enable japicmp 
on Java 9 
URL: https://github.com/apache/flink/pull/8271
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-12353) Add missing module to collect_japicmp_reports.sh

[Chesnay Schepler (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829352#comment-16829352
 ] 

Chesnay Schepler commented on FLINK-12353:
--

This is one of those scripts that maybe shouldn't exist in the first place. 
From what I've gathered it should be possible to set an outputDirectory for the 
japicmp-plugin by setting {{projectBuildDir}} in the configuration section to 
something like {{some/constant/path/${project.artifactId}}. This would allow 
you to trivially move all reports under a single directory.

> Add missing module to collect_japicmp_reports.sh
> 
>
> Key: FLINK-12353
> URL: https://issues.apache.org/jira/browse/FLINK-12353
> Project: Flink
>  Issue Type: Bug
>  Components: Build System
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Minor
>
> Currently, there are eight modules using japicmp plugin. However, only four 
> of them would collect japicmp reports in {{collect_japicmp_reports.sh}}. I 
> have to modify the shell script to collect all reports and therefore I plan 
> to contribute this change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] walterddr commented on a change in pull request #8248: [FLINK-12309][JDBC] JDBCOoutputFormat and JDBCAppendTableSink float behavior is not align

[GitBox]

walterddr commented on a change in pull request #8248: [FLINK-12309][JDBC] 
JDBCOoutputFormat and JDBCAppendTableSink float behavior is not align
URL: https://github.com/apache/flink/pull/8248#discussion_r279415433
 
 

 ##
 File path: 
flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java
 ##
 @@ -160,7 +160,11 @@ public void writeRecord(Row row) throws IOException {
break;
case 
java.sql.Types.FLOAT:
case 
java.sql.Types.DOUBLE:
-   
upload.setDouble(index + 1, (double) row.getField(index));
+   if 
(row.getField(index).getClass() == Float.class) {
 
 Review comment:
   I am not sure this is the right approach. shouldn't we do something like:
   ```
   case java.sql.Types.FLOAT:
 upload.setFloat(index + 1, (float) row.getField(index));
 break;
   case java.sql.Types.DOUBLE:
 upload.setDouble(index + 1, (double) row.getField(index));
 break;
   ```
   that would make more sense to me since the row is float, the type mapping is 
also float. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8248: [FLINK-12309][JDBC] JDBCOoutputFormat and JDBCAppendTableSink float behavior is not align

[GitBox]

walterddr commented on a change in pull request #8248: [FLINK-12309][JDBC] 
JDBCOoutputFormat and JDBCAppendTableSink float behavior is not align
URL: https://github.com/apache/flink/pull/8248#discussion_r279414601
 
 

 ##
 File path: 
flink-connectors/flink-jdbc/src/test/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormatTest.java
 ##
 @@ -133,6 +133,32 @@ public void testExceptionOnInvalidType() throws 
IOException {
jdbcOutputFormat.writeRecord(row);
}
 
+   @Test(expected = RuntimeException.class)
+   public void testCastFloatToDoubleType() throws IOException {
+   jdbcOutputFormat = JDBCOutputFormat.buildJDBCOutputFormat()
+   .setDrivername(DRIVER_CLASS)
+   .setDBUrl(DB_URL)
+   .setQuery(String.format(INSERT_TEMPLATE, INPUT_TABLE))
+   .setSqlTypes(new int[] {
+   Types.INTEGER,
+   Types.VARCHAR,
+   Types.VARCHAR,
+   Types.FLOAT,
+   Types.VARCHAR})
+   .finish();
+   jdbcOutputFormat.open(0, 1);
+
+   Row row = new Row(5);
+   row.setField(0, 4);
+   row.setField(1, "hello");
+   row.setField(2, "world");
+   row.setField(3, 0.99f); // jdbcOutputFormat will cast float to 
double
 
 Review comment:
   I usually avoid trailing comments (it's already pretty self-explanatory from 
your test function name


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] walterddr commented on a change in pull request #8248: [FLINK-12309][JDBC] JDBCOoutputFormat and JDBCAppendTableSink float behavior is not align

[GitBox]

walterddr commented on a change in pull request #8248: [FLINK-12309][JDBC] 
JDBCOoutputFormat and JDBCAppendTableSink float behavior is not align
URL: https://github.com/apache/flink/pull/8248#discussion_r279414202
 
 

 ##
 File path: 
flink-connectors/flink-jdbc/src/test/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormatTest.java
 ##
 @@ -133,6 +133,32 @@ public void testExceptionOnInvalidType() throws 
IOException {
jdbcOutputFormat.writeRecord(row);
}
 
+   @Test(expected = RuntimeException.class)
 
 Review comment:
   I am assuming you are trying to catch the `"imthewrongtype"` on the last 
column. but if the jdbc cast float to double fails. It will throw the same 
exception and the test will still pass. I suggest let this pass by setting the 
last field to `INTEGER`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-12202) Consider introducing batch metric register in NetworkEnviroment

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-12202:
-
Component/s: Runtime / Metrics

> Consider introducing batch metric register in NetworkEnviroment
> ---
>
> Key: FLINK-12202
> URL: https://issues.apache.org/jira/browse/FLINK-12202
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Metrics, Runtime / Network
>Reporter: Andrey Zagrebin
>Assignee: zhijiang
>Priority: Major
>
> As we have some network specific metrics registered in TaskIOMetricGroup 
> (In/OutputBuffersGauge, In/OutputBufferPoolUsageGauge), we can introduce 
> batch metric registering in 
> NetworkEnviroment.registerMetrics(ProxyMetricGroup, partitions, gates), where 
> task passes its TaskIOMetricGroup into ProxyMetricGroup. This way we could 
> break a tie between task and NetworkEnviroment. 
> TaskIOMetricGroup.initializeBufferMetrics, In/OutputBuffersGauge, 
> In/OutputBufferPoolUsageGauge could be moved into 
> NetworkEnviroment.registerMetrics and network code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] tillrohrmann closed pull request #8227: [FLINK-12180][Tests] Port ExecutionVertexCancelTest to new codebase

[GitBox]

tillrohrmann closed pull request #8227: [FLINK-12180][Tests] Port 
ExecutionVertexCancelTest to new codebase
URL: https://github.com/apache/flink/pull/8227
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (FLINK-12180) Port ExecutionVertexCancelTest to new codebase

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-12180.
---
Resolution: Fixed

Fixed via 7054a8b05738fb43ca856849299263bf4f584067

> Port ExecutionVertexCancelTest to new codebase
> --
>
> Key: FLINK-12180
> URL: https://issues.apache.org/jira/browse/FLINK-12180
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Coordination, Tests
>Affects Versions: 1.9.0
>Reporter: TisonKun
>Assignee: leesf
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Port {{ExecutionVertexCancelTest}} to new codebase.
> Mainly get rid of the usage of {{Instance}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12343) Allow set file.replication in Yarn Configuration

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829324#comment-16829324
 ] 

Robert Metzger commented on FLINK-12343:


The PR proposes to set {{yarn.file-replication}}, which will set a "global" 
replication factor for all files uploaded by the YARN client.

> Allow set file.replication in Yarn Configuration
> 
>
> Key: FLINK-12343
> URL: https://issues.apache.org/jira/browse/FLINK-12343
> Project: Flink
>  Issue Type: Improvement
>  Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
>Reporter: Zhenqiu Huang
>Assignee: Zhenqiu Huang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, FlinkYarnSessionCli upload jars into hdfs with default 3 
> replications. From our production experience, we find that 3 replications 
> will block big job (256 containers) to launch, when the HDFS is slow due to 
> big workload for batch pipelines. Thus, we want to make the factor 
> customizable from FlinkYarnSessionCli by adding an option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] wuchong commented on a change in pull request #8302: [FLINK-12269][table-blink] Support Temporal Table Join in blink planner and runtime

[GitBox]

wuchong commented on a change in pull request #8302: [FLINK-12269][table-blink] 
Support Temporal Table Join in blink planner and runtime
URL: https://github.com/apache/flink/pull/8302#discussion_r279395163
 
 

 ##
 File path: 
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/join/temporal/TemporalTableJoinAsyncRunner.java
 ##
 @@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.join.temporal;
+
+import org.apache.flink.api.common.functions.util.FunctionUtils;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.functions.async.AsyncFunction;
+import org.apache.flink.streaming.api.functions.async.ResultFuture;
+import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.table.dataformat.BaseRow;
+import org.apache.flink.table.dataformat.DataFormatConverters;
+import org.apache.flink.table.dataformat.DataFormatConverters.RowConverter;
+import org.apache.flink.table.dataformat.GenericRow;
+import org.apache.flink.table.dataformat.JoinedRow;
+import org.apache.flink.table.generated.GeneratedFunction;
+import org.apache.flink.table.generated.GeneratedResultFuture;
+import org.apache.flink.table.runtime.collector.TableFunctionResultFuture;
+import org.apache.flink.table.typeutils.BaseRowTypeInfo;
+import org.apache.flink.types.Row;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+/**
+ * The async join runner to lookup the dimension table.
+ */
+public class TemporalTableJoinAsyncRunner extends RichAsyncFunction {
 
 Review comment:
   Sure


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wuchong commented on a change in pull request #8302: [FLINK-12269][table-blink] Support Temporal Table Join in blink planner and runtime

[GitBox]

wuchong commented on a change in pull request #8302: [FLINK-12269][table-blink] 
Support Temporal Table Join in blink planner and runtime
URL: https://github.com/apache/flink/pull/8302#discussion_r279395068
 
 

 ##
 File path: 
flink-table/flink-table-planner-blink/src/test/resources/org/apache/flink/table/plan/batch/sql/join/TemporalTableJoinTest.xml
 ##
 @@ -0,0 +1,335 @@
+
+
+
+  
+
+  
+
+
+  
+
+
+  
+
+  
+  
+
+  
+
+
+  
+
+
+  
+
+  
+  
+
+  

[GitHub] [flink] flinkbot edited a comment on issue #8290: [FLINK-12070] [network] Implement new bounded blocking subpartitions

[GitBox]

flinkbot edited a comment on issue #8290: [FLINK-12070] [network] Implement new 
bounded blocking subpartitions
URL: https://github.com/apache/flink/pull/8290#issuecomment-487296924
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8300: [FLINK-11638][docs-zh] Translate Savepoints page into Chinese

[GitBox]

flinkbot edited a comment on issue #8300: [FLINK-11638][docs-zh] Translate 
Savepoints page into Chinese
URL: https://github.com/apache/flink/pull/8300#issuecomment-487363436
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8290: [FLINK-12070] [network] Implement new bounded blocking subpartitions

[GitBox]

rmetzger commented on issue #8290: [FLINK-12070] [network] Implement new 
bounded blocking subpartitions
URL: https://github.com/apache/flink/pull/8290#issuecomment-487606252
 
 
   @flinkbot approve description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8305: [hotfix] regenerate rest-docs to latest code

[GitBox]

flinkbot edited a comment on issue #8305: [hotfix] regenerate rest-docs to 
latest code
URL: https://github.com/apache/flink/pull/8305#issuecomment-487389173
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❗ 3. Needs [attention] from.
   - Needs attention by @zentol [PMC]
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8300: [FLINK-11638][docs-zh] Translate Savepoints page into Chinese

[GitBox]

rmetzger commented on issue #8300: [FLINK-11638][docs-zh] Translate Savepoints 
page into Chinese
URL: https://github.com/apache/flink/pull/8300#issuecomment-487605966
 
 
   @flinkbot approve description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8305: [hotfix] regenerate rest-docs to latest code

[GitBox]

rmetzger commented on issue #8305: [hotfix] regenerate rest-docs to latest code
URL: https://github.com/apache/flink/pull/8305#issuecomment-487605589
 
 
   @flinkbot attention @zentol 
   @flinkbot approve description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8307: [FLINK-12359][metrics][tests] Harden SystemResourcesMetricsITCase

[GitBox]

flinkbot edited a comment on issue #8307: [FLINK-12359][metrics][tests] Harden 
SystemResourcesMetricsITCase
URL: https://github.com/apache/flink/pull/8307#issuecomment-487392775
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ✅ 2. There is [consensus] that the contribution should go into to Flink.
   - Approved by @rmetzger [PMC]
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8307: [FLINK-12359][metrics][tests] Harden SystemResourcesMetricsITCase

[GitBox]

rmetzger commented on issue #8307: [FLINK-12359][metrics][tests] Harden 
SystemResourcesMetricsITCase
URL: https://github.com/apache/flink/pull/8307#issuecomment-487604812
 
 
   @flinkbot approve description consensus


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8311: [FLINK-10976][table] Add Aggregate operator to Table API

[GitBox]

flinkbot edited a comment on issue #8311: [FLINK-10976][table] Add Aggregate 
operator to Table API
URL: https://github.com/apache/flink/pull/8311#issuecomment-487553255
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #8117: [FLINK-12115] [filesystems]: Add support for AzureFS

[GitBox]

flinkbot edited a comment on issue #8117: [FLINK-12115] [filesystems]: Add 
support for AzureFS
URL: https://github.com/apache/flink/pull/8117#issuecomment-480408700
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ✅ 1. The [description] looks good.
   - Approved by @rmetzger [PMC]
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8311: [FLINK-10976][table] Add Aggregate operator to Table API

[GitBox]

rmetzger commented on issue #8311: [FLINK-10976][table] Add Aggregate operator 
to Table API
URL: https://github.com/apache/flink/pull/8311#issuecomment-487604013
 
 
   @flinkbot approve description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] rmetzger commented on issue #8117: [FLINK-12115] [filesystems]: Add support for AzureFS

[GitBox]

rmetzger commented on issue #8117: [FLINK-12115] [filesystems]: Add support for 
AzureFS
URL: https://github.com/apache/flink/pull/8117#issuecomment-487603641
 
 
   @flinkbot approve description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wuchong commented on a change in pull request #8302: [FLINK-12269][table-blink] Support Temporal Table Join in blink planner and runtime

[GitBox]

wuchong commented on a change in pull request #8302: [FLINK-12269][table-blink] 
Support Temporal Table Join in blink planner and runtime
URL: https://github.com/apache/flink/pull/8302#discussion_r279386668
 
 

 ##
 File path: 
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/codegen/CalcCodeGenerator.scala
 ##
 @@ -69,6 +71,42 @@ object CalcCodeGenerator {
 new OneInputOperatorWrapper(genOperator)
   }
 
+  private[flink] def generateFunction[T <: Function](
 
 Review comment:
   Because it actually generate code for a `CalcProgram`, it needs to access 
the private method `generateProcessCode` in this file. And 
`FunctionCodeGenerator` only accepts code body as  parameter, not the 
`CalcProgram.
   
   What do you think about renaming the method name to `generateCalcFunction` 
to align with `generateCalcOperator` ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-12351) AsyncWaitOperator should deep copy StreamElement when object reuse is enabled

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829298#comment-16829298
 ] 

Jark Wu commented on FLINK-12351:
-

[~rmetzger] Thanks for the reminder. 

> AsyncWaitOperator should deep copy StreamElement when object reuse is enabled
> -
>
> Key: FLINK-12351
> URL: https://issues.apache.org/jira/browse/FLINK-12351
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.9.0
>
>
> Currently, AsyncWaitOperator directly put the input StreamElement into 
> {{StreamElementQueue}}. But when object reuse is enabled, the StreamElement 
> is reused, which means the element in {{StreamElementQueue}} will be 
> modified. As a result, the output of AsyncWaitOperator might be wrong.
> An easy way to fix this might be deep copy the input StreamElement when 
> object reuse is enabled, like this: 
> https://github.com/apache/flink/blob/blink/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java#L209



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12367) Create rich versions of functions for Scala

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLINK-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dominik Wosiński updated FLINK-12367:
-
Description: 
Currently for some of the functions like for example _AsyncFunctions_ have 
their rich versions available only for Java, which makes using them more 
complicated to use in Scala. For example using _RichAsyncFunction_ that allows 
state for example requires a lot of changes, since AFAIK it will not normally 
work with _DataStream_ from scala API. This will force user to make various 
changes in the code to be compatible with the API_._ This also includes the 
fact that the variance is different for Scala and Java collections, since the 
former ones are covariant by default and the latter ones are invariant by 
default.

 

 

  was:Currently for some of the functions like for example _AsyncFunctions_ 
have their rich versions available only for Java, which makes using them more 
complicated to use in Scala. For example using _RichAsyncFunction_ that allows 
state for example requires a lot of changes, since AFAIK it will not normally 
work with _DataStream_ from scala API. This will force user to make various 
changes in the code to be compatible with the API.


> Create rich versions of functions for Scala
> ---
>
> Key: FLINK-12367
> URL: https://issues.apache.org/jira/browse/FLINK-12367
> Project: Flink
>  Issue Type: Improvement
>Reporter: Dominik Wosiński
>Assignee: Dominik Wosiński
>Priority: Major
>
> Currently for some of the functions like for example _AsyncFunctions_ have 
> their rich versions available only for Java, which makes using them more 
> complicated to use in Scala. For example using _RichAsyncFunction_ that 
> allows state for example requires a lot of changes, since AFAIK it will not 
> normally work with _DataStream_ from scala API. This will force user to make 
> various changes in the code to be compatible with the API_._ This also 
> includes the fact that the variance is different for Scala and Java 
> collections, since the former ones are covariant by default and the latter 
> ones are invariant by default.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] XuQianJin-Stars commented on issue #8300: [FLINK-11638][docs-zh] Translate Savepoints page into Chinese

[GitBox]

XuQianJin-Stars commented on issue #8300: [FLINK-11638][docs-zh] Translate 
Savepoints page into Chinese
URL: https://github.com/apache/flink/pull/8300#issuecomment-487597420
 
 
   hi @Leev  Thank you for review this PR. I have modified it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (FLINK-12367) Create rich versions of functions for Scala

[JIRA]

Dominik Wosiński created FLINK-12367:


 Summary: Create rich versions of functions for Scala
 Key: FLINK-12367
 URL: https://issues.apache.org/jira/browse/FLINK-12367
 Project: Flink
  Issue Type: Improvement
Reporter: Dominik Wosiński
Assignee: Dominik Wosiński


Currently for some of the functions like for example _AsyncFunctions_ have 
their rich versions available only for Java, which makes using them more 
complicated to use in Scala. For example using _RichAsyncFunction_ that allows 
state for example requires a lot of changes, since AFAIK it will not normally 
work with _DataStream_ from scala API. This will force user to make various 
changes in the code to be compatible with the API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] flinkbot commented on issue #8312: [FLINK-12366][table] Clean up Catalog APIs to make them more consiste…

[GitBox]

flinkbot commented on issue #8312: [FLINK-12366][table] Clean up Catalog APIs 
to make them more consiste…
URL: https://github.com/apache/flink/pull/8312#issuecomment-487587486
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/reviewing-prs.html) for a full explanation of 
the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8226: [FLINK-12181][Tests] Port ExecutionGraphRestartTest to new codebase

[GitBox]

azagrebin commented on a change in pull request #8226: [FLINK-12181][Tests] 
Port ExecutionGraphRestartTest to new codebase
URL: https://github.com/apache/flink/pull/8226#discussion_r279341488
 
 

 ##
 File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
 ##
 @@ -733,44 +815,67 @@ public void testFailureWhileRestarting() throws 
Exception {
//  Utilities
// 

 
-   private Scheduler createSchedulerWithInstances(int num, 
TaskManagerGateway taskManagerGateway) {
-   final Scheduler scheduler = new Scheduler(executor);
-   final Instance[] instances = new Instance[num];
+   private Scheduler createSchedulerWithSlots(int num, TaskManagerGateway 
taskManagerGateway, SlotPool slotPool) throws Exception {
+   setupSlotPool(slotPool, resourceManagerGateway, 
mainThreadExecutor);
+   Scheduler scheduler = setupScheduler(slotPool, 
mainThreadExecutor);
+   
slotPool.registerTaskManager(taskManagerLocation.getResourceID());
 
-   for (int i = 0; i < instances.length; i++) {
-   instances[i] = createInstance(taskManagerGateway, 55443 
+ i);
-   scheduler.newInstanceAvailable(instances[i]);
+   final List slotOffers = new ArrayList<>(NUM_TASKS);
+   for (int i = 0; i < num; i++) {
+   final AllocationID allocationId = new AllocationID();
+   final SlotOffer slotOffer = new SlotOffer(allocationId, 
0, ResourceProfile.UNKNOWN);
+   slotOffers.add(slotOffer);
}
 
+   slotPool.offerSlots(taskManagerLocation, taskManagerGateway, 
slotOffers);
+
return scheduler;
}
 
-   private static Instance createInstance(TaskManagerGateway 
taskManagerGateway, int port) {
-   final HardwareDescription resources = new 
HardwareDescription(4, 1_000_000_000, 500_000_000, 400_000_000);
-   final TaskManagerLocation location = new TaskManagerLocation(
-   ResourceID.generate(), 
InetAddress.getLoopbackAddress(), port);
-   return new Instance(taskManagerGateway, location, new 
InstanceID(), resources, 1);
+   // 

+
+   private static void setupSlotPool(
+   SlotPool slotPool,
+   ResourceManagerGateway resourceManagerGateway,
+   ComponentMainThreadExecutor mainThreadExecutable) throws 
Exception {
+   final String jobManagerAddress = "foobar";
+
+   slotPool.start(JobMasterId.generate(), jobManagerAddress, 
mainThreadExecutable);
+
+   slotPool.connectToResourceManager(resourceManagerGateway);
}
 
-   // 

+   private static Scheduler setupScheduler(
+   SlotPool slotPool,
+   ComponentMainThreadExecutor mainThreadExecutable) {
+   Scheduler scheduler = new 
SchedulerImpl(LocationPreferenceSlotSelectionStrategy.INSTANCE, slotPool);
+   scheduler.start(mainThreadExecutable);
+   return scheduler;
+   }
 
-   private Tuple2 
createExecutionGraph(RestartStrategy restartStrategy) throws Exception {
-   Instance instance = ExecutionGraphTestUtils.getInstance(
-   new SimpleAckingTaskManagerGateway(),
-   NUM_TASKS);
+   private ExecutionGraph createExecutionGraph(RestartStrategy 
restartStrategy, SlotPool slotPool) throws Exception {
+   setupSlotPool(slotPool, resourceManagerGateway, 
mainThreadExecutor);
 
 Review comment:
   The beginning of `createExecutionGraph` looks like the same as just calling 
`createSchedulerWithSlots(NUM_TASKS, taskManagerGateway, slotPool)`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8226: [FLINK-12181][Tests] Port ExecutionGraphRestartTest to new codebase

[GitBox]

azagrebin commented on a change in pull request #8226: [FLINK-12181][Tests] 
Port ExecutionGraphRestartTest to new codebase
URL: https://github.com/apache/flink/pull/8226#discussion_r279342636
 
 

 ##
 File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
 ##
 @@ -99,6 +106,23 @@
 
private TestingComponentMainThreadExecutorServiceAdapter 
mainThreadExecutor = 
TestingComponentMainThreadExecutorServiceAdapter.forMainThread();
 
+   private JobID jobId;
+
+   private TaskManagerLocation taskManagerLocation;
+
+   private SimpleAckingTaskManagerGateway taskManagerGateway;
+
+   private TestingResourceManagerGateway resourceManagerGateway;
+
+   @Before
+   public void setUp() throws Exception {
+   this.jobId = new JobID();
+
+   taskManagerLocation = new LocalTaskManagerLocation();
+   taskManagerGateway = new SimpleAckingTaskManagerGateway();
 
 Review comment:
   Why do we need these variables created globally? They could be created and 
used only in private graph/scheduler creation methods after some code 
deduplication.
   
   Also `taskManagerGateway` is created again locally in some tests: 
`testLocalFailAndRestart`, `testRestartWithEagerSchedulingAndSlotSharing`, 
`testRestartWithSlotSharingAndNotEnoughResources` which might not be needed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] azagrebin commented on a change in pull request #8226: [FLINK-12181][Tests] Port ExecutionGraphRestartTest to new codebase

[GitBox]

azagrebin commented on a change in pull request #8226: [FLINK-12181][Tests] 
Port ExecutionGraphRestartTest to new codebase
URL: https://github.com/apache/flink/pull/8226#discussion_r279345597
 
 

 ##
 File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
 ##
 @@ -136,149 +162,163 @@ private void 
executeOperationForAllExecutions(ExecutionGraph eg, Consumer executionGraphInstanceTuple =
-   
createExecutionGraph(TestRestartStrategy.directExecuting());
+   try (SlotPool slotPool = new SlotPoolImpl(jobId)) {
+   ExecutionGraph eg = 
createExecutionGraph(TestRestartStrategy.directExecuting(), slotPool);
 
-   ExecutionGraph eg = executionGraphInstanceTuple.f0;
+   restartAfterFailure(eg, new FiniteDuration(2, 
TimeUnit.MINUTES), true);
+   }
 
-   restartAfterFailure(eg, new FiniteDuration(2, 
TimeUnit.MINUTES), true);
}
 
@Test
public void testCancelWhileRestarting() throws Exception {
// We want to manually control the restart and delay
RestartStrategy restartStrategy = new 
InfiniteDelayRestartStrategy();
-   Tuple2 executionGraphInstanceTuple = 
createExecutionGraph(restartStrategy);
-   ExecutionGraph executionGraph = executionGraphInstanceTuple.f0;
-   Instance instance = executionGraphInstanceTuple.f1;
+   try (SlotPool slotPool = new SlotPoolImpl(jobId)) {
+   ExecutionGraph executionGraph = 
createExecutionGraph(restartStrategy, slotPool);
+
+   // Release the TaskManager and wait for the job to 
restart
+   
slotPool.releaseTaskManager(taskManagerLocation.getResourceID(), new 
Exception("Test Exception"));
+   assertEquals(JobStatus.RESTARTING, 
executionGraph.getState());
 
-   // Kill the instance and wait for the job to restart
-   instance.markDead();
-   Assert.assertEquals(JobStatus.RESTARTING, 
executionGraph.getState());
+   assertEquals(JobStatus.RESTARTING, 
executionGraph.getState());
 
-   assertEquals(JobStatus.RESTARTING, executionGraph.getState());
+   // Canceling needs to abort the restart
+   executionGraph.cancel();
 
-   // Canceling needs to abort the restart
-   executionGraph.cancel();
+   assertEquals(JobStatus.CANCELED, 
executionGraph.getState());
 
-   assertEquals(JobStatus.CANCELED, executionGraph.getState());
+   // The restart has been aborted
+   
executionGraph.restart(executionGraph.getGlobalModVersion());
 
-   // The restart has been aborted
-   executionGraph.restart(executionGraph.getGlobalModVersion());
+   assertEquals(JobStatus.CANCELED, 
executionGraph.getState());
+   }
 
-   assertEquals(JobStatus.CANCELED, executionGraph.getState());
}
 
@Test
public void testFailWhileRestarting() throws Exception {
-   Scheduler scheduler = new 
Scheduler(TestingUtils.defaultExecutionContext());
+   try (SlotPool slotPool = new SlotPoolImpl(jobId)) {
+   setupSlotPool(slotPool, resourceManagerGateway, 
mainThreadExecutor);
 
 Review comment:
   The beginning looks like just calling `createExecutionGraph` with custom 
`JobGraph`. If `createExecutionGraph` accepted `JobGraph`, this code could be 
deduplicated.
   The same for other tests with similar code duplication.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-12366) Clean up Catalog APIs to make them more consistent and coherent

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-12366:
---
Labels: pull-request-available  (was: )

> Clean up Catalog APIs to make them more consistent and coherent 
> 
>
> Key: FLINK-12366
> URL: https://issues.apache.org/jira/browse/FLINK-12366
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / API
>Affects Versions: 1.9.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>  Labels: pull-request-available
>
> Some of the APIs seem inconsistent with others in terms of exception thrown 
> and error handling. This is to clean them up to maintain consistency and 
> coherence.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] xuefuz opened a new pull request #8312: [FLINK-12366][table] Clean up Catalog APIs to make them more consiste…

[GitBox]

xuefuz opened a new pull request #8312: [FLINK-12366][table] Clean up Catalog 
APIs to make them more consiste…
URL: https://github.com/apache/flink/pull/8312
 
 
   …nt and coherent
   
   
   
   ## What is the purpose of the change
   
   Clean up existing catalog APIs to make them more consistent and coherent
   
   
   ## Brief change log
   
   
 - Clean up API signature regarding exception thrown
 - Improve error handling

   ## Verifying this change
   
   This change is already covered by existing tests. Tests are modified to 
reflect the proposed changes.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (avaDocs)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (FLINK-12203) Refactor ResultPartitionManager to break tie with Task

[Andrey Zagrebin (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829273#comment-16829273
 ] 

Andrey Zagrebin commented on FLINK-12203:
-

Hey [~rmetzger], thanks for correction. I will try to keep components up to 
date.

> Refactor ResultPartitionManager to break tie with Task
> --
>
> Key: FLINK-12203
> URL: https://issues.apache.org/jira/browse/FLINK-12203
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Network
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At the moment, we have ResultPartitionManager.releasePartitionsProducedBy 
> which uses indexing by task in network environment. These methods are 
> eventually used only by Task which already knows its partitions so Task can 
> use ResultPartition.fail(cause) and TaskExecutor.failPartition could directly 
> use NetworkEnviroment.releasePartitions(Collection). This 
> also requires that JM Execution sends produced partition ids instead of just 
> ExecutionAttemptID.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] becketqin commented on a change in pull request #7677: [FLINK-11249][kafka] Add migration tests for FlinkKafkaProdcuer and FlinkKafkaProducer011

[GitBox]

becketqin commented on a change in pull request #7677: [FLINK-11249][kafka] Add 
migration tests for FlinkKafkaProdcuer and FlinkKafkaProducer011
URL: https://github.com/apache/flink/pull/7677#discussion_r279369052
 
 

 ##
 File path: 
flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaMigrationTestBase.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kafka;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import 
org.apache.flink.api.common.serialization.TypeInformationSerializationSchema;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import 
org.apache.flink.streaming.connectors.kafka.internals.KeyedSerializationSchemaWrapper;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.apache.flink.streaming.util.OperatorSnapshotUtil;
+import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
+import org.apache.flink.testutils.migration.MigrationVersion;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Optional;
+import java.util.Properties;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The base class with migration tests for the Kafka Exactly-Once Producer.
+ */
+@SuppressWarnings("serial")
+public abstract class KafkaMigrationTestBase extends KafkaTestBase {
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(KafkaMigrationTestBase.class);
+   protected static final String TOPIC = 
"flink-kafka-producer-migration-test";
+
+   protected final MigrationVersion testMigrateVersion;
+   protected final TypeInformationSerializationSchema 
integerSerializationSchema =
+   new 
TypeInformationSerializationSchema<>(BasicTypeInfo.INT_TYPE_INFO, new 
ExecutionConfig());
+   protected final KeyedSerializationSchema 
integerKeyedSerializationSchema =
+   new 
KeyedSerializationSchemaWrapper<>(integerSerializationSchema);
+
+   /**
+* TODO change this to the corresponding savepoint version to be 
written (e.g. {@link MigrationVersion#v1_3} for 1.3)
+* TODO and remove all @Ignore annotations on write*Snapshot() methods 
to generate savepoints
+* TODO Note: You should generate the savepoint based on the release 
branch instead of the master.
+*/
+   protected final Optional 
flinkGenerateSavepointVersion = Optional.empty();
+
+   public KafkaMigrationTestBase(MigrationVersion testMigrateVersion) {
+   this.testMigrateVersion = checkNotNull(testMigrateVersion);
+   }
+
+   public String getOperatorSnapshotPath() {
+   return getOperatorSnapshotPath(testMigrateVersion);
+   }
+
+   public String getOperatorSnapshotPath(MigrationVersion version) {
+   return 
"src/test/resources/kafka-migration-kafka-producer-flink-" + version + 
"-snapshot";
+   }
+
+   /**
+* Override {@link KafkaTestBase}. Kafka Migration Tests are starting 
up Kafka/ZooKeeper cluster manually
+*/
+   @BeforeClass
+   public static void prepare() throws Exception {
+   }
+
+   /**
+* Override {@link KafkaTestBase}. Kafka Migration Tests are starting 
up Kafka/ZooKeeper cluster manually
+*/
+   @AfterClass
+   public static void shutDownServices() throws Exception {
+   }
+
+   /**
+* Manually run this to write binary snapshot data.
+*/
+   @Ignore
+   @Test
+   public void writeSnapshot() throws Exception {
+   try {
+   checkState(flinkGenerateSavepointVersion.isPresent());
+   startClusters();
+
+   OperatorSubtaskState snapshot = initializeTestState();
+   OperatorSnapshotUtil.writeStateHandle(snapshot, 

[jira] [Created] (FLINK-12366) Clean up Catalog APIs to make them more consistent and coherent

[Xuefu Zhang (JIRA)]

Xuefu Zhang created FLINK-12366:
---

 Summary: Clean up Catalog APIs to make them more consistent and 
coherent 
 Key: FLINK-12366
 URL: https://issues.apache.org/jira/browse/FLINK-12366
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Affects Versions: 1.9.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


Some of the APIs seem inconsistent with others in terms of exception thrown and 
error handling. This is to clean them up to maintain consistency and coherence.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12365) Add stats related catalog APIs

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated FLINK-12365:

Description: This is to support  (table and column)  stats for 
table/partition with related to catalog.  (was: This is to support functions 
(UDFs) with related to catalog.)

> Add stats related catalog APIs
> --
>
> Key: FLINK-12365
> URL: https://issues.apache.org/jira/browse/FLINK-12365
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / API
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>
> This is to support  (table and column)  stats for table/partition with 
> related to catalog.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12365) Add stats related catalog APIs

[Xuefu Zhang (JIRA)]

Xuefu Zhang created FLINK-12365:
---

 Summary: Add stats related catalog APIs
 Key: FLINK-12365
 URL: https://issues.apache.org/jira/browse/FLINK-12365
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Fix For: 1.9.0


This is to support functions (UDFs) with related to catalog.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [flink] fhueske commented on issue #8034: [FLINK-11733] Provide HadoopMapFunction for org.apache.hadoop.mapreduce.Mapper

[GitBox]

fhueske commented on issue #8034: [FLINK-11733] Provide HadoopMapFunction for 
org.apache.hadoop.mapreduce.Mapper
URL: https://github.com/apache/flink/pull/8034#issuecomment-487579573
 
 
   Sorry, I was out for a few days vacation. Will continue with my review soon. 
   Thanks, Fabian


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] fhueske commented on issue #8064: [FLINK-7244] Add parquet table source

[GitBox]

fhueske commented on issue #8064: [FLINK-7244] Add parquet table source
URL: https://github.com/apache/flink/pull/8064#issuecomment-487579379
 
 
   Sorry, I was out for a few days vacation. Will continue with my review soon. 
   Thanks, Fabian


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (FLINK-12293) Fix some comment typos in flink-streaming-java.

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12293:
---
Component/s: API / DataStream

> Fix some comment typos in flink-streaming-java.
> ---
>
> Key: FLINK-12293
> URL: https://issues.apache.org/jira/browse/FLINK-12293
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Ji Liu
>Assignee: Ji Liu
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Some comments typos in flink-streaming-java should be fixed. I will provide a 
> PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12352) [FLIP-36] [Phase 1] Support cache() / invalidateCache() in Table with default ShuffleService and NetworkStack

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12352:
---
Component/s: Table SQL / API

> [FLIP-36] [Phase 1] Support cache() / invalidateCache() in Table with default 
> ShuffleService and NetworkStack
> -
>
> Key: FLINK-12352
> URL: https://issues.apache.org/jira/browse/FLINK-12352
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / API
>Reporter: Jiangjie Qin
>Priority: Major
> Fix For: 1.9.0
>
>
> The goals of this phase are following:
>  * cache and release intermediate result with shuffle service.
>  * benefit from locality of default shuffle service



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12351) AsyncWaitOperator should deep copy StreamElement when object reuse is enabled

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829245#comment-16829245
 ] 

Robert Metzger commented on FLINK-12351:


Hey [~jark], thanks a lot for opening a ticket in the FLINK bug tracker. I just 
manually assigned the ticket to a component.

For future tickets, please remember to always assign a new issue to a 
component, so that the component owner can pick it up.

> AsyncWaitOperator should deep copy StreamElement when object reuse is enabled
> -
>
> Key: FLINK-12351
> URL: https://issues.apache.org/jira/browse/FLINK-12351
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.9.0
>
>
> Currently, AsyncWaitOperator directly put the input StreamElement into 
> {{StreamElementQueue}}. But when object reuse is enabled, the StreamElement 
> is reused, which means the element in {{StreamElementQueue}} will be 
> modified. As a result, the output of AsyncWaitOperator might be wrong.
> An easy way to fix this might be deep copy the input StreamElement when 
> object reuse is enabled, like this: 
> https://github.com/apache/flink/blob/blink/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java#L209



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12351) AsyncWaitOperator should deep copy StreamElement when object reuse is enabled

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12351:
---
Component/s: API / DataStream

> AsyncWaitOperator should deep copy StreamElement when object reuse is enabled
> -
>
> Key: FLINK-12351
> URL: https://issues.apache.org/jira/browse/FLINK-12351
> Project: Flink
>  Issue Type: Bug
>  Components: API / DataStream
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.9.0
>
>
> Currently, AsyncWaitOperator directly put the input StreamElement into 
> {{StreamElementQueue}}. But when object reuse is enabled, the StreamElement 
> is reused, which means the element in {{StreamElementQueue}} will be 
> modified. As a result, the output of AsyncWaitOperator might be wrong.
> An easy way to fix this might be deep copy the input StreamElement when 
> object reuse is enabled, like this: 
> https://github.com/apache/flink/blob/blink/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/async/AsyncWaitOperator.java#L209



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12338) Update Apache Avro test to use try-with-resources

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12338:
---
Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)

> Update Apache Avro test to use try-with-resources
> -
>
> Key: FLINK-12338
> URL: https://issues.apache.org/jira/browse/FLINK-12338
> Project: Flink
>  Issue Type: Improvement
>  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.8.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Update Apache Avro test to use try-with-resources. Right now some resources 
> aren't close at all. Having the try-with-resources increases readability of 
> the code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12364) Introduce a CheckpointFailureManager to centralized manage checkpoint failure

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12364:
---
Component/s: Runtime / Checkpointing

> Introduce a CheckpointFailureManager to centralized manage checkpoint failure
> -
>
> Key: FLINK-12364
> URL: https://issues.apache.org/jira/browse/FLINK-12364
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Checkpointing
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> This issue tracks the work of T2 section about in design document : 
> [https://docs.google.com/document/d/1ce7RtecuTxcVUJlnU44hzcO2Dwq9g4Oyd8_biy94hJc/edit?usp=sharing]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12279) Create source to allow streaming data from websocket.

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829244#comment-16829244
 ] 

Robert Metzger commented on FLINK-12279:


I'm not sure if we want to have such a connector in Flink. It will never 
provide any advanced delivery guarantees, and it is probably fairly easy to 
implement.

This feels like a great extension that could exist outside of Flink's main 
codebase.
Once it has been created as an extension, I'm happy to refer to it on 
[https://flink.apache.org/ecosystem.html].

> Create source to allow streaming data from websocket.
> -
>
> Key: FLINK-12279
> URL: https://issues.apache.org/jira/browse/FLINK-12279
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common
>Reporter: Dominik Wosiński
>Priority: Major
>
> Currently, there exists an API that allows user to read data from regular 
> Java socket. I think we should also create an API that will allow reading and 
> streaming data from websockets too. Java does have the `javax.websocket-api` 
> that allows asynchronous reading from webockets and I think it could be used 
> for this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12299) ExecutionConfig#setAutoWatermarkInterval should check param(interval should not less than zero)

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12299:
---
Component/s: Runtime / Coordination

> ExecutionConfig#setAutoWatermarkInterval should check param(interval should 
> not less than zero)
> ---
>
> Key: FLINK-12299
> URL: https://issues.apache.org/jira/browse/FLINK-12299
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Coordination
>Reporter: shiwuliang
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In any scenario, `autoWatermarkInterval` should not be less than or equal to 
> zero.
> First of all, this does not correspond to the meaning of 
> `autoWatermarkInterval`.
> Second, in the case where `autoWatermarkInterval` is less than 0, we will not 
> be able to register ourselves in 
> `TimestampsAndPeriodicWatermarksOperator#open`, which will result in the 
> water level of this stream being kept at the lowest level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12279) Create source to allow streaming data from websocket.

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12279:
---
Component/s: Connectors / Common

> Create source to allow streaming data from websocket.
> -
>
> Key: FLINK-12279
> URL: https://issues.apache.org/jira/browse/FLINK-12279
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Common
>Reporter: Dominik Wosiński
>Priority: Major
>
> Currently, there exists an API that allows user to read data from regular 
> Java socket. I think we should also create an API that will allow reading and 
> streaming data from websockets too. Java does have the `javax.websocket-api` 
> that allows asynchronous reading from webockets and I think it could be used 
> for this case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12225) Simplify the interface of the PartFileWriter

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829240#comment-16829240
 ] 

Robert Metzger commented on FLINK-12225:


Hey [~Fokko], thanks a lot for opening a ticket in the FLINK bug tracker. I 
just manually assigned the ticket to a component.

For future tickets, please remember to always assign a new issue to a 
component, so that the component owner can pick it up.

> Simplify the interface of the PartFileWriter
> 
>
> Key: FLINK-12225
> URL: https://issues.apache.org/jira/browse/FLINK-12225
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The Path is not being used, so no sense in including it in the interface



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12250) Rewrite assembleNewPartPath to let it return a new PartPath

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12250:
---
Component/s: Connectors / FileSystem

> Rewrite assembleNewPartPath to let it return a new PartPath
> ---
>
> Key: FLINK-12250
> URL: https://issues.apache.org/jira/browse/FLINK-12250
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Affects Versions: 1.8.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> While debugging some code, I've noticed assembleNewPartPath does not really 
> return a new path. Also rewrote the code a bit so the mutable inProgressPart 
> is changed in a single place



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12225) Simplify the interface of the PartFileWriter

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12225:
---
Component/s: Connectors / FileSystem

> Simplify the interface of the PartFileWriter
> 
>
> Key: FLINK-12225
> URL: https://issues.apache.org/jira/browse/FLINK-12225
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / FileSystem
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The Path is not being used, so no sense in including it in the interface



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12203) Refactor ResultPartitionManager to break tie with Task

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829239#comment-16829239
 ] 

Robert Metzger commented on FLINK-12203:


Hey [~azagrebin], thanks a lot for opening a ticket in the FLINK bug tracker. I 
just manually assigned the ticket to a component.

For future tickets, please remember to always assign a new issue to a 
component, so that the component owner can pick it up.

> Refactor ResultPartitionManager to break tie with Task
> --
>
> Key: FLINK-12203
> URL: https://issues.apache.org/jira/browse/FLINK-12203
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Network
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At the moment, we have ResultPartitionManager.releasePartitionsProducedBy 
> which uses indexing by task in network environment. These methods are 
> eventually used only by Task which already knows its partitions so Task can 
> use ResultPartition.fail(cause) and TaskExecutor.failPartition could directly 
> use NetworkEnviroment.releasePartitions(Collection). This 
> also requires that JM Execution sends produced partition ids instead of just 
> ExecutionAttemptID.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-12203) Refactor ResultPartitionManager to break tie with Task

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-12203:
---
Component/s: Runtime / Network

> Refactor ResultPartitionManager to break tie with Task
> --
>
> Key: FLINK-12203
> URL: https://issues.apache.org/jira/browse/FLINK-12203
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Network
>Reporter: Andrey Zagrebin
>Assignee: Andrey Zagrebin
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At the moment, we have ResultPartitionManager.releasePartitionsProducedBy 
> which uses indexing by task in network environment. These methods are 
> eventually used only by Task which already knows its partitions so Task can 
> use ResultPartition.fail(cause) and TaskExecutor.failPartition could directly 
> use NetworkEnviroment.releasePartitions(Collection). This 
> also requires that JM Execution sends produced partition ids instead of just 
> ExecutionAttemptID.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-12333) Add documentation for all async operations through REST

[Yun Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829232#comment-16829232
 ] 

Yun Tang commented on FLINK-12333:
--

[~till.rohrmann] I have already refactored 
{{AbstractAsynchronousOperationHandlers}} and its sub-classes to add 
description. However, I noticed that current rest documentations are not 
generated to the latest code, and I have created a hot-fix PR, would you please 
take a look [https://github.com/apache/flink/pull/8305] ?

> Add documentation for all async operations through REST 
> 
>
> Key: FLINK-12333
> URL: https://issues.apache.org/jira/browse/FLINK-12333
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation, Runtime / REST
>Affects Versions: 1.8.0
>Reporter: Vishal Santoshi
>Assignee: Yun Tang
>Priority: Minor
>
> Exit code 2 seen ( after 5 minutes ) when 
> {code:java}
> curl  --header "Content-Type: application/json" --request POST --data 
> '{"target-directory":"***","cancel-job":true}'    
> https://***/jobs//savepoints{code}
> It seems that when a REST call is 
> "Triggered the cancel with savepoint command from via the REST call. This 
> command is an asynchronous operation which produces a result (the savepoint 
> path). In order to deliver asynchronous results to the caller, Flink waits 
> before shutting down until they are delivered or until it times out after 5 
> minutes."
>  
> That implies that one has to execute 
> {code:java}
> curl  --request GET   
> https://**/jobs//savepoints/[Request_id]
> {code}
> on the request_id returned by the first call ( within 5 minutes ) , for a 
> clean exit ( code 0 ) 
>  
> Please add this life cycle in flink documentation , in all probability here 
> [https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-jobid-savepoints]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)