[GitHub] [flink] flinkbot edited a comment on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool.
flinkbot edited a comment on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool. URL: https://github.com/apache/flink/pull/9993#issuecomment-546219804 ## CI report: * f47a9f60dff6710ce5a7d5fe341a94d0fffb2d6d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133499201) * 501e86a6e9e8eab7fc26f030d284268d530e093e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134129471) * 3e3a090cfc7d9216701b68664e2c8fa4f34861f7 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134135547) * bb53297bace0091789a1c0fa07e7261a339022b0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134324937) * 9af1c8d0e395ce1b197fef96e53512e7189aa12d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134356941) * d0a3717005a6e4145f7b51aeb5d04003808b31a0 : UNKNOWN * 1df7a8290658f9a7079bfa61524172a39001c6cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134400890) * b9187f5b0d929de6878159ca4915a84db7358ed5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134650245) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool.
flinkbot edited a comment on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool. URL: https://github.com/apache/flink/pull/9993#issuecomment-546219804 ## CI report: * f47a9f60dff6710ce5a7d5fe341a94d0fffb2d6d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133499201) * 501e86a6e9e8eab7fc26f030d284268d530e093e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134129471) * 3e3a090cfc7d9216701b68664e2c8fa4f34861f7 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134135547) * bb53297bace0091789a1c0fa07e7261a339022b0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134324937) * 9af1c8d0e395ce1b197fef96e53512e7189aa12d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134356941) * d0a3717005a6e4145f7b51aeb5d04003808b31a0 : UNKNOWN * 1df7a8290658f9a7079bfa61524172a39001c6cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134400890) * b9187f5b0d929de6878159ca4915a84db7358ed5 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134650245) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool.
flinkbot edited a comment on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool. URL: https://github.com/apache/flink/pull/9993#issuecomment-546219804 ## CI report: * f47a9f60dff6710ce5a7d5fe341a94d0fffb2d6d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133499201) * 501e86a6e9e8eab7fc26f030d284268d530e093e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134129471) * 3e3a090cfc7d9216701b68664e2c8fa4f34861f7 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134135547) * bb53297bace0091789a1c0fa07e7261a339022b0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134324937) * 9af1c8d0e395ce1b197fef96e53512e7189aa12d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134356941) * d0a3717005a6e4145f7b51aeb5d04003808b31a0 : UNKNOWN * 1df7a8290658f9a7079bfa61524172a39001c6cc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134400890) * b9187f5b0d929de6878159ca4915a84db7358ed5 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wsry commented on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool.
wsry commented on issue #9993: [FLINK-14498][runtime]Introduce NetworkBufferPool#isAvailable() for interacting with LocalBufferPool. URL: https://github.com/apache/flink/pull/9993#issuecomment-549005574 I adopted @pnowojski 's suggestion and increased the stack depth. Now the PR has been updated and all other concerns are also resolved. @zhijiangW @pnowojski This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
xuefuz commented on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-549005403 To my understanding, loading/unloading modules is exposed in TableEnvironment, so it's possible that user programmatically call those methods. The original code did declare throwing exceptions. As majority of APIs in TableEnv doesn't throw any exception, I'm okay not to declare any exception and this PR can merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
bowenli86 edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548984963 > Thanks for the contribution. The diff itself looks fine. However, wouldn't it be better if we change existing exception classes (such as ModuleNotFoundException) to runtime? > > The downside of having a general exception (such as ValidationException) is the difficult for an exception handler to differentiate the root cause. (Otherwise, one would have to parse the exception msg.) > > Thus, I prefer retaining the existing exception classes but changing them to runtime. maybe catching unchecked exception isn't optimal for users as they are not declared in API and users aren't sure what unchecked exceptions will be thrown unless looking at impls. Loading/unloading module will most likely happen via config or interactive programming, where error messages matter more. If users ran into this in a program, their program would have bugs and they'd better fix it rather than catching the exceptions. Thus throwing ValidationException is fine, and it aligns better with other parts of Flink SQL. I'm also fine with making ModuleNotExistException/ModuleAlreadyExistException RuntimeException or extend ValidationException. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14570) KeyedStateBootstrapFunction missed onTimer callback method to fire timer
[
https://issues.apache.org/jira/browse/FLINK-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965195#comment-16965195
]
vinoyang commented on FLINK-14570:
--
[~sjwiesman] One more question, is there any plan about providing state
processor scala API?
> KeyedStateBootstrapFunction missed onTimer callback method to fire timer
>
>
> Key: FLINK-14570
> URL: https://issues.apache.org/jira/browse/FLINK-14570
> Project: Flink
> Issue Type: New Feature
> Components: API / State Processor
>Reporter: vinoyang
>Priority: Major
>
> In {{KeyedStateBootstrapFunction#Context}} we provided a {{TimerService}}
> which can be used to querying time and register timer. However, we may not be
> able to fire the timer.
> IMO, we'd better provide a {{onTimer}} callback method(just like
> {{KeyedProcessFunction}}) to expose this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14570) KeyedStateBootstrapFunction missed onTimer callback method to fire timer
[
https://issues.apache.org/jira/browse/FLINK-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965192#comment-16965192
]
vinoyang commented on FLINK-14570:
--
[~sjwiesman] Yes, It seems I misunderstood. Thanks for your explaination.
> KeyedStateBootstrapFunction missed onTimer callback method to fire timer
>
>
> Key: FLINK-14570
> URL: https://issues.apache.org/jira/browse/FLINK-14570
> Project: Flink
> Issue Type: New Feature
> Components: API / State Processor
>Reporter: vinoyang
>Priority: Major
>
> In {{KeyedStateBootstrapFunction#Context}} we provided a {{TimerService}}
> which can be used to querying time and register timer. However, we may not be
> able to fire the timer.
> IMO, we'd better provide a {{onTimer}} callback method(just like
> {{KeyedProcessFunction}}) to expose this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations f
flinkbot edited a comment on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for CoreModule URL: https://github.com/apache/flink/pull/10070#issuecomment-548987396 ## CI report: * dafab2657b7312662d54199ca6aab810018c7a8e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134641448) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations f
flinkbot edited a comment on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for CoreModule URL: https://github.com/apache/flink/pull/10070#issuecomment-548987396 ## CI report: * dafab2657b7312662d54199ca6aab810018c7a8e : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134641448) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for Core
flinkbot commented on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for CoreModule URL: https://github.com/apache/flink/pull/10070#issuecomment-548987396 ## CI report: * dafab2657b7312662d54199ca6aab810018c7a8e : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
bowenli86 edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548984963 > Thanks for the contribution. The diff itself looks fine. However, wouldn't it be better if we change existing exception classes (such as ModuleNotFoundException) to runtime? > > The downside of having a general exception (such as ValidationException) is the difficult for an exception handler to differentiate the root cause. (Otherwise, one would have to parse the exception msg.) > > Thus, I prefer retaining the existing exception classes but changing them to runtime. I'm not sure if catching the unchecked exception here is optimal for users as they are not declared in API signature and users aren't sure what unchecked exceptions will be thrown unless looking at implementations. Besides, loading/unloading module will most likely happen via config or interactive programming, in which case the error message matters more. If users ran into this via a table API program, their program has bugs and they'd better fix it rather than catching the exceptions. Thus I think throwing ValidationException is fine, and it aligns better with other parts of Flink SQL. Though it seems a bit unnecessary given the above use case explained, I also don't have strong opinions of making ModuleNotExistException/ModuleAlreadyExistException RuntimeException or extend ValidationException. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
bowenli86 commented on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548984963 > Thanks for the contribution. The diff itself looks fine. However, wouldn't it be better if we change existing exception classes (such as ModuleNotFoundException) to runtime? > > The downside of having a general exception (such as ValidationException) is the difficult for an exception handler to differentiate the root cause. (Otherwise, one would have to parse the exception msg.) > > Thus, I prefer retaining the existing exception classes but changing them to runtime. I'm not sure if catching the unchecked exception here is optimal for users as they are not declared in API signature and users aren't sure what unchecked exceptions will be thrown unless looking at implementations. Besides, loading/unloading module will most likely happen via config or interactive programming, in which case the error message matters more. If users ran into this via a table API program, that's a bug and they should fix their program rather than catching the exceptions. Thus I think throwing ValidationException is fine, and it aligns better with other parts of Flink SQL. Though it seems a bit unnecessary given the above use case explained, I also don't have strong opinions of making ModuleNotExistException/ModuleAlreadyExistException RuntimeException or extend ValidationException. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive]
Create HiveModule to provide Hive built-in functions
URL: https://github.com/apache/flink/pull/9988#discussion_r341781944
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV100.java
##
@@ -335,4 +338,18 @@ public CatalogColumnStatisticsDataDate
toFlinkDateColStats(ColumnStatisticsData
throw new CatalogException("Failed to get table schema
from deserializer", e);
}
}
+
+ @Override
+ public Set listBuiltInFunctions() {
+ // FunctionInfo doesn't have isBuiltIn() API to tell whether
it's a builtin function or not
+ // prior to Hive 1.2.0
+ throw new UnsupportedOperationException("Listing built in
functions are not supported until Hive 1.2.0");
+ }
+
+ @Override
+ public Optional getBuiltInFunctionInfo(String name) {
+ // FunctionInfo doesn't have isBuiltIn() API to tell whether
it's a builtin function or not
+ // prior to Hive 1.2.0
+ throw new UnsupportedOperationException("Getting built in
functions are not supported until Hive 1.2.0");
+ }
Review comment:
I haven't seen a requirement to list/get non built-in functions yet, thus
probably better not to do it now. Limitations will be mentioned when I write
docs for HiveModule.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for Core
flinkbot commented on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for CoreModule URL: https://github.com/apache/flink/pull/10070#issuecomment-548982015 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e9da0dca2bf5b3ba29976f5387ca58d0ea171575 (Fri Nov 01 23:22:44 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for Cor
bowenli86 commented on issue #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for CoreModule URL: https://github.com/apache/flink/pull/10070#issuecomment-548981854 @xuefuz @lirui-apache @zjuwangg This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14419) Add ModuleFactory, ModuleDescriptor, ModuleValidator for factory discovery service
[ https://issues.apache.org/jira/browse/FLINK-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14419: --- Labels: pull-request-available (was: ) > Add ModuleFactory, ModuleDescriptor, ModuleValidator for factory discovery > service > -- > > Key: FLINK-14419 > URL: https://issues.apache.org/jira/browse/FLINK-14419 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] bowenli86 opened a new pull request #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations
bowenli86 opened a new pull request #10070: [FLINK-14419][table] Add ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator for factory discovery service, and add implementations for CoreModule URL: https://github.com/apache/flink/pull/10070 ## What is the purpose of the change Add ModuleFactory, ModuleDescriptor, ModuleValidator for factory discovery service, and add implementations for CoreModule. ## Brief change log - added ModuleFactory, ModuleDescriptor, ModuleDescriptorValidator interfaces for factory discovery service - added CoreModuleFactory, CoreModuleDescriptor, CoreModuleDescriptorValidator - added UT ## Verifying this change This change added tests and can be verified as CoreModuleFactoryTest and CoreModuleDescriptorTest ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs / JavaDocs) docs will be added later This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
flinkbot edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548157512 ## CI report: * 64e1df1cf04686053e67d20573f76bdf60e5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134296780) * 13b9dd366376e21a2c66714f264cdb1c9b8dbda8 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134458897) * d1cde42bff7217728b2dcbe440f36f04a66625df : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134627901) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-546044427 ## CI report: * 1a5aaeee9aef16c74fc6f2b7e30a07175b4a52a9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133431616) * 79d4d8ab004633d97e0d7d77efc8159a4f19dde5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133440152) * ac4bce2f7aeae8c777b9b8d19ef6cd854498dd21 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133456104) * e54b182364c4ac57b3e5f654ce2ac509066bed01 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134096720) * c53aa33d82422724b78a4b7a38fbe265bd8b86a0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134244695) * dd5c68a06debc3dc7b3d2004597fa685d2322641 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134462762) * b53b983b2587418ba6f6919d9565e976cf3877bc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134611769) * 92a13d72ff0bd4e55d9531b5d51809045d872168 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134627876) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
flinkbot edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548157512 ## CI report: * 64e1df1cf04686053e67d20573f76bdf60e5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134296780) * 13b9dd366376e21a2c66714f264cdb1c9b8dbda8 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134458897) * d1cde42bff7217728b2dcbe440f36f04a66625df : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134627901) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-546044427 ## CI report: * 1a5aaeee9aef16c74fc6f2b7e30a07175b4a52a9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133431616) * 79d4d8ab004633d97e0d7d77efc8159a4f19dde5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133440152) * ac4bce2f7aeae8c777b9b8d19ef6cd854498dd21 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133456104) * e54b182364c4ac57b3e5f654ce2ac509066bed01 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134096720) * c53aa33d82422724b78a4b7a38fbe265bd8b86a0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134244695) * dd5c68a06debc3dc7b3d2004597fa685d2322641 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134462762) * b53b983b2587418ba6f6919d9565e976cf3877bc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134611769) * 92a13d72ff0bd4e55d9531b5d51809045d872168 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134627876) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
flinkbot edited a comment on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548157512 ## CI report: * 64e1df1cf04686053e67d20573f76bdf60e5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134296780) * 13b9dd366376e21a2c66714f264cdb1c9b8dbda8 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134458897) * d1cde42bff7217728b2dcbe440f36f04a66625df : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on a change in pull request #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
xuefuz commented on a change in pull request #9988: [FLINK-14418][hive] Create
HiveModule to provide Hive built-in functions
URL: https://github.com/apache/flink/pull/9988#discussion_r341750761
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV100.java
##
@@ -335,4 +338,18 @@ public CatalogColumnStatisticsDataDate
toFlinkDateColStats(ColumnStatisticsData
throw new CatalogException("Failed to get table schema
from deserializer", e);
}
}
+
+ @Override
+ public Set listBuiltInFunctions() {
+ // FunctionInfo doesn't have isBuiltIn() API to tell whether
it's a builtin function or not
+ // prior to Hive 1.2.0
+ throw new UnsupportedOperationException("Listing built in
functions are not supported until Hive 1.2.0");
+ }
+
+ @Override
+ public Optional getBuiltInFunctionInfo(String name) {
+ // FunctionInfo doesn't have isBuiltIn() API to tell whether
it's a builtin function or not
+ // prior to Hive 1.2.0
+ throw new UnsupportedOperationException("Getting built in
functions are not supported until Hive 1.2.0");
+ }
Review comment:
I think we have two options: 1. Document the restriction (with a JIRA to
track) 2. Just list all functions which might include non-built-in ones.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-546044427 ## CI report: * 1a5aaeee9aef16c74fc6f2b7e30a07175b4a52a9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133431616) * 79d4d8ab004633d97e0d7d77efc8159a4f19dde5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133440152) * ac4bce2f7aeae8c777b9b8d19ef6cd854498dd21 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133456104) * e54b182364c4ac57b3e5f654ce2ac509066bed01 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134096720) * c53aa33d82422724b78a4b7a38fbe265bd8b86a0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134244695) * dd5c68a06debc3dc7b3d2004597fa685d2322641 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134462762) * b53b983b2587418ba6f6919d9565e976cf3877bc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134611769) * 92a13d72ff0bd4e55d9531b5d51809045d872168 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception
bowenli86 commented on issue #10053: [FLINK-14578][table] load/unloadModule() should throw RuntimeException rather than checked exception URL: https://github.com/apache/flink/pull/10053#issuecomment-548941834 @dawidwys @lirui-apache @zjuwangg This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
bowenli86 commented on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-548941387 @xuefuz @lirui-apache @zjuwangg This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-546044427 ## CI report: * 1a5aaeee9aef16c74fc6f2b7e30a07175b4a52a9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133431616) * 79d4d8ab004633d97e0d7d77efc8159a4f19dde5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133440152) * ac4bce2f7aeae8c777b9b8d19ef6cd854498dd21 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133456104) * e54b182364c4ac57b3e5f654ce2ac509066bed01 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134096720) * c53aa33d82422724b78a4b7a38fbe265bd8b86a0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134244695) * dd5c68a06debc3dc7b3d2004597fa685d2322641 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134462762) * b53b983b2587418ba6f6919d9565e976cf3877bc : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134611769) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14354) Provide interfaces instead of abstract classes in org.apache.flink.state.api.functions
[
https://issues.apache.org/jira/browse/FLINK-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965058#comment-16965058
]
Seth Wiesman commented on FLINK-14354:
--
-1 I don't believe this is an appropriate change.
1) This would break backwards compatibility.
2) RichFunction#open is an overloaded method. Along with registering state
descriptors users also perform arbitrary life-cycle events (spawning threads,
creating db connections, etc). State reads likely want to perform a different
set of background actions.
3) I don't believe it is reasonable to assume that a state reader application
has access to the operator class that was used to write the savepoint.
> Provide interfaces instead of abstract classes in
> org.apache.flink.state.api.functions
> --
>
> Key: FLINK-14354
> URL: https://issues.apache.org/jira/browse/FLINK-14354
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor
>Reporter: Mitch Wasson
>Priority: Minor
>
> I've started using the new state processing API in Flink 1.9. Super useful
> and works great for the most part.
> However, I think there is opportunity to simplify implementations that use
> the API. My request to enable these simplifications is to provides interfaces
> instead of (or in addition to) abstract classes in
> org.apache.flink.state.api.functions. Then have the state processing API
> require those interfaces.
> My use case involves maintaining and processing keyed state. This is
> accomplished with a KeyedProcessFunction:
> {color:#cc7832}class {color}BooleanProcess {color:#cc7832}extends
> {color}KeyedProcessFunction[{color:#4e807d}String{color}{color:#cc7832},
> {color}{color:#4e807d}String{color}{color:#cc7832},
> {color}{color:#4e807d}String{color}] {
> {color:#cc7832}var {color}{color:#9876aa}bool{color}:
> ValueState[{color:#cc7832}Boolean{color}] = _
> {color:#cc7832}override def {color}{color:#ffc66d}open{color}(parameters:
> Configuration) {
> {color:#9876aa}bool {color}=
> getRuntimeContext.getState({color:#cc7832}new
> {color}ValueStateDescriptor({color:#6a8759}"boolean-state"{color}{color:#cc7832},
> {color}classOf[{color:#cc7832}Boolean{color}]))
> }
> {color:#cc7832}override def
> {color}{color:#ffc66d}processElement{color}(value:
> {color:#4e807d}String{color}{color:#cc7832}, {color}ctx:
> KeyedProcessFunction[{color:#4e807d}String{color}{color:#cc7832},
> {color}{color:#4e807d}String{color}{color:#cc7832},
> {color}{color:#4e807d}String{color}]#Context{color:#cc7832}, {color}out:
> Collector[{color:#4e807d}String{color}]): {color:#cc7832}Unit {color}= {
> {color:#cc7832}if {color}({color:#9876aa}bool{color}.value) {
> out.collect(value)
> } {color:#cc7832}else {color}{
> {color:#cc7832}if {color}(Math.random < {color:#6897bb}0.005{color}) {
> {color:#9876aa}bool{color}.update({color:#cc7832}true{color})
> out.collect(value)
> }
> }
> }
> }
>
> I then use a KeyedStateReaderFunction like this to inspect
> savepoints/checkpoints:
>
> {color:#cc7832}class {color}BooleanProcessStateReader {color:#cc7832}extends
> {color}KeyedStateReaderFunction[{color:#4e807d}String{color}{color:#cc7832},
> {color}{color:#4e807d}String{color}] {
> {color:#cc7832}var {color}{color:#9876aa}bool{color}:
> ValueState[{color:#cc7832}Boolean{color}] = _
> {color:#cc7832}override def {color}{color:#ffc66d}open{color}(parameters:
> Configuration) {
> {color:#9876aa}bool {color}=
> getRuntimeContext.getState({color:#cc7832}new
> {color}ValueStateDescriptor({color:#6a8759}"boolean-state"{color}{color:#cc7832},
> {color}classOf[{color:#cc7832}Boolean{color}]))
> }
> {color:#cc7832}override def {color}{color:#ffc66d}readKey{color}(key:
> {color:#4e807d}String{color}{color:#cc7832}, {color}ctx:
> KeyedStateReaderFunction.Context{color:#cc7832}, {color}out:
> Collector[{color:#4e807d}String{color}]): {color:#cc7832}Unit {color}=
> { out.collect(key) }
> }
> Ideally, I would like my KeyedStateReaderFunction to look like this:
> {color:#cc7832}class {color}BooleanProcessStateReader {color:#cc7832}extends
> BooleanProcess{color} implements
> KeyedStateReaderFunction[String{color:#cc7832},
> {color}{color:#4e807d}String{color}] {
> {color:#cc7832}override def {color}{color:#ffc66d}readKey{color}(key:
> {color:#4e807d}String{color}{color:#cc7832}, {color}ctx:
> KeyedStateReaderFunction.Context{color:#cc7832}, {color}out:
> Collector[{color:#4e807d}String{color}]): {color:#cc7832}Unit {color}=
> { out.collect(key) }
> }
> However, this can't be done with the current API due Java's single
> inheritance and KeyedStateReaderFunction being an abstract class.
>
[jira] [Commented] (FLINK-14570) KeyedStateBootstrapFunction missed onTimer callback method to fire timer
[
https://issues.apache.org/jira/browse/FLINK-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965055#comment-16965055
]
Seth Wiesman commented on FLINK-14570:
--
[~yanghua] I think you are misunderstanding.
The bootstrap function can set timers that will be written in the savepoint.
Those timers will then fire inside the data stream application that restores
from that state. For example, you could bootstrap a KeyedProcessFunction and
pre-set timers that will call the KeyedProcessFunction#onTimer method.
> KeyedStateBootstrapFunction missed onTimer callback method to fire timer
>
>
> Key: FLINK-14570
> URL: https://issues.apache.org/jira/browse/FLINK-14570
> Project: Flink
> Issue Type: New Feature
> Components: API / State Processor
>Reporter: vinoyang
>Priority: Major
>
> In {{KeyedStateBootstrapFunction#Context}} we provided a {{TimerService}}
> which can be used to querying time and register timer. However, we may not be
> able to fire the timer.
> IMO, we'd better provide a {{onTimer}} callback method(just like
> {{KeyedProcessFunction}}) to expose this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] derjust commented on issue #8371: [FLINK-9679] - Add AvroSerializationSchema
derjust commented on issue #8371: [FLINK-9679] - Add AvroSerializationSchema
URL: https://github.com/apache/flink/pull/8371#issuecomment-548919899
I used this PR fairly successful over a week producing ~50 mil messages in
total.
There is only one catch in the PR that needs to be fixed:
https://github.com/apache/flink/pull/8371/files#diff-c60754c7ce564f4229c22913d783c339R128
```java
int schemaId= schemaCoderProvider.get()
.writeSchema(getSchema());
```
This creates a new http client each time a message is serialized which
creates a significant overhead due to establishing the HTTP connections but
also never leveraging the schema caching
The `schemaCoderProvider.get()` should be cached/kept local after it has
been fetched and used for all subsequent `serialize()` calls.
In fact, it should be done in the same fashion as in the existing source
implementation:
```java
@Override
protected void checkAvroInitialized() {
super.checkAvroInitialized();
if (schemaCoder == null) {
this.schemaCoder = schemaCoderProvider.get();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-546044427 ## CI report: * 1a5aaeee9aef16c74fc6f2b7e30a07175b4a52a9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133431616) * 79d4d8ab004633d97e0d7d77efc8159a4f19dde5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133440152) * ac4bce2f7aeae8c777b9b8d19ef6cd854498dd21 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133456104) * e54b182364c4ac57b3e5f654ce2ac509066bed01 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134096720) * c53aa33d82422724b78a4b7a38fbe265bd8b86a0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134244695) * dd5c68a06debc3dc7b3d2004597fa685d2322641 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134462762) * b53b983b2587418ba6f6919d9565e976cf3877bc : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134611769) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
flinkbot edited a comment on issue #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions URL: https://github.com/apache/flink/pull/9988#issuecomment-546044427 ## CI report: * 1a5aaeee9aef16c74fc6f2b7e30a07175b4a52a9 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133431616) * 79d4d8ab004633d97e0d7d77efc8159a4f19dde5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133440152) * ac4bce2f7aeae8c777b9b8d19ef6cd854498dd21 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133456104) * e54b182364c4ac57b3e5f654ce2ac509066bed01 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134096720) * c53aa33d82422724b78a4b7a38fbe265bd8b86a0 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134244695) * dd5c68a06debc3dc7b3d2004597fa685d2322641 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134462762) * b53b983b2587418ba6f6919d9565e976cf3877bc : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version
flinkbot edited a comment on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version URL: https://github.com/apache/flink/pull/10069#issuecomment-548871205 ## CI report: * 7221802e32b1d0cdcd08028d6f64ccd3c39f1d93 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134595542) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive]
Create HiveModule to provide Hive built-in functions
URL: https://github.com/apache/flink/pull/9988#discussion_r341686746
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV120.java
##
@@ -168,4 +174,47 @@ public CatalogColumnStatisticsDataDate
toFlinkDateColStats(ColumnStatisticsData
throw new CatalogException("Failed to create Flink
statistics for date column", e);
}
}
+
+ @Override
+ public Set listBuiltInFunctions() {
+ try {
+ Method method =
FunctionRegistry.class.getMethod("getFunctionNames");
+ // makeSpecFromName is a static method
+ Set names = (Set) method.invoke(null);
+
+ return names.stream()
+ .filter(n -> getFunctionInfo(n).isBuiltIn())
Review comment:
good catch. Seems I didn't rebuilt src but just reran tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (FLINK-14588) Support Hive version 1.0.0 and 1.0.1
[ https://issues.apache.org/jira/browse/FLINK-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-14588. Release Note: supports Hive 1.0.0 and 1.0.1 Resolution: Fixed master: 15f8f3c52a1bf11ecf9f550388eee550b7fc763e > Support Hive version 1.0.0 and 1.0.1 > > > Key: FLINK-14588 > URL: https://issues.apache.org/jira/browse/FLINK-14588 > Project: Flink > Issue Type: Task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version
flinkbot edited a comment on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version URL: https://github.com/apache/flink/pull/10069#issuecomment-548871205 ## CI report: * 7221802e32b1d0cdcd08028d6f64ccd3c39f1d93 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134595542) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14588) Support Hive version 1.0.0 and 1.0.1
[ https://issues.apache.org/jira/browse/FLINK-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-14588: - Fix Version/s: 1.10.0 > Support Hive version 1.0.0 and 1.0.1 > > > Key: FLINK-14588 > URL: https://issues.apache.org/jira/browse/FLINK-14588 > Project: Flink > Issue Type: Task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14588) Support Hive version 1.0.0 and 1.0.1
[ https://issues.apache.org/jira/browse/FLINK-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li reassigned FLINK-14588: Assignee: Rui Li > Support Hive version 1.0.0 and 1.0.1 > > > Key: FLINK-14588 > URL: https://issues.apache.org/jira/browse/FLINK-14588 > Project: Flink > Issue Type: Task > Components: Connectors / Hive >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] asfgit closed pull request #10062: [FLINK-14588][hive] Support Hive version 1.0.0 and 1.0.1
asfgit closed pull request #10062: [FLINK-14588][hive] Support Hive version 1.0.0 and 1.0.1 URL: https://github.com/apache/flink/pull/10062 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on issue #10062: [FLINK-14588][hive] Support Hive version 1.0.0 and 1.0.1
bowenli86 commented on issue #10062: [FLINK-14588][hive] Support Hive version 1.0.0 and 1.0.1 URL: https://github.com/apache/flink/pull/10062#issuecomment-548879835 LGTM, merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14221) support drop temp system functions and temp catalog functions
[ https://issues.apache.org/jira/browse/FLINK-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li closed FLINK-14221. Resolution: Fixed master: 34a9c104f8679cf866a2de01ca6b87bd8be961a2 > support drop temp system functions and temp catalog functions > - > > Key: FLINK-14221 > URL: https://issues.apache.org/jira/browse/FLINK-14221 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] asfgit closed pull request #10054: [FLINK-14221][table] support drop temp system functions and temp catalog functions
asfgit closed pull request #10054: [FLINK-14221][table] support drop temp system functions and temp catalog functions URL: https://github.com/apache/flink/pull/10054 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive]
Create HiveModule to provide Hive built-in functions
URL: https://github.com/apache/flink/pull/9988#discussion_r341667362
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV110.java
##
@@ -294,4 +296,19 @@ public boolean isDateStats(ColumnStatisticsData
colStatsData) {
public CatalogColumnStatisticsDataDate
toFlinkDateColStats(ColumnStatisticsData hiveDateColStats) {
throw new UnsupportedOperationException("DATE column stats are
not supported until Hive 1.2.0");
}
+
+ @Override
+ public Set listBuiltInFunctions() {
+ // FunctionInfo doesn't have isBuiltIn() API to tell whether
it's a builtin function or not
+ // prior to Hive 1.2.0
+ throw new UnsupportedOperationException("Listing built in
functions are not supported until Hive 1.2.0");
Review comment:
no, persistent function is also native
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive] Create HiveModule to provide Hive built-in functions
bowenli86 commented on a change in pull request #9988: [FLINK-14418][hive]
Create HiveModule to provide Hive built-in functions
URL: https://github.com/apache/flink/pull/9988#discussion_r341667362
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimV110.java
##
@@ -294,4 +296,19 @@ public boolean isDateStats(ColumnStatisticsData
colStatsData) {
public CatalogColumnStatisticsDataDate
toFlinkDateColStats(ColumnStatisticsData hiveDateColStats) {
throw new UnsupportedOperationException("DATE column stats are
not supported until Hive 1.2.0");
}
+
+ @Override
+ public Set listBuiltInFunctions() {
+ // FunctionInfo doesn't have isBuiltIn() API to tell whether
it's a builtin function or not
+ // prior to Hive 1.2.0
+ throw new UnsupportedOperationException("Listing built in
functions are not supported until Hive 1.2.0");
Review comment:
unfortunately no, persistent function is also native
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version
flinkbot commented on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version URL: https://github.com/apache/flink/pull/10069#issuecomment-548871205 ## CI report: * 7221802e32b1d0cdcd08028d6f64ccd3c39f1d93 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-11127) Make metrics query service establish connection to JobManager
[
https://issues.apache.org/jira/browse/FLINK-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964969#comment-16964969
]
Ken commented on FLINK-11127:
-
[~hwanju] [~trohrmann], for context, the "fix" I mentioned is from Sergei in
this thread.
https://issues.apache.org/jira/browse/FLINK-11127?focusedCommentId=16719651=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16719651
First I don't know if the issue I am seeing was a non-issue in 1.7. Because we
are running flink in k8s, we had to implement what Sergei suggested in order
for the JM to identify TMs. I can only confirm that the error shows up with 1.8
whenever I manually cancel the job through the UI, and this job is a forever
running job using default restart strategy. What I did not post is the message
prior to the error message, which is related to connecting to JM failed with a
different IP (starts with 170.x.x.x) that where my JM is actually at (starts
with 10.x.x.x).
> Make metrics query service establish connection to JobManager
> -
>
> Key: FLINK-11127
> URL: https://issues.apache.org/jira/browse/FLINK-11127
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Runtime / Coordination, Runtime
> / Metrics
>Affects Versions: 1.7.0
>Reporter: Ufuk Celebi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> As part of FLINK-10247, the internal metrics query service has been separated
> into its own actor system. Before this change, the JobManager (JM) queried
> TaskManager (TM) metrics via the TM actor. Now, the JM needs to establish a
> separate connection to the TM metrics query service actor.
> In the context of Kubernetes, this is problematic as the JM will typically
> *not* be able to resolve the TMs by name, resulting in warnings as follows:
> {code}
> 2018-12-11 08:32:33,962 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183] has
> failed, address is now gated for [50] ms. Reason: [Association failed with
> [akka.tcp://flink-metrics@flink-task-manager-64b868487c-x9l4b:39183]] Caused
> by: [flink-task-manager-64b868487c-x9l4b: Name does not resolve]
> {code}
> In order to expose the TMs by name in Kubernetes, users require a service
> *for each* TM instance which is not practical.
> This currently results in the web UI not being to display some basic metrics
> about number of sent records. You can reproduce this by following the READMEs
> in {{flink-container/kubernetes}}.
> This worked before, because the JM is typically exposed via a service with a
> known name and the TMs establish the connection to it which the metrics query
> service piggybacked on.
> A potential solution to this might be to let the query service connect to the
> JM similar to how the TMs register.
> I tagged this ticket as an improvement, but in the context of Kubernetes I
> would consider this to be a bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #10048: [FLINK-14462][coordination] Remove JobGraph#allowQueuedScheduling flag because it is always true
tillrohrmann commented on a change in pull request #10048:
[FLINK-14462][coordination] Remove JobGraph#allowQueuedScheduling flag because
it is always true
URL: https://github.com/apache/flink/pull/10048#discussion_r341654375
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmanager/scheduler/SchedulerIsolatedTasksTest.java
##
@@ -212,14 +154,12 @@ public void testScheduleWithDyingInstances() throws
Exception {
// cannot get another slot, since all instances are dead
try {
- testingSlotProvider.allocateSlot(new
ScheduledUnit(getDummyTask()), false, SlotProfile.noRequirements(),
TestingUtils.infiniteTime()).get();
+ testingSlotProvider
+ .allocateSlot(new
ScheduledUnit(getDummyTask()), SlotProfile.noRequirements(),
TestingUtils.infiniteTime())
+ .get(2000L, TimeUnit.MILLISECONDS);
Review comment:
This change looks a bit weird. I think we only see the `TimeoutExeception`
because of the `get(2000L, TimeUnit.MILLISECONDS)` and not from the
`testingSlotProvider` because we use the
`ComponentMainThreadExecutorServiceAdapter.forMainThread` which swallows the
timeout of the slot allocation. I suggest to change this and to set a very low
slot allocation timeout to fail fast.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10048: [FLINK-14462][coordination] Remove JobGraph#allowQueuedScheduling flag because it is always true
tillrohrmann commented on a change in pull request #10048:
[FLINK-14462][coordination] Remove JobGraph#allowQueuedScheduling flag because
it is always true
URL: https://github.com/apache/flink/pull/10048#discussion_r341656391
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionGraphRestartTest.java
##
@@ -632,52 +629,6 @@ public void
testRestartWithEagerSchedulingAndSlotSharing() throws Exception {
}
}
- @Test
- public void testRestartWithSlotSharingAndNotEnoughResources() throws
Exception {
Review comment:
Yes and no I would say. If I'm not mistaken then this test should also fail
if I set `new SimpleSlotProvider(TEST_JOB_ID, parallelism)` because we don't do
proper slot sharing. What one could argue though, is that we don't need to test
recovery with slot sharing if we test restart with no slot sharing and slot
sharing individually.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version
flinkbot commented on issue #10069: [FLINK-14593][client] Port ClusterClient to asynchronous interface version URL: https://github.com/apache/flink/pull/10069#issuecomment-548857874 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 50655691fb4da47096429a90fbfeced0c0062263 (Fri Nov 01 16:39:54 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10027: [FLINK-14549][Table SQL/Planner] Bring more detail by using logicalTy…
flinkbot edited a comment on issue #10027: [FLINK-14549][Table SQL/Planner] Bring more detail by using logicalTy… URL: https://github.com/apache/flink/pull/10027#issuecomment-547346216 ## CI report: * a36cfd4c31b55aca1b05afe59025c2c2d51a893a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133973282) * cb8fd65958950dc26548fdac2c603f1d745089ff : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/133987033) * ec4b87db3dd2a657bd9008befc872a7c0704e0fe : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/133990871) * fdf2798461cf27243bf6c2eb575fea7ced488b42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133994349) * 45ddcc3eb65176473f85c64910deaf7ed6516d90 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134576569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14593) Port ClusterClient to asynchronous interface version
[ https://issues.apache.org/jira/browse/FLINK-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14593: --- Labels: pull-request-available (was: ) > Port ClusterClient to asynchronous interface version > > > Key: FLINK-14593 > URL: https://issues.apache.org/jira/browse/FLINK-14593 > Project: Flink > Issue Type: Sub-task > Components: Client / Job Submission >Reporter: Zili Chen >Assignee: Zili Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TisonKun opened a new pull request #10069: [FLINK-FLINK-14593][client] Port ClusterClient to asynchronous interface version
TisonKun opened a new pull request #10069: [FLINK-FLINK-14593][client] Port ClusterClient to asynchronous interface version URL: https://github.com/apache/flink/pull/10069 ## What is the purpose of the change Port ClusterClient to asynchronous interface version ## Brief change log 1. Port ClusterClient to interface 2. Port `cancel`/`cancelWithSavepoint`/`stopWithSavepoint`/`getAccumulators` to asynchronous version 3. Introduce a settable `TestingClusterClient` and remove `FakeClusterClient` 4. Adjust related code. ## Verifying this change This change is a refactor and covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13905) Separate checkpoint triggering into stages
[
https://issues.apache.org/jira/browse/FLINK-13905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964944#comment-16964944
]
Biao Liu commented on FLINK-13905:
--
[~pnowojski],
{quote}If all things equal, I would prefer to postpone FLINK-13848 as much as
possible, as that would be another chunk of code to review/maintain.{quote}
Fair enough. FLINK-13848 is quite independent. We could come back to it when we
really need it.
I'll try to write the ideas down to see what it looks like. Maybe we will find
a better solution based on that.
> Separate checkpoint triggering into stages
> --
>
> Key: FLINK-13905
> URL: https://issues.apache.org/jira/browse/FLINK-13905
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently {{CheckpointCoordinator#triggerCheckpoint}} includes some heavy IO
> operations. We plan to separate the triggering into different stages. The IO
> operations are executed in IO threads, while other on-memory operations are
> not.
> This is a preparation for making all on-memory operations of
> {{CheckpointCoordinator}} single threaded (in main thread).
> Note that we could not put on-memory operations of triggering into main
> thread directly now. Because there are still some operations on a heavy lock
> (coordinator-wide).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever
flinkbot edited a comment on issue #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever URL: https://github.com/apache/flink/pull/9950#issuecomment-544414033 ## CI report: * decba8623c001e8bbe5dc797de2bc421b2e216ef : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132777340) * 6c7ffb9f9049016ce874b9790cd51ed337cfec00 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132974469) * f381f5765ae1a654d545aff4c679eaf9e4812ddc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132985132) * bdd0016a7387ee18c2b9e7aea879063426065427 : UNKNOWN * 08cc39399fd916635b26ed3da9993d9cb57cb680 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133366893) * 237d6084269371b165908a4e657d7c9fec75155c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133381142) * ba6c666f591a4cbf21f020997225938c8079a1be : UNKNOWN * 41471d53c691b2a01a2c43fa431ab6a65a6c1bab : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133506229) * 8fb4f6710f9363d9326603e64276241013a2ff4f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133821363) * de9bf092853b0d181c97e4eeaffd3b6aed67c317 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134430208) * 6b95831e382b336b7b2025c629bab8f3703bb2ba : UNKNOWN * 7f3ff0ad6096f3718ebfeb9a9cdf598a842c1715 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134571751) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341645333
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java
##
@@ -513,6 +514,76 @@ public static Configuration
cloneConfiguration(Configuration configuration) {
return clonedConfiguration;
}
+ /**
+* Format the default gc logging options
+* @param logDirectory to save the gc log
+* @return the formatted gc logging options string
+*/
+ public static String getGCLoggingOpts(String logDirectory) {
+return "-Xloggc:" + logDirectory + "/gc.log " +
+ "-XX:+PrintGCApplicationStoppedTime " +
+ "-XX:+PrintGCDetails " +
+ "-XX:+PrintGCDateStamps " +
+ "-XX:+UseGCLogFileRotation " +
+ "-XX:NumberOfGCLogFiles=10 " +
+ "-XX:GCLogFileSize=10M " +
+ "-XX:+PrintPromotionFailure " +
+ "-XX:+PrintGCCause";
+ }
+
+ /**
+* Format the default heapdump options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+* @param heapdumpDir to save heap dump file
+* @return the formatted heapdump options string
+*/
+ public static String getHeapdumpOpts(String appId, String ident, String
logDirectory, String heapdumpDir) {
+String dumpDestName = String.format("flink-%s-heapdump.hprof", ident);
+String dumpFileDestPath = new File(heapdumpDir, appId + "-" +
dumpDestName).getAbsolutePath();
+String oomScript = String.format("echo -e 'OutOfMemoryError! Killing
current process %%p...\n" +
+"Check gc logs and heapdump file(%s) for details.' > " +
+logDirectory + "/%s.err; " +
+"kill -9 %%p",
+ dumpFileDestPath, ident);
+return String.format("-XX:+HeapDumpOnOutOfMemoryError " +
+"-XX:HeapDumpPath=%s " +
+"-XX:OnOutOfMemoryError=\"%s\"",
+ dumpFileDestPath,
+ oomScript);
+ }
+
+ /**
+* Get the common jvm options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+ * @param conf flink configuration
+ */
+ public static String getCommonJvmOpts(
+ String appId,
+ String ident,
+ String logDirectory,
+ Configuration conf) {
+String commonOpts = "";
+ boolean enableGCLogging =
conf.getBoolean(CoreOptions.FLINK_JVM_DEFAULT_GC_LOGGING);
+if (enableGCLogging) {
+ // Add default gc logging options if enabled
+ commonOpts += getGCLoggingOpts(logDirectory);
+}
+ boolean enableHeapDump =
conf.getBoolean(CoreOptions.FLINK_JVM_HEAPDUMP_ON_OOM);
+if (enableHeapDump) {
+ // Add default heap dump options if enabled
+ String heapdumpDir =
conf.getString(CoreOptions.FLINK_JVM_HEAPDUMP_DIRECTORY);
+ commonOpts += " " + getHeapdumpOpts(appId, ident, logDirectory,
heapdumpDir);
+}
+if (conf.getString(CoreOptions.FLINK_JVM_OPTIONS).length() > 0) {
+ commonOpts += " " +
conf.getString(CoreOptions.FLINK_JVM_OPTIONS);
+}
+return commonOpts;
+ }
Review comment:
Here as well. Please use tabs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341643976
##
File path: flink-dist/src/main/flink-bin/mesos-bin/mesos-appmaster-job.sh
##
@@ -31,6 +31,13 @@ CC_CLASSPATH=`manglePathList
$(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPA
log="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.log"
log_setting="-Dlog.file="$log"
-Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j.properties
-Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml"
+gclog="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.gc_log"
+
+FLINK_HEAPDUMP_NAME="flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.hprof"
+rm -rf ${FLINK_JVM_HEAPDUMP_DIRECTORY}/${FLINK_HEAPDUMP_NAME}
+FLINK_JVM_GC_LOGGING_OPTS=`getGCLoggingOpts $gclog`
+FLINK_JVM_HEAPDUMP_OPTS=`getHeapdumpOpts $FLINK_HEAPDUMP_NAME $log`
Review comment:
```suggestion
FLINK_JVM_HEAPDUMP_OPTS=$(getHeapdumpOpts $FLINK_HEAPDUMP_NAME ${log})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341644104
##
File path: flink-dist/src/main/flink-bin/mesos-bin/mesos-appmaster.sh
##
@@ -31,6 +31,13 @@ CC_CLASSPATH=`manglePathList
$(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPA
log="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.log"
log_setting="-Dlog.file="$log"
-Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j.properties
-Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml"
+gclog="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.gc_log"
+
+FLINK_HEAPDUMP_NAME="flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.hprof"
+rm -rf ${FLINK_JVM_HEAPDUMP_DIRECTORY}/${FLINK_HEAPDUMP_NAME}
+FLINK_JVM_GC_LOGGING_OPTS=`getGCLoggingOpts $gclog`
+FLINK_JVM_HEAPDUMP_OPTS=`getHeapdumpOpts $FLINK_HEAPDUMP_NAME $log`
Review comment:
```suggestion
FLINK_JVM_HEAPDUMP_OPTS=$(getHeapdumpOpts $FLINK_HEAPDUMP_NAME ${log})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341645497
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java
##
@@ -513,6 +514,76 @@ public static Configuration
cloneConfiguration(Configuration configuration) {
return clonedConfiguration;
}
+ /**
+* Format the default gc logging options
+* @param logDirectory to save the gc log
+* @return the formatted gc logging options string
+*/
+ public static String getGCLoggingOpts(String logDirectory) {
+return "-Xloggc:" + logDirectory + "/gc.log " +
+ "-XX:+PrintGCApplicationStoppedTime " +
+ "-XX:+PrintGCDetails " +
+ "-XX:+PrintGCDateStamps " +
+ "-XX:+UseGCLogFileRotation " +
+ "-XX:NumberOfGCLogFiles=10 " +
+ "-XX:GCLogFileSize=10M " +
+ "-XX:+PrintPromotionFailure " +
+ "-XX:+PrintGCCause";
+ }
+
+ /**
+* Format the default heapdump options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+* @param heapdumpDir to save heap dump file
+* @return the formatted heapdump options string
+*/
+ public static String getHeapdumpOpts(String appId, String ident, String
logDirectory, String heapdumpDir) {
+String dumpDestName = String.format("flink-%s-heapdump.hprof", ident);
+String dumpFileDestPath = new File(heapdumpDir, appId + "-" +
dumpDestName).getAbsolutePath();
+String oomScript = String.format("echo -e 'OutOfMemoryError! Killing
current process %%p...\n" +
+"Check gc logs and heapdump file(%s) for details.' > " +
+logDirectory + "/%s.err; " +
+"kill -9 %%p",
+ dumpFileDestPath, ident);
+return String.format("-XX:+HeapDumpOnOutOfMemoryError " +
+"-XX:HeapDumpPath=%s " +
+"-XX:OnOutOfMemoryError=\"%s\"",
+ dumpFileDestPath,
+ oomScript);
+ }
+
+ /**
+* Get the common jvm options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+ * @param conf flink configuration
+ */
+ public static String getCommonJvmOpts(
+ String appId,
+ String ident,
+ String logDirectory,
+ Configuration conf) {
+String commonOpts = "";
+ boolean enableGCLogging =
conf.getBoolean(CoreOptions.FLINK_JVM_DEFAULT_GC_LOGGING);
+if (enableGCLogging) {
+ // Add default gc logging options if enabled
+ commonOpts += getGCLoggingOpts(logDirectory);
+}
+ boolean enableHeapDump =
conf.getBoolean(CoreOptions.FLINK_JVM_HEAPDUMP_ON_OOM);
+if (enableHeapDump) {
+ // Add default heap dump options if enabled
+ String heapdumpDir =
conf.getString(CoreOptions.FLINK_JVM_HEAPDUMP_DIRECTORY);
+ commonOpts += " " + getHeapdumpOpts(appId, ident, logDirectory,
heapdumpDir);
+}
+if (conf.getString(CoreOptions.FLINK_JVM_OPTIONS).length() > 0) {
+ commonOpts += " " +
conf.getString(CoreOptions.FLINK_JVM_OPTIONS);
+}
+return commonOpts;
+ }
Review comment:
Also the formatting is off due to some lines being indented via tabs and
other via spaces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341645233
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java
##
@@ -513,6 +514,76 @@ public static Configuration
cloneConfiguration(Configuration configuration) {
return clonedConfiguration;
}
+ /**
+* Format the default gc logging options
+* @param logDirectory to save the gc log
+* @return the formatted gc logging options string
+*/
+ public static String getGCLoggingOpts(String logDirectory) {
+return "-Xloggc:" + logDirectory + "/gc.log " +
+ "-XX:+PrintGCApplicationStoppedTime " +
+ "-XX:+PrintGCDetails " +
+ "-XX:+PrintGCDateStamps " +
+ "-XX:+UseGCLogFileRotation " +
+ "-XX:NumberOfGCLogFiles=10 " +
+ "-XX:GCLogFileSize=10M " +
+ "-XX:+PrintPromotionFailure " +
+ "-XX:+PrintGCCause";
+ }
+
+ /**
+* Format the default heapdump options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+* @param heapdumpDir to save heap dump file
+* @return the formatted heapdump options string
+*/
+ public static String getHeapdumpOpts(String appId, String ident, String
logDirectory, String heapdumpDir) {
+String dumpDestName = String.format("flink-%s-heapdump.hprof", ident);
+String dumpFileDestPath = new File(heapdumpDir, appId + "-" +
dumpDestName).getAbsolutePath();
+String oomScript = String.format("echo -e 'OutOfMemoryError! Killing
current process %%p...\n" +
+"Check gc logs and heapdump file(%s) for details.' > " +
+logDirectory + "/%s.err; " +
+"kill -9 %%p",
+ dumpFileDestPath, ident);
+return String.format("-XX:+HeapDumpOnOutOfMemoryError " +
+"-XX:HeapDumpPath=%s " +
+"-XX:OnOutOfMemoryError=\"%s\"",
+ dumpFileDestPath,
+ oomScript);
+ }
Review comment:
Same here. Please use tabs for indentation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341642848
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/BootstrapToolsTest.java
##
@@ -140,16 +140,38 @@ public void testSubstituteConfigKeyPrefix() {
}
}
+ public static String getGCLoggingOpts(String logDirectory) {
+ return "-Xloggc:" + logDirectory + "/gc.log " +
+ "-XX:+PrintGCApplicationStoppedTime " +
+ "-XX:+PrintGCDetails " +
+ "-XX:+PrintGCDateStamps " +
+ "-XX:+UseGCLogFileRotation " +
+ "-XX:NumberOfGCLogFiles=10 " +
+ "-XX:GCLogFileSize=10M " +
+ "-XX:+PrintPromotionFailure " +
+ "-XX:+PrintGCCause";
+ }
+
+ public static String getHeapdumpOpts(String appId, String ident, String
logDirectory, String heapdumpDir) {
+ return String.format("-XX:+HeapDumpOnOutOfMemoryError "
+
+ "-XX:HeapDumpPath=%s/%s-flink-%s-heapdump.hprof
" +
+ "-XX:OnOutOfMemoryError=\"echo -e
'OutOfMemoryError! Killing current process %%p...\nCheck gc logs and heapdump
file(%s/%s-flink-%s-heapdump.hprof) for details.' > " +
+ "%s/%s.err; kill -9 %%p\"", heapdumpDir, appId,
ident, heapdumpDir, appId, ident, logDirectory, ident);
+ }
Review comment:
I guess we could simply use the methods in the `BootstrapTools` for this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341645591
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java
##
@@ -513,6 +514,76 @@ public static Configuration
cloneConfiguration(Configuration configuration) {
return clonedConfiguration;
}
+ /**
+* Format the default gc logging options
+* @param logDirectory to save the gc log
+* @return the formatted gc logging options string
+*/
+ public static String getGCLoggingOpts(String logDirectory) {
+return "-Xloggc:" + logDirectory + "/gc.log " +
+ "-XX:+PrintGCApplicationStoppedTime " +
+ "-XX:+PrintGCDetails " +
+ "-XX:+PrintGCDateStamps " +
+ "-XX:+UseGCLogFileRotation " +
+ "-XX:NumberOfGCLogFiles=10 " +
+ "-XX:GCLogFileSize=10M " +
+ "-XX:+PrintPromotionFailure " +
+ "-XX:+PrintGCCause";
+ }
+
+ /**
+* Format the default heapdump options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+* @param heapdumpDir to save heap dump file
+* @return the formatted heapdump options string
+*/
+ public static String getHeapdumpOpts(String appId, String ident, String
logDirectory, String heapdumpDir) {
+String dumpDestName = String.format("flink-%s-heapdump.hprof", ident);
+String dumpFileDestPath = new File(heapdumpDir, appId + "-" +
dumpDestName).getAbsolutePath();
+String oomScript = String.format("echo -e 'OutOfMemoryError! Killing
current process %%p...\n" +
+"Check gc logs and heapdump file(%s) for details.' > " +
+logDirectory + "/%s.err; " +
+"kill -9 %%p",
+ dumpFileDestPath, ident);
+return String.format("-XX:+HeapDumpOnOutOfMemoryError " +
+"-XX:HeapDumpPath=%s " +
+"-XX:OnOutOfMemoryError=\"%s\"",
+ dumpFileDestPath,
+ oomScript);
+ }
+
+ /**
+* Get the common jvm options
+* @param appId application id
+* @param ident the ident of the process, taskmanager/jobmanager
+* @param logDirectory to print some logs
+ * @param conf flink configuration
+ */
+ public static String getCommonJvmOpts(
+ String appId,
+ String ident,
+ String logDirectory,
+ Configuration conf) {
+String commonOpts = "";
+ boolean enableGCLogging =
conf.getBoolean(CoreOptions.FLINK_JVM_DEFAULT_GC_LOGGING);
+if (enableGCLogging) {
+ // Add default gc logging options if enabled
+ commonOpts += getGCLoggingOpts(logDirectory);
+}
+ boolean enableHeapDump =
conf.getBoolean(CoreOptions.FLINK_JVM_HEAPDUMP_ON_OOM);
+if (enableHeapDump) {
+ // Add default heap dump options if enabled
+ String heapdumpDir =
conf.getString(CoreOptions.FLINK_JVM_HEAPDUMP_DIRECTORY);
+ commonOpts += " " + getHeapdumpOpts(appId, ident, logDirectory,
heapdumpDir);
+}
+if (conf.getString(CoreOptions.FLINK_JVM_OPTIONS).length() > 0) {
Review comment:
```suggestion
if (!conf.getString(CoreOptions.FLINK_JVM_OPTIONS).isEmpty()) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341643736
##
File path: flink-dist/src/main/flink-bin/bin/flink-daemon.sh
##
@@ -85,6 +85,13 @@ id=$([ -f "$pid" ] && echo $(wc -l < "$pid") || echo "0")
FLINK_LOG_PREFIX="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-${DAEMON}-${id}-${HOSTNAME}"
log="${FLINK_LOG_PREFIX}.log"
out="${FLINK_LOG_PREFIX}.out"
+gclog="${FLINK_LOG_PREFIX}.gc_log"
+
+FLINK_HEAPDUMP_NAME="flink-${FLINK_IDENT_STRING}-${DAEMON}-${id}-${HOSTNAME}.hprof"
+rm -rf ${FLINK_JVM_HEAPDUMP_DIRECTORY}/${FLINK_HEAPDUMP_NAME}
+FLINK_JVM_GC_LOGGING_OPTS=`getGCLoggingOpts $gclog`
+FLINK_JVM_HEAPDUMP_OPTS=`getHeapdumpOpts $FLINK_HEAPDUMP_NAME $log`
Review comment:
```suggestion
FLINK_JVM_HEAPDUMP_OPTS=$(getHeapdumpOpts $FLINK_HEAPDUMP_NAME ${log})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341645141
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java
##
@@ -513,6 +514,76 @@ public static Configuration
cloneConfiguration(Configuration configuration) {
return clonedConfiguration;
}
+ /**
+* Format the default gc logging options
+* @param logDirectory to save the gc log
+* @return the formatted gc logging options string
+*/
+ public static String getGCLoggingOpts(String logDirectory) {
+return "-Xloggc:" + logDirectory + "/gc.log " +
+ "-XX:+PrintGCApplicationStoppedTime " +
+ "-XX:+PrintGCDetails " +
+ "-XX:+PrintGCDateStamps " +
+ "-XX:+UseGCLogFileRotation " +
+ "-XX:NumberOfGCLogFiles=10 " +
+ "-XX:GCLogFileSize=10M " +
+ "-XX:+PrintPromotionFailure " +
+ "-XX:+PrintGCCause";
+ }
Review comment:
Indentation with spaces. Please use tabs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341643899
##
File path: flink-dist/src/main/flink-bin/mesos-bin/mesos-appmaster-job.sh
##
@@ -31,6 +31,13 @@ CC_CLASSPATH=`manglePathList
$(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPA
log="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.log"
log_setting="-Dlog.file="$log"
-Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j.properties
-Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml"
+gclog="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.gc_log"
+
+FLINK_HEAPDUMP_NAME="flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.hprof"
+rm -rf ${FLINK_JVM_HEAPDUMP_DIRECTORY}/${FLINK_HEAPDUMP_NAME}
+FLINK_JVM_GC_LOGGING_OPTS=`getGCLoggingOpts $gclog`
Review comment:
```suggestion
FLINK_JVM_GC_LOGGING_OPTS=$(getGCLoggingOpts ${gclog})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341644052
##
File path: flink-dist/src/main/flink-bin/mesos-bin/mesos-appmaster.sh
##
@@ -31,6 +31,13 @@ CC_CLASSPATH=`manglePathList
$(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPA
log="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.log"
log_setting="-Dlog.file="$log"
-Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j.properties
-Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml"
+gclog="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.gc_log"
+
+FLINK_HEAPDUMP_NAME="flink-${FLINK_IDENT_STRING}-mesos-appmaster-${HOSTNAME}.hprof"
+rm -rf ${FLINK_JVM_HEAPDUMP_DIRECTORY}/${FLINK_HEAPDUMP_NAME}
+FLINK_JVM_GC_LOGGING_OPTS=`getGCLoggingOpts $gclog`
Review comment:
```suggestion
FLINK_JVM_GC_LOGGING_OPTS=$(getGCLoggingOpts ${gclog})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add default GC options for flink on yarn to facilitate debugging
tillrohrmann commented on a change in pull request #9703: [FLINK-14038]Add
default GC options for flink on yarn to facilitate debugging
URL: https://github.com/apache/flink/pull/9703#discussion_r341643797
##
File path: flink-dist/src/main/flink-bin/bin/flink-daemon.sh
##
@@ -85,6 +85,13 @@ id=$([ -f "$pid" ] && echo $(wc -l < "$pid") || echo "0")
FLINK_LOG_PREFIX="${FLINK_LOG_DIR}/flink-${FLINK_IDENT_STRING}-${DAEMON}-${id}-${HOSTNAME}"
log="${FLINK_LOG_PREFIX}.log"
out="${FLINK_LOG_PREFIX}.out"
+gclog="${FLINK_LOG_PREFIX}.gc_log"
+
+FLINK_HEAPDUMP_NAME="flink-${FLINK_IDENT_STRING}-${DAEMON}-${id}-${HOSTNAME}.hprof"
+rm -rf ${FLINK_JVM_HEAPDUMP_DIRECTORY}/${FLINK_HEAPDUMP_NAME}
+FLINK_JVM_GC_LOGGING_OPTS=`getGCLoggingOpts $gclog`
Review comment:
```suggestion
FLINK_JVM_GC_LOGGING_OPTS=$(getGCLoggingOpts ${gclog})
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-14344) Snapshot master hook state asynchronously
[
https://issues.apache.org/jira/browse/FLINK-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964939#comment-16964939
]
Biao Liu commented on FLINK-14344:
--
Hi [~pnowojski],
{quote}this is `PublicEvolving` class, so we are allowed to change it.
{quote}
Yes we could. I'm just not sure if it's designed to support feature like this,
initializing an external system before snapshotting task state. Maybe the
original author [~sewen] could give us more background informations?
The Peavega connector maintainer gave me a response that my guess is right.
They executes the `triggerCheckpoint` asynchronously. I have left another
message to ask them if they depend on the sequential order of triggering master
hook and snapshotting task state.
{quote}how difficult would it be to provide the 2nd semantic? Would dropping
the synchronous hooks support and providing just the 1st semantic be
significantly easier to write/support/maintain?
{quote}
Based on my POC, it's not much harder to provide the 2nd semantic from the
perspective of implementation.
If it's a kind of "pure" master state, not master hook of
{{ExternallyInducedSource}}, I think I'll choose the 1st semantic. There should
be no dependence between master state and task state. But the semantic of
master hook is somewhat special, you know, I tend to leave it alone (at-least
before we have thought about it clearly). Of course, we'd better wait a bit for
the feedback of Pervega connector maintainers.
> Snapshot master hook state asynchronously
> -
>
> Key: FLINK-14344
> URL: https://issues.apache.org/jira/browse/FLINK-14344
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently we snapshot the master hook state synchronously. As a part of
> reworking threading model of {{CheckpointCoordinator}}, we have to make this
> non-blocking to satisfy the requirement of running in main thread.
> The behavior of snapshotting master hook state is similar to task state
> snapshotting. Master state snapshotting is taken before task state
> snapshotting. Because in master hook, there might be external system
> initialization which task state snapshotting might depend on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10067: [FLINK-14375][runtime] Avoid to notify scheduler about fake or outdated state update
flinkbot edited a comment on issue #10067: [FLINK-14375][runtime] Avoid to notify scheduler about fake or outdated state update URL: https://github.com/apache/flink/pull/10067#issuecomment-548812084 ## CI report: * d3341d9f0bff2cb97bd0b5bf507725ae6d670b88 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134571714) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14593) Port ClusterClient to asynchronous interface version
[ https://issues.apache.org/jira/browse/FLINK-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-14593: -- Summary: Port ClusterClient to asynchronous interface version (was: Make ClusterClient an interface) > Port ClusterClient to asynchronous interface version > > > Key: FLINK-14593 > URL: https://issues.apache.org/jira/browse/FLINK-14593 > Project: Flink > Issue Type: Sub-task > Components: Client / Job Submission >Reporter: Zili Chen >Assignee: Zili Chen >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-14593) Make ClusterClient an interface
Zili Chen created FLINK-14593: - Summary: Make ClusterClient an interface Key: FLINK-14593 URL: https://issues.apache.org/jira/browse/FLINK-14593 Project: Flink Issue Type: Sub-task Components: Client / Job Submission Reporter: Zili Chen Assignee: Zili Chen Fix For: 1.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-5339) Ensure Flink works with Log4j version 2
[ https://issues.apache.org/jira/browse/FLINK-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964929#comment-16964929 ] Georgi Stoyanov commented on FLINK-5339: [~coolder] could you share the steps to integrate log4j2 as a logging framework in flink? We are using slf4j now and we want to migrate > Ensure Flink works with Log4j version 2 > --- > > Key: FLINK-5339 > URL: https://issues.apache.org/jira/browse/FLINK-5339 > Project: Flink > Issue Type: Improvement > Components: Build System, Documentation >Reporter: Robert Metzger >Priority: Major > > Flink supports using different logging backends. > We currently use Log4j 1.2 by default and there is a documentation page for > using Flink with Logback. > With this JIRA, I would like to make all required changes to support Log4j 2 > and document how users can set up Flink to use that logging backend. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14565) Shutdown SystemResourcesCounter on (JM|TM)MetricGroup closed
[ https://issues.apache.org/jira/browse/FLINK-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-14565: - Assignee: (was: Zili Chen) > Shutdown SystemResourcesCounter on (JM|TM)MetricGroup closed > > > Key: FLINK-14565 > URL: https://issues.apache.org/jira/browse/FLINK-14565 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Reporter: Zili Chen >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently, we start SystemResourcesCounter when initialize > (JM|TM)MetricGroup. This thread doesn't exit on (JM|TM)MetricGroup closed and > even there is not exit logic of them. > It possibly causes thread leak. For example, on our platform which supports > previewing sample SQL execution, it starts a MiniCluster in the same process > as the platform. When the preview job finished MiniCluster closed and also > (JM|TM)MetricGroup. However these SystemResourcesCounter threads remain. > I propose when creating SystemResourcesCounter, track it in > (JM|TM)MetricGroup, and on (JM|TM)MetricGroup closed, shutdown > SystemResourcesCounter. This way, we survive from thread leaks. > CC [~chesnay] [~trohrmann] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TisonKun commented on issue #10056: [FLINK-14565][metric] Shut down SystemResourcesCounter on Metric closed to prevent thread leak
TisonKun commented on issue #10056: [FLINK-14565][metric] Shut down SystemResourcesCounter on Metric closed to prevent thread leak URL: https://github.com/apache/flink/pull/10056#issuecomment-548843806 Closed as non-consensus. Thanks for your review @tillrohrmann ! I think @zentol is working on this issue as described in JIRA :-) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun closed pull request #10056: [FLINK-14565][metric] Shut down SystemResourcesCounter on Metric closed to prevent thread leak
TisonKun closed pull request #10056: [FLINK-14565][metric] Shut down SystemResourcesCounter on Metric closed to prevent thread leak URL: https://github.com/apache/flink/pull/10056 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever
flinkbot edited a comment on issue #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever URL: https://github.com/apache/flink/pull/9950#issuecomment-544414033 ## CI report: * decba8623c001e8bbe5dc797de2bc421b2e216ef : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132777340) * 6c7ffb9f9049016ce874b9790cd51ed337cfec00 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132974469) * f381f5765ae1a654d545aff4c679eaf9e4812ddc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132985132) * bdd0016a7387ee18c2b9e7aea879063426065427 : UNKNOWN * 08cc39399fd916635b26ed3da9993d9cb57cb680 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133366893) * 237d6084269371b165908a4e657d7c9fec75155c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133381142) * ba6c666f591a4cbf21f020997225938c8079a1be : UNKNOWN * 41471d53c691b2a01a2c43fa431ab6a65a6c1bab : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133506229) * 8fb4f6710f9363d9326603e64276241013a2ff4f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133821363) * de9bf092853b0d181c97e4eeaffd3b6aed67c317 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134430208) * 6b95831e382b336b7b2025c629bab8f3703bb2ba : UNKNOWN * 7f3ff0ad6096f3718ebfeb9a9cdf598a842c1715 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134571751) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-14464) Introduce the 'AbstractUserClassPathJobGraphRetriever' which could construct the 'user code class path' from the "job" dir.
[ https://issues.apache.org/jira/browse/FLINK-14464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann resolved FLINK-14464. --- Fix Version/s: 1.10.0 Resolution: Done Done via 1fa4e3dd3e3992111a955508ac1bc1640209c51e 9e13af6b4acb1e3a5505bed75053a2be05299a88 > Introduce the 'AbstractUserClassPathJobGraphRetriever' which could construct > the 'user code class path' from the "job" dir. > --- > > Key: FLINK-14464 > URL: https://issues.apache.org/jira/browse/FLINK-14464 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14464) Introduce the 'AbstractUserClassPathJobGraphRetriever' which could construct the 'user code class path' from the "job" dir.
[ https://issues.apache.org/jira/browse/FLINK-14464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-14464: --- Labels: pull-request-available (was: ) > Introduce the 'AbstractUserClassPathJobGraphRetriever' which could construct > the 'user code class path' from the "job" dir. > --- > > Key: FLINK-14464 > URL: https://issues.apache.org/jira/browse/FLINK-14464 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] tillrohrmann closed pull request #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever
tillrohrmann closed pull request #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever URL: https://github.com/apache/flink/pull/9950 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10068: [FLINK-13702][FLINK-13740] Fixed issues with generic types and materialization of lazy binary formats in blink planner
flinkbot edited a comment on issue #10068: [FLINK-13702][FLINK-13740] Fixed issues with generic types and materialization of lazy binary formats in blink planner URL: https://github.com/apache/flink/pull/10068#issuecomment-548812127 ## CI report: * 46b1c66fda1da82789d513c67e82b39be041afaa : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134571723) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10056: [FLINK-14565][metric] Shut down SystemResourcesCounter on Metric closed to prevent thread leak
tillrohrmann commented on a change in pull request #10056:
[FLINK-14565][metric] Shut down SystemResourcesCounter on Metric closed to
prevent thread leak
URL: https://github.com/apache/flink/pull/10056#discussion_r341628869
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/ProcessMetricGroup.java
##
@@ -18,18 +18,40 @@
package org.apache.flink.runtime.metrics.groups;
+import org.apache.flink.api.common.time.Time;
import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.util.SystemResourcesCounter;
/**
* {@link AbstractImitatingJobManagerMetricGroup} implementation for process
related metrics.
*/
public class ProcessMetricGroup extends AbstractImitatingJobManagerMetricGroup
{
+ private volatile SystemResourcesCounter systemResourcesCounter;
Review comment:
I think this is not the right place. I would rather keep this part out of
the `MetricGroup`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10027: [FLINK-14549][Table SQL/Planner] Bring more detail by using logicalTy…
flinkbot edited a comment on issue #10027: [FLINK-14549][Table SQL/Planner] Bring more detail by using logicalTy… URL: https://github.com/apache/flink/pull/10027#issuecomment-547346216 ## CI report: * a36cfd4c31b55aca1b05afe59025c2c2d51a893a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133973282) * cb8fd65958950dc26548fdac2c603f1d745089ff : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/133987033) * ec4b87db3dd2a657bd9008befc872a7c0704e0fe : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/133990871) * fdf2798461cf27243bf6c2eb575fea7ced488b42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133994349) * 45ddcc3eb65176473f85c64910deaf7ed6516d90 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134576569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14022) Add validation check for places where Python ScalarFunction cannot be used
[ https://issues.apache.org/jira/browse/FLINK-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964912#comment-16964912 ] Hequn Cheng commented on FLINK-14022: - Resolved in 1.10.0 via 22930e379d8d4557f5a960691b9b0d676f5c87d4..84f830b4f7bcb52ef6f94c96182c95aeffcd3a72 > Add validation check for places where Python ScalarFunction cannot be used > -- > > Key: FLINK-14022 > URL: https://issues.apache.org/jira/browse/FLINK-14022 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Dian Fu >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, there are places where Python ScalarFunction could not be used, > for example: > # Python UDF could not be used in MatchRecognize > # Python UDFs could not be used in Join condition which take the columns > from both the left table and the right table as inputs > We should add validation check for places where it’s not supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14022) Add validation check for places where Python ScalarFunction cannot be used
[ https://issues.apache.org/jira/browse/FLINK-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng closed FLINK-14022. --- Resolution: Resolved > Add validation check for places where Python ScalarFunction cannot be used > -- > > Key: FLINK-14022 > URL: https://issues.apache.org/jira/browse/FLINK-14022 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Dian Fu >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, there are places where Python ScalarFunction could not be used, > for example: > # Python UDF could not be used in MatchRecognize > # Python UDFs could not be used in Join condition which take the columns > from both the left table and the right table as inputs > We should add validation check for places where it’s not supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 closed pull request #9969: [FLINK-14022][table-planner][table-planner-blink] Add validation check for places where Python ScalarFunction cannot be used
hequn8128 closed pull request #9969: [FLINK-14022][table-planner][table-planner-blink] Add validation check for places where Python ScalarFunction cannot be used URL: https://github.com/apache/flink/pull/9969 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-14583) Remove progressLock from ExecutionGraph
[
https://issues.apache.org/jira/browse/FLINK-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-14583:
-
Assignee: vinoyang
> Remove progressLock from ExecutionGraph
> ---
>
> Key: FLINK-14583
> URL: https://issues.apache.org/jira/browse/FLINK-14583
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There is a lock object named {{progressLock}} which used to secure all access
> to mutable fields, especially the tracking of progress within the job. Now,
> the {{ExecutionGraph}} can only be accessed by a single thread at any time
> point. IMO, we can remove this lock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964903#comment-16964903
]
Till Rohrmann commented on FLINK-14566:
---
Let's make this then a low priority improvement which do once the rest of the
feature has been completed.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-14566:
--
Priority: Minor (was: Major)
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Minor
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> I'd propose to define special values of
> onHeapManagedMemory/offHeapManagedMemory
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed
> memory is used.
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on
> heap managed memory is used, while onHeapManagedMemory==null means not. For
> non-UNKNOWN resource spec, managed memory size larger than zero indicates it
> is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters:
> setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for
> non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages
> for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-14536) Make clear the way to aggregate specified cpuCores resources
[
https://issues.apache.org/jira/browse/FLINK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-14536:
-
Assignee: Zhu Zhu
> Make clear the way to aggregate specified cpuCores resources
>
>
> Key: FLINK-14536
> URL: https://issues.apache.org/jira/browse/FLINK-14536
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> I'm raising this question because I find {{cpuCores}} in
> {{ResourceSpec#merge}} are aggregated with {{max()}}, while in
> {{ResourceProfile#merge}} it is {{sum()}}.
> This means that when calculating resources of a vertex from within operators,
> the {{cpuCores}} is the max value. While it is a sum(or subtraction in
> {{ResourceProfile#subtract}}) when dealing with shared slot bookkeeping and
> related checks.
> This is confusing to me, especially when I'm considering how to generate a
> shared slot resource spec merged from all vertices in it(see FLINK-14314).
> I'm not pretty sure if we already have a concise definition for it?
> If there is, we need to respect it and change {{ResourceSpec}} or
> {{ResourceProfile}} accordingly.
> If not, we need to decide it first before we can move on with fine grained
> resources.
> Need to mention that if we take the {{max()}} way, we need to re-consider how
> we can conduct a correct {{ResourceProfile#subtract}} regarding {{cpuCores}},
> since there is not a clear way to reverse a {{max()}} computation.
> cc [~trohrmann] [~xintongsong] [~azagrebin]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14536) Make clear the way to aggregate specified cpuCores resources
[
https://issues.apache.org/jira/browse/FLINK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964901#comment-16964901
]
Till Rohrmann commented on FLINK-14536:
---
I've assigned the ticket to you [~zhuzh].
> Make clear the way to aggregate specified cpuCores resources
>
>
> Key: FLINK-14536
> URL: https://issues.apache.org/jira/browse/FLINK-14536
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Fix For: 1.10.0
>
>
> I'm raising this question because I find {{cpuCores}} in
> {{ResourceSpec#merge}} are aggregated with {{max()}}, while in
> {{ResourceProfile#merge}} it is {{sum()}}.
> This means that when calculating resources of a vertex from within operators,
> the {{cpuCores}} is the max value. While it is a sum(or subtraction in
> {{ResourceProfile#subtract}}) when dealing with shared slot bookkeeping and
> related checks.
> This is confusing to me, especially when I'm considering how to generate a
> shared slot resource spec merged from all vertices in it(see FLINK-14314).
> I'm not pretty sure if we already have a concise definition for it?
> If there is, we need to respect it and change {{ResourceSpec}} or
> {{ResourceProfile}} accordingly.
> If not, we need to decide it first before we can move on with fine grained
> resources.
> Need to mention that if we take the {{max()}} way, we need to re-consider how
> we can conduct a correct {{ResourceProfile#subtract}} regarding {{cpuCores}},
> since there is not a clear way to reverse a {{max()}} computation.
> cc [~trohrmann] [~xintongsong] [~azagrebin]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13969) Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-13969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964897#comment-16964897
]
Till Rohrmann commented on FLINK-13969:
---
Then this sounds like a bug. We should also do the check in step 3.
> Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end
> test fails on Travis
> -
>
> Key: FLINK-13969
> URL: https://issues.apache.org/jira/browse/FLINK-13969
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The {{Resuming Externalized Checkpoint (rocks, incremental, scale down)}}
> end-to-end test fails on Travis because its log contains an exception
> {code}
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete
> snapshot 16 for operator ArtificalKeyedStateMapper_Avro ->
> ArtificalOperatorStateMapper (2/4). Failure reason: Checkpoint was declined.
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:431)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1302)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1236)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:892)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:797)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:728)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:88)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:177)
> at
> org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:155)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.pollNextNullable(StreamTaskNetworkInput.java:118)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.pollNextNullable(StreamTaskNetworkInput.java:48)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:144)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:277)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.execution.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:147)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:404)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Cannot register Closeable, registry is
> already closed. Closing argument.
> at
> org.apache.flink.util.AbstractCloseableRegistry.registerCloseable(AbstractCloseableRegistry.java:85)
> at
> org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.(AsyncSnapshotCallable.java:122)
> at
> org.apache.flink.runtime.state.AsyncSnapshotCallable$AsyncSnapshotTask.(AsyncSnapshotCallable.java:110)
> at
> org.apache.flink.runtime.state.AsyncSnapshotCallable.toAsyncSnapshotFutureTask(AsyncSnapshotCallable.java:104)
> at
> org.apache.flink.contrib.streaming.state.snapshot.RocksIncrementalSnapshotStrategy.doSnapshot(RocksIncrementalSnapshotStrategy.java:170)
> at
> org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.snapshot(RocksDBSnapshotStrategyBase.java:126)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:439)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:411)
> ... 17 more
> {code}
> https://api.travis-ci.org/v3/job/580915660/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever
flinkbot edited a comment on issue #9950: [FLINK-14464][runtime] Introduce the AbstractUserClassPathJobGraphRetriever URL: https://github.com/apache/flink/pull/9950#issuecomment-544414033 ## CI report: * decba8623c001e8bbe5dc797de2bc421b2e216ef : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132777340) * 6c7ffb9f9049016ce874b9790cd51ed337cfec00 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132974469) * f381f5765ae1a654d545aff4c679eaf9e4812ddc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132985132) * bdd0016a7387ee18c2b9e7aea879063426065427 : UNKNOWN * 08cc39399fd916635b26ed3da9993d9cb57cb680 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133366893) * 237d6084269371b165908a4e657d7c9fec75155c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133381142) * ba6c666f591a4cbf21f020997225938c8079a1be : UNKNOWN * 41471d53c691b2a01a2c43fa431ab6a65a6c1bab : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133506229) * 8fb4f6710f9363d9326603e64276241013a2ff4f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133821363) * de9bf092853b0d181c97e4eeaffd3b6aed67c317 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134430208) * 6b95831e382b336b7b2025c629bab8f3703bb2ba : UNKNOWN * 7f3ff0ad6096f3718ebfeb9a9cdf598a842c1715 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10066: [FLINK-14586][coordination] JM issues promote calls on successful job
tillrohrmann commented on a change in pull request #10066:
[FLINK-14586][coordination] JM issues promote calls on successful job
URL: https://github.com/apache/flink/pull/10066#discussion_r341611565
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterPartitionReleaseTest.java
##
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.checkpoint.StandaloneCheckpointRecoveryFactory;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.heartbeat.HeartbeatServices;
+import
org.apache.flink.runtime.highavailability.TestingHighAvailabilityServices;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionID;
+import
org.apache.flink.runtime.io.network.partition.TestingJobMasterPartitionTracker;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobmaster.utils.JobMasterBuilder;
+import org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
+import org.apache.flink.runtime.messages.Acknowledge;
+import
org.apache.flink.runtime.resourcemanager.utils.TestingResourceManagerGateway;
+import org.apache.flink.runtime.rpc.RpcUtils;
+import org.apache.flink.runtime.rpc.TestingRpcService;
+import org.apache.flink.runtime.taskexecutor.TaskExecutorGateway;
+import org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGateway;
+import org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGatewayBuilder;
+import org.apache.flink.runtime.taskexecutor.slot.SlotOffer;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskExecutionState;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.runtime.testtasks.NoOpInvokable;
+import org.apache.flink.runtime.util.TestingFatalErrorHandler;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+
+/**
+ * Tests for the partition release logic of the {@link JobMaster}.
+ */
+public class JobMasterPartitionReleaseTest extends TestLogger {
+
+ @ClassRule
+ public static TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private static final Time testingTimeout = Time.seconds(10L);
+
+ private static final long heartbeatInterval = 1000L;
+ private static final long heartbeatTimeout = 5_000_000L;
+
+ private static TestingRpcService rpcService;
+
+ private static HeartbeatServices heartbeatServices;
+
+ private Configuration configuration;
+
+ private JobMasterId jobMasterId;
+
+ private TestingHighAvailabilityServices haServices;
+
+ private SettableLeaderRetrievalService rmLeaderRetrievalService;
+
+ private TestingFatalErrorHandler testingFatalErrorHandler;
+
+ @BeforeClass
+ public static void setupClass() {
+
[GitHub] [flink] tillrohrmann commented on a change in pull request #10066: [FLINK-14586][coordination] JM issues promote calls on successful job
tillrohrmann commented on a change in pull request #10066: [FLINK-14586][coordination] JM issues promote calls on successful job URL: https://github.com/apache/flink/pull/10066#discussion_r341609645 ## File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/utils/JobMasterBuilder.java ## @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.jobmaster.utils; + +import org.apache.flink.configuration.Configuration; +import org.apache.flink.runtime.checkpoint.StandaloneCheckpointRecoveryFactory; +import org.apache.flink.runtime.clusterframework.types.ResourceID; +import org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph; +import org.apache.flink.runtime.heartbeat.HeartbeatServices; +import org.apache.flink.runtime.highavailability.HighAvailabilityServices; +import org.apache.flink.runtime.highavailability.TestingHighAvailabilityServices; +import org.apache.flink.runtime.io.network.partition.NoOpJobMasterPartitionTracker; +import org.apache.flink.runtime.io.network.partition.PartitionTrackerFactory; +import org.apache.flink.runtime.jobgraph.JobGraph; +import org.apache.flink.runtime.jobmanager.OnCompletionActions; +import org.apache.flink.runtime.jobmaster.JobManagerSharedServices; +import org.apache.flink.runtime.jobmaster.JobMaster; +import org.apache.flink.runtime.jobmaster.JobMasterConfiguration; +import org.apache.flink.runtime.jobmaster.TestingJobManagerSharedServicesBuilder; +import org.apache.flink.runtime.jobmaster.factories.UnregisteredJobManagerJobMetricGroupFactory; +import org.apache.flink.runtime.jobmaster.slotpool.DefaultSchedulerFactory; +import org.apache.flink.runtime.jobmaster.slotpool.DefaultSlotPoolFactory; +import org.apache.flink.runtime.jobmaster.slotpool.SchedulerFactory; +import org.apache.flink.runtime.jobmaster.slotpool.SlotPoolFactory; +import org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService; +import org.apache.flink.runtime.rpc.FatalErrorHandler; +import org.apache.flink.runtime.rpc.RpcService; +import org.apache.flink.runtime.scheduler.LegacySchedulerFactory; +import org.apache.flink.runtime.scheduler.SchedulerNGFactory; +import org.apache.flink.runtime.shuffle.NettyShuffleMaster; +import org.apache.flink.runtime.shuffle.ShuffleMaster; + +import java.util.concurrent.CompletableFuture; + +/** + * TODO: Add javadoc. Review comment: todo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10066: [FLINK-14586][coordination] JM issues promote calls on successful job
tillrohrmann commented on a change in pull request #10066:
[FLINK-14586][coordination] JM issues promote calls on successful job
URL: https://github.com/apache/flink/pull/10066#discussion_r341610510
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/utils/JobMasterBuilder.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.utils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.checkpoint.StandaloneCheckpointRecoveryFactory;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph;
+import org.apache.flink.runtime.heartbeat.HeartbeatServices;
+import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
+import
org.apache.flink.runtime.highavailability.TestingHighAvailabilityServices;
+import
org.apache.flink.runtime.io.network.partition.NoOpJobMasterPartitionTracker;
+import org.apache.flink.runtime.io.network.partition.PartitionTrackerFactory;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobmanager.OnCompletionActions;
+import org.apache.flink.runtime.jobmaster.JobManagerSharedServices;
+import org.apache.flink.runtime.jobmaster.JobMaster;
+import org.apache.flink.runtime.jobmaster.JobMasterConfiguration;
+import
org.apache.flink.runtime.jobmaster.TestingJobManagerSharedServicesBuilder;
+import
org.apache.flink.runtime.jobmaster.factories.UnregisteredJobManagerJobMetricGroupFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.DefaultSchedulerFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.DefaultSlotPoolFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.SchedulerFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotPoolFactory;
+import org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.rpc.RpcService;
+import org.apache.flink.runtime.scheduler.LegacySchedulerFactory;
+import org.apache.flink.runtime.scheduler.SchedulerNGFactory;
+import org.apache.flink.runtime.shuffle.NettyShuffleMaster;
+import org.apache.flink.runtime.shuffle.ShuffleMaster;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * TODO: Add javadoc.
+ */
+public class JobMasterBuilder {
+
+ private static final long heartbeatInterval = 1000L;
+ private static final long heartbeatTimeout = 5_000_000L;
+ private static final HeartbeatServices DEFAULT_HEARTBEAT_SERVICES = new
HeartbeatServices(heartbeatInterval, heartbeatTimeout);
+
+ private Configuration configuration = new Configuration();
+
+ private final JobGraph jobGraph;
+ private final RpcService rpcService;
+
+ private HighAvailabilityServices highAvailabilityServices;
+
+ private JobManagerSharedServices jobManagerSharedServices = new
TestingJobManagerSharedServicesBuilder().build();
+
+ private HeartbeatServices heartbeatServices =
DEFAULT_HEARTBEAT_SERVICES;
+
+ private SlotPoolFactory slotPoolFactory = null;
+
+ private SchedulerFactory schedulerFactory = null;
+
+ private SchedulerNGFactory schedulerNGFactory = null;
+
+ private OnCompletionActions onCompletionActions = new
TestingOnCompletionActions();
+
+ private ShuffleMaster shuffleMaster = NettyShuffleMaster.INSTANCE;
+
+ private PartitionTrackerFactory partitionTrackerFactory =
NoOpJobMasterPartitionTracker.FACTORY;
+
+ private ResourceID jmResourceId = ResourceID.generate();
+
+ private FatalErrorHandler fatalErrorHandler = error -> {
+ };
+
+ public JobMasterBuilder(JobGraph jobGraph, RpcService rpcService) {
+ TestingHighAvailabilityServices testingHighAvailabilityServices
= new TestingHighAvailabilityServices();
+
testingHighAvailabilityServices.setCheckpointRecoveryFactory(new
StandaloneCheckpointRecoveryFactory());
+
+ SettableLeaderRetrievalService rmLeaderRetrievalService = new
SettableLeaderRetrievalService(
+
[GitHub] [flink] tillrohrmann commented on a change in pull request #10066: [FLINK-14586][coordination] JM issues promote calls on successful job
tillrohrmann commented on a change in pull request #10066:
[FLINK-14586][coordination] JM issues promote calls on successful job
URL: https://github.com/apache/flink/pull/10066#discussion_r341610559
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/utils/JobMasterBuilder.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.utils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.checkpoint.StandaloneCheckpointRecoveryFactory;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.executiongraph.ArchivedExecutionGraph;
+import org.apache.flink.runtime.heartbeat.HeartbeatServices;
+import org.apache.flink.runtime.highavailability.HighAvailabilityServices;
+import
org.apache.flink.runtime.highavailability.TestingHighAvailabilityServices;
+import
org.apache.flink.runtime.io.network.partition.NoOpJobMasterPartitionTracker;
+import org.apache.flink.runtime.io.network.partition.PartitionTrackerFactory;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobmanager.OnCompletionActions;
+import org.apache.flink.runtime.jobmaster.JobManagerSharedServices;
+import org.apache.flink.runtime.jobmaster.JobMaster;
+import org.apache.flink.runtime.jobmaster.JobMasterConfiguration;
+import
org.apache.flink.runtime.jobmaster.TestingJobManagerSharedServicesBuilder;
+import
org.apache.flink.runtime.jobmaster.factories.UnregisteredJobManagerJobMetricGroupFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.DefaultSchedulerFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.DefaultSlotPoolFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.SchedulerFactory;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotPoolFactory;
+import org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
+import org.apache.flink.runtime.rpc.FatalErrorHandler;
+import org.apache.flink.runtime.rpc.RpcService;
+import org.apache.flink.runtime.scheduler.LegacySchedulerFactory;
+import org.apache.flink.runtime.scheduler.SchedulerNGFactory;
+import org.apache.flink.runtime.shuffle.NettyShuffleMaster;
+import org.apache.flink.runtime.shuffle.ShuffleMaster;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * TODO: Add javadoc.
+ */
+public class JobMasterBuilder {
+
+ private static final long heartbeatInterval = 1000L;
+ private static final long heartbeatTimeout = 5_000_000L;
+ private static final HeartbeatServices DEFAULT_HEARTBEAT_SERVICES = new
HeartbeatServices(heartbeatInterval, heartbeatTimeout);
+
+ private Configuration configuration = new Configuration();
+
+ private final JobGraph jobGraph;
+ private final RpcService rpcService;
+
+ private HighAvailabilityServices highAvailabilityServices;
+
+ private JobManagerSharedServices jobManagerSharedServices = new
TestingJobManagerSharedServicesBuilder().build();
+
+ private HeartbeatServices heartbeatServices =
DEFAULT_HEARTBEAT_SERVICES;
+
+ private SlotPoolFactory slotPoolFactory = null;
+
+ private SchedulerFactory schedulerFactory = null;
+
+ private SchedulerNGFactory schedulerNGFactory = null;
+
+ private OnCompletionActions onCompletionActions = new
TestingOnCompletionActions();
+
+ private ShuffleMaster shuffleMaster = NettyShuffleMaster.INSTANCE;
+
+ private PartitionTrackerFactory partitionTrackerFactory =
NoOpJobMasterPartitionTracker.FACTORY;
+
+ private ResourceID jmResourceId = ResourceID.generate();
+
+ private FatalErrorHandler fatalErrorHandler = error -> {
+ };
+
+ public JobMasterBuilder(JobGraph jobGraph, RpcService rpcService) {
+ TestingHighAvailabilityServices testingHighAvailabilityServices
= new TestingHighAvailabilityServices();
+
testingHighAvailabilityServices.setCheckpointRecoveryFactory(new
StandaloneCheckpointRecoveryFactory());
+
+ SettableLeaderRetrievalService rmLeaderRetrievalService = new
SettableLeaderRetrievalService(
+
[GitHub] [flink] flinkbot edited a comment on issue #10067: [FLINK-14375][runtime] Avoid to notify scheduler about fake or outdated state update
flinkbot edited a comment on issue #10067: [FLINK-14375][runtime] Avoid to notify scheduler about fake or outdated state update URL: https://github.com/apache/flink/pull/10067#issuecomment-548812084 ## CI report: * d3341d9f0bff2cb97bd0b5bf507725ae6d670b88 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134571714) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10068: [FLINK-13702][FLINK-13740] Fixed issues with generic types and materialization of lazy binary formats in blink planner
flinkbot edited a comment on issue #10068: [FLINK-13702][FLINK-13740] Fixed issues with generic types and materialization of lazy binary formats in blink planner URL: https://github.com/apache/flink/pull/10068#issuecomment-548812127 ## CI report: * 46b1c66fda1da82789d513c67e82b39be041afaa : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134571723) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10034: [FLINK-14400] Shrink scope of MemoryManager from TaskExecutor to slot
flinkbot edited a comment on issue #10034: [FLINK-14400] Shrink scope of MemoryManager from TaskExecutor to slot URL: https://github.com/apache/flink/pull/10034#issuecomment-547440502 ## CI report: * 85d93043d366498b8bdaed5351b74ca47c77d575 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134010434) * 6dd1e38a230348bb9e8b489c6f99a6a125cd2cc8 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134406685) * af1c6979c7139bd621da8f81f694dae93eca5141 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134530779) * 7158595bad2fa7fb563fab4ead11f7b98251df66 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134541262) * 2b8ee53ce0331ec08ee191ee9439c0370305498f : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134560324) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10027: [FLINK-14549][Table SQL/Planner] Bring more detail by using logicalTy…
flinkbot edited a comment on issue #10027: [FLINK-14549][Table SQL/Planner] Bring more detail by using logicalTy… URL: https://github.com/apache/flink/pull/10027#issuecomment-547346216 ## CI report: * a36cfd4c31b55aca1b05afe59025c2c2d51a893a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133973282) * cb8fd65958950dc26548fdac2c603f1d745089ff : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/133987033) * ec4b87db3dd2a657bd9008befc872a7c0704e0fe : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/133990871) * fdf2798461cf27243bf6c2eb575fea7ced488b42 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133994349) * 45ddcc3eb65176473f85c64910deaf7ed6516d90 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
