[jira] [Closed] (FLINK-5452) Make table unit tests pass under cluster mode
[ https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-5452. Resolution: Fixed Fix Version/s: 1.3.0 1.2.0 Fixed for 1.2.0 with 7f7692619b0fcf66a560fb8a20e9425f4fe02747 Fixed for 1.3.0 with 0ea996a64ca0ff9589ffcd5b89967f51aee1ffe6 > Make table unit tests pass under cluster mode > - > > Key: FLINK-5452 > URL: https://issues.apache.org/jira/browse/FLINK-5452 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Kurt Young >Assignee: Kurt Young > Fix For: 1.2.0, 1.3.0 > > > Currently if we change the test execution mode to > {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will > fail. Need to figure out whether it's the case design problem or there are > some bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (FLINK-5452) Make table unit tests pass under cluster mode
[ https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske reopened FLINK-5452: -- > Make table unit tests pass under cluster mode > - > > Key: FLINK-5452 > URL: https://issues.apache.org/jira/browse/FLINK-5452 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Kurt Young >Assignee: Kurt Young > > Currently if we change the test execution mode to > {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will > fail. Need to figure out whether it's the case design problem or there are > some bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5386) Refactoring Window Clause
[ https://issues.apache.org/jira/browse/FLINK-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831354#comment-15831354 ] ASF GitHub Bot commented on FLINK-5386: --- Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3046 Hi, @fhueske thanks a lot for the review. I have updated the PR according to your comments. Let me know if I miss something,thanks again!! > Refactoring Window Clause > - > > Key: FLINK-5386 > URL: https://issues.apache.org/jira/browse/FLINK-5386 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: sunjincheng >Assignee: sunjincheng > > Similar to the SQL, window clause is defined "as" a symbol which is > explicitly used in groupby/over. We are proposing to refactor the way to > write groupby+window tableAPI as follows: > {code} > tab //Table('a,'b,'c) > .window( Slide over 10.milli every 5.milli as 'w1as 'w1) // WindowedTable > .groupBy('w1,'a,'b) // WindowGroupedTable > .select('a,'b,c.count as 'mycount) // Table > .window(Tumble over 5.milli on 'b as 'w2) > .groupBy('w2) > .select('a.count, 'w2.start, 'w2.end) > {code} > In this way, for row-window, we anyway need to define window clause as a > symbol. This change will make the API of window and row-window consistent, > example for row-window: > {code} > .window(RowXXXWindow as ‘x, RowYYYWindow as ‘y) // WindowedTable > .select(‘a, ‘b.count over ‘x as ‘xcnt, ‘c.count over ‘y as ‘ycnt, ‘x.start, > ‘x.end) > {code} > What do you think? [~fhueske] [~twalthr] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5520) Disable outer joins with non-equality predicates
[ https://issues.apache.org/jira/browse/FLINK-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Hueske closed FLINK-5520. Resolution: Done Fix Version/s: 1.3.0 Fixed for 1.2.0 with 9073c53f902df7b3dbbdec2a30d86030e49fe27e Fixed for 1.3.0 with d1301c82b85c00284d90e8f5bdac4fd86dc5b173 > Disable outer joins with non-equality predicates > > > Key: FLINK-5520 > URL: https://issues.apache.org/jira/browse/FLINK-5520 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: lincoln.lee >Priority: Blocker > Fix For: 1.2.0, 1.3.0 > > > Outer joins with non-equality predicates (and at least one equality > predicate) compute incorrect results. > Since this is not a very common requirement, I propose to disable this > feature for the 1.2.0 release and correctly implement it for a later version. > The fix should add checks in the Table API validation phase (to get a good > error message) and in the DataSetJoinRule to prevent translation of SQL > queries with non-equality predicates on outer joins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #3046: [FLINK-5386][Table API & SQL] refactoring Window Clause
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3046 Hi, @fhueske thanks a lot for the review. I have updated the PR according to your comments. Let me know if I miss somethingï¼thanks again!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5580) Kerberos keytabs not working for YARN deployment mode
[ https://issues.apache.org/jira/browse/FLINK-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831322#comment-15831322 ] ASF GitHub Bot commented on FLINK-5580: --- GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3177 [FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in YARN mode Previously, the local path of the shipped keytab was set _after_ `SecurityConfiguration` was created, causing the picked up keytab path to be invalid and validation of the security configuration not passing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tzulitai/flink FLINK-5580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3177.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3177 commit 871c6a6601af33f57c3794d6f627a874ac4c8731 Author: Tzu-Li (Gordon) TaiDate: 2017-01-20T00:41:05Z [FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in YARN mode > Kerberos keytabs not working for YARN deployment mode > - > > Key: FLINK-5580 > URL: https://issues.apache.org/jira/browse/FLINK-5580 > Project: Flink > Issue Type: Bug > Components: Security, YARN >Reporter: Tzu-Li (Gordon) Tai >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > Fix For: 1.2.0 > > > Setup: Kerberos security using keytabs, Flink session on YARN deployment (in > standalone, it works fine without problems). > I’m getting these error messages in the YARN node managers, causing the > TaskManager containers to fail to start properly: > {{org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:tzulitai (auth:SIMPLE) > cause:org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN, KERBEROS]}} > The security configuration for Hadoop has been set to "kerberos", to the > "auto: SIMPLE" seems very strange. It also seems as if credential tokens has > not been properly set for the {{ContainerLaunchContext}} s, which may be an > issue causing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3177: [FLINK-5580] [security] Fix path setting of shippe...
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/3177 [FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in YARN mode Previously, the local path of the shipped keytab was set _after_ `SecurityConfiguration` was created, causing the picked up keytab path to be invalid and validation of the security configuration not passing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tzulitai/flink FLINK-5580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3177.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3177 commit 871c6a6601af33f57c3794d6f627a874ac4c8731 Author: Tzu-Li (Gordon) TaiDate: 2017-01-20T00:41:05Z [FLINK-5580] [security] Fix path setting of shipped Kerberos keytabs in YARN mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5571) add open and close methods for UserDefinedFunction in TableAPI & SQL
[ https://issues.apache.org/jira/browse/FLINK-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831296#comment-15831296 ] ASF GitHub Bot commented on FLINK-5571: --- GitHub user godfreyhe opened a pull request: https://github.com/apache/flink/pull/3176 [FLINK-5571] [table] add open and close methods for UserDefinedFunction Currently, a User Defined Function (UDF) in table API & SQL works on zero, one, or multiple values in custom evaluation method. Many UDFs need more complex features, e.g. report metrics, get parameters from job configuration, or get extra data from distribute cache file, etc. Adding open and close methods in UserDefinedFunction class can solve this problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/godfreyhe/flink udf-open-close Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3176.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3176 commit 4ac2be7b3ab4dc0e8a18e43c79a031d7a16ee1ea Author: godfreyheDate: 2017-01-20T06:42:12Z add open and close methods for UserDefinedFunction > add open and close methods for UserDefinedFunction in TableAPI & SQL > > > Key: FLINK-5571 > URL: https://issues.apache.org/jira/browse/FLINK-5571 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: godfrey he >Assignee: godfrey he > > Currently, a User Defined Function (UDF) in table API & SQL works on zero, > one, or multiple values in custom evaluation method. Many UDFs need more > complex features, e.g. report metrics, get parameters from job configuration, > or get extra data from distribute cache file, etc. Adding open and close > methods in UserDefinedFunction class can solve this problem. The code cloud > look like: > {code} > trait UserDefinedFunction { > def open(context: UDFContext): Unit = {} > def close(): Unit = {} > } > {code} > UDFContext contains the information about metric reporters, job parameters, > distribute cache, etc. The code cloud look like: > {code} > class UDFContext(context: RuntimeContext) { > def getMetricGroup: MetricGroup = ??? > def getDistributedCacheFile(name: String): File = ??? > def getJobParameter(key: String, default: String): String = ??? > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3176: [FLINK-5571] [table] add open and close methods fo...
GitHub user godfreyhe opened a pull request: https://github.com/apache/flink/pull/3176 [FLINK-5571] [table] add open and close methods for UserDefinedFunction Currently, a User Defined Function (UDF) in table API & SQL works on zero, one, or multiple values in custom evaluation method. Many UDFs need more complex features, e.g. report metrics, get parameters from job configuration, or get extra data from distribute cache file, etc. Adding open and close methods in UserDefinedFunction class can solve this problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/godfreyhe/flink udf-open-close Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3176.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3176 commit 4ac2be7b3ab4dc0e8a18e43c79a031d7a16ee1ea Author: godfreyheDate: 2017-01-20T06:42:12Z add open and close methods for UserDefinedFunction --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-5452) Make table unit tests pass under cluster mode
[ https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young closed FLINK-5452. - > Make table unit tests pass under cluster mode > - > > Key: FLINK-5452 > URL: https://issues.apache.org/jira/browse/FLINK-5452 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Kurt Young >Assignee: Kurt Young > > Currently if we change the test execution mode to > {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will > fail. Need to figure out whether it's the case design problem or there are > some bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-5452) Make table unit tests pass under cluster mode
[ https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young resolved FLINK-5452. --- Resolution: Fixed > Make table unit tests pass under cluster mode > - > > Key: FLINK-5452 > URL: https://issues.apache.org/jira/browse/FLINK-5452 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Kurt Young >Assignee: Kurt Young > > Currently if we change the test execution mode to > {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will > fail. Need to figure out whether it's the case design problem or there are > some bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5493) FlinkDistributionOverlay does not properly display missing environment variables
[ https://issues.apache.org/jira/browse/FLINK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5493. --- Resolution: Fixed Fix Version/s: 1.2.0 master: d830cac813254995e3cc04fbb7ab1b471bdd6c90 1.2: 82a6b83ec4c1660dfe97fd299ac3169e5c10088b > FlinkDistributionOverlay does not properly display missing environment > variables > > > Key: FLINK-5493 > URL: https://issues.apache.org/jira/browse/FLINK-5493 > Project: Flink > Issue Type: Bug > Components: Cluster Management >Affects Versions: 1.2.0, 1.3.0 >Reporter: Till Rohrmann >Assignee: Boris Osipov >Priority: Minor > Fix For: 1.2.0, 1.3.0 > > > The class {{FlinkDistributionOverlay}} does not properly log missing > environment variables in case of an error. This should be changed so that the > user knows which variables he has to set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5493) FlinkDistributionOverlay does not properly display missing environment variables
[ https://issues.apache.org/jira/browse/FLINK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5493: Assignee: Boris Osipov > FlinkDistributionOverlay does not properly display missing environment > variables > > > Key: FLINK-5493 > URL: https://issues.apache.org/jira/browse/FLINK-5493 > Project: Flink > Issue Type: Bug > Components: Cluster Management >Affects Versions: 1.2.0, 1.3.0 >Reporter: Till Rohrmann >Assignee: Boris Osipov >Priority: Minor > Fix For: 1.3.0 > > > The class {{FlinkDistributionOverlay}} does not properly log missing > environment variables in case of an error. This should be changed so that the > user knows which variables he has to set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5113) Make all Testing Functions implement CheckpointedFunction Interface.
[ https://issues.apache.org/jira/browse/FLINK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5113. --- Resolution: Fixed Fix Version/s: 1.3.0 master: 525edf1e6925b55302d991ddf537a2f16caba21d 1.2: 8b069fde3adccdcd5143de90d3d4834f33b5acff > Make all Testing Functions implement CheckpointedFunction Interface. > > > Key: FLINK-5113 > URL: https://issues.apache.org/jira/browse/FLINK-5113 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas > Fix For: 1.2.0, 1.3.0 > > > Currently stateful functions implement the (old) Checkpointed interface. > This is issue aims at porting all these function to the new > CheckpointedFunction interface, so that they can leverage the new > capabilities by it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4920) Add a Scala Function Gauge
[ https://issues.apache.org/jira/browse/FLINK-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-4920. --- Resolution: Fixed Fix Version/s: 1.3.0 master: 570dbc8d2597ee9688579f399b8743636e70f891 > Add a Scala Function Gauge > -- > > Key: FLINK-4920 > URL: https://issues.apache.org/jira/browse/FLINK-4920 > Project: Flink > Issue Type: Improvement > Components: Metrics, Scala API >Reporter: Stephan Ewen >Assignee: Pattarawat Chormai > Labels: easyfix, starter > Fix For: 1.3.0 > > > A useful metrics utility for the Scala API would be to add a Gauge that > obtains its value by calling a Scala Function0. > That way, one can add Gauges in Scala programs using Scala lambda notation or > function references. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5113) Make all Testing Functions implement CheckpointedFunction Interface.
[ https://issues.apache.org/jira/browse/FLINK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5113: Affects Version/s: 1.3.0 > Make all Testing Functions implement CheckpointedFunction Interface. > > > Key: FLINK-5113 > URL: https://issues.apache.org/jira/browse/FLINK-5113 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0, 1.3.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas > Fix For: 1.2.0 > > > Currently stateful functions implement the (old) Checkpointed interface. > This is issue aims at porting all these function to the new > CheckpointedFunction interface, so that they can leverage the new > capabilities by it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService
[ https://issues.apache.org/jira/browse/FLINK-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5321. --- Resolution: Fixed Fix Version/s: 1.3.0 master: b36b43b9ad3f4b04a2a701b9139bb3eaf17b9ff8 1.2: 2bbc92cf637b2caf612c43b0f35ba2282b5bc981 > FlinkMiniCluster does not start Jobmanager MetricQueryService > - > > Key: FLINK-5321 > URL: https://issues.apache.org/jira/browse/FLINK-5321 > Project: Flink > Issue Type: Bug > Components: JobManager, Metrics >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.2.0, 1.3.0 > > > The JobManager MetricQueryService is never started when using the > LocalFlinkMiniCluster. It lacks the call to > MetricRegistry#startQueryService(). > As a result jobmanager metrics aren't reporter to the web frontend, and it > causes repeated logging of exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5417) Fix the wrong config file name
[ https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5417. --- Resolution: Fixed Fix Version/s: 1.3.0 1.2.0 master: 6c03491ce30ba3f31fb85ffb864731aa3f5aa482 1.2: 7f20bd0d65cdce9e6180c20777959e4c522c55ce > Fix the wrong config file name > --- > > Key: FLINK-5417 > URL: https://issues.apache.org/jira/browse/FLINK-5417 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.2.0, 1.3.0 >Reporter: Tao Wang >Priority: Trivial > Fix For: 1.2.0, 1.3.0 > > > As the config file name is conf/flink-conf.yaml, the usage > "conf/flink-config.yaml" in document is wrong and easy to confuse user. We > should correct them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService
[ https://issues.apache.org/jira/browse/FLINK-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5321: Affects Version/s: 1.3.0 > FlinkMiniCluster does not start Jobmanager MetricQueryService > - > > Key: FLINK-5321 > URL: https://issues.apache.org/jira/browse/FLINK-5321 > Project: Flink > Issue Type: Bug > Components: JobManager, Metrics >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.2.0, 1.3.0 > > > The JobManager MetricQueryService is never started when using the > LocalFlinkMiniCluster. It lacks the call to > MetricRegistry#startQueryService(). > As a result jobmanager metrics aren't reporter to the web frontend, and it > causes repeated logging of exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5417) Fix the wrong config file name
[ https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5417: Affects Version/s: 1.3.0 1.2.0 > Fix the wrong config file name > --- > > Key: FLINK-5417 > URL: https://issues.apache.org/jira/browse/FLINK-5417 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.2.0, 1.3.0 >Reporter: Tao Wang >Priority: Trivial > > As the config file name is conf/flink-conf.yaml, the usage > "conf/flink-config.yaml" in document is wrong and easy to confuse user. We > should correct them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5424) Improve Restart Strategy Logging
[ https://issues.apache.org/jira/browse/FLINK-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5424. --- Resolution: Fixed Fix Version/s: 1.3.0 1.2.0 master: c210ff37a4c55c835f76c031f2d9cf18165812aa 1.2: fc4b45e4781a4ccd595ad31aefeba448fdea9700 > Improve Restart Strategy Logging > > > Key: FLINK-5424 > URL: https://issues.apache.org/jira/browse/FLINK-5424 > Project: Flink > Issue Type: Improvement > Components: Core >Reporter: Shannon Carey >Assignee: Shannon Carey >Priority: Minor > Fix For: 1.2.0, 1.3.0 > > > I'll be submitting a PR which includes some minor improvements to logging > related to restart strategies. > Specifically, I added a toString so that the log contains better info about > failure-rate restart strategy, and I added an explanation in the log when the > restart strategy is responsible for preventing job restart (currently, > there's no indication that the restart strategy had anything to do with it). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5419) Taskmanager metrics not accessible via REST
[ https://issues.apache.org/jira/browse/FLINK-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5419. --- Resolution: Fixed master: c93e04c16a7865dbbc90d89799b46f91831e0a55 1.2: f90edc483000db42caaff0de650c1f5bca11bf47 > Taskmanager metrics not accessible via REST > --- > > Key: FLINK-5419 > URL: https://issues.apache.org/jira/browse/FLINK-5419 > Project: Flink > Issue Type: Bug > Components: Metrics, Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.2.0, 1.3.0 > > > There is currently a URL clash between the TaskManagersHandler and > TaskManagerMetricsHandler, with both being routed to > {code} > /taskmanagers/:taskmanagerid/metrics > {code} > As a result it is not possible to query the full set of metrics for a > taskmanager, but only the hard-coded subset that is displayed on the metrics > tab on the taskmanager page. > This is a side-effect of 6d53bbc4b92e651786ecc8c2c6dfeb8e450a16a3 making the > URL's more consistent. The TaskManager page in the web-interface has 3 tabs: > Metrics, Log and Stdout. > The URLs for these tabs are > {code} > /taskmanager//metrics > /taskmanager//log > /taskmanager//stdout > {code} > which correspond to the REST URL's used. Previously, the metrics tab used > {code}/taskmanager/{code} > However, 70704de0c82cbb7b143dd696221e11999feb3600 then exposed the metrics > gathered by the metrics system through the REST API. The assumption was that > general information for the taskmanagers are retrieved via > /taskmanager/, similar to how the job-related URL's are structured, > which sadly isn't the case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs
[ https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5434. --- Resolution: Fixed Fix Version/s: 1.3.0 1.2.0 master: a7437cca0eeeb636f839ecbc8e96ede64405ab1b 1.2: ebf4443a87ec0a46cd821adc85187d9cb58ae992 > Remove unsupported project() transformation from Scala DataStream docs > -- > > Key: FLINK-5434 > URL: https://issues.apache.org/jira/browse/FLINK-5434 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.2.0, 1.3.0 >Reporter: Vasia Kalavri >Assignee: GaoLun >Priority: Minor > Fix For: 1.2.0, 1.3.0 > > > The Scala DataStream does not have a project() transformation, yet the docs > include it as a supported operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs
[ https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5434: Affects Version/s: 1.3.0 1.2.0 > Remove unsupported project() transformation from Scala DataStream docs > -- > > Key: FLINK-5434 > URL: https://issues.apache.org/jira/browse/FLINK-5434 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.2.0, 1.3.0 >Reporter: Vasia Kalavri >Assignee: GaoLun > Fix For: 1.2.0, 1.3.0 > > > The Scala DataStream does not have a project() transformation, yet the docs > include it as a supported operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs
[ https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5434: Priority: Minor (was: Major) > Remove unsupported project() transformation from Scala DataStream docs > -- > > Key: FLINK-5434 > URL: https://issues.apache.org/jira/browse/FLINK-5434 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.2.0, 1.3.0 >Reporter: Vasia Kalavri >Assignee: GaoLun >Priority: Minor > Fix For: 1.2.0, 1.3.0 > > > The Scala DataStream does not have a project() transformation, yet the docs > include it as a supported operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5329) Metric list is being cut off in the WebFrontend
[ https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5329: Fix Version/s: 1.3.0 > Metric list is being cut off in the WebFrontend > --- > > Key: FLINK-5329 > URL: https://issues.apache.org/jira/browse/FLINK-5329 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 > Environment: FireFox 50.0.2 >Reporter: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0, 1.3.0 > > Attachments: ss_metrics.png > > > The list for selecting metrics is being cut off, see the attached screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-5329) Metric list is being cut off in the WebFrontend
[ https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-5329: --- Assignee: Chesnay Schepler > Metric list is being cut off in the WebFrontend > --- > > Key: FLINK-5329 > URL: https://issues.apache.org/jira/browse/FLINK-5329 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 > Environment: FireFox 50.0.2 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0, 1.3.0 > > Attachments: ss_metrics.png > > > The list for selecting metrics is being cut off, see the attached screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5434) Remove unsupported project() transformation from Scala DataStream docs
[ https://issues.apache.org/jira/browse/FLINK-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5434: Assignee: GaoLun > Remove unsupported project() transformation from Scala DataStream docs > -- > > Key: FLINK-5434 > URL: https://issues.apache.org/jira/browse/FLINK-5434 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: Vasia Kalavri >Assignee: GaoLun > > The Scala DataStream does not have a project() transformation, yet the docs > include it as a supported operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5329) Metric list is being cut off in the WebFrontend
[ https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5329. --- Resolution: Fixed master: b8c34d455029907a237230e2f8c49a8b6b651970 1.2: 8cdbe44aea1f7e544e4a2907e5274012b1481c57 > Metric list is being cut off in the WebFrontend > --- > > Key: FLINK-5329 > URL: https://issues.apache.org/jira/browse/FLINK-5329 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 > Environment: FireFox 50.0.2 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0, 1.3.0 > > Attachments: ss_metrics.png > > > The list for selecting metrics is being cut off, see the attached screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5380) Number of outgoing records not reported in web interface
[ https://issues.apache.org/jira/browse/FLINK-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5380. --- Resolution: Fixed master: cb05915759b1d5ea4dbfcdd3ff76dcfd9cebe601 1.2: 792f7e45216377fa1d6f29dfc767d83cf1a84f37 > Number of outgoing records not reported in web interface > > > Key: FLINK-5380 > URL: https://issues.apache.org/jira/browse/FLINK-5380 > Project: Flink > Issue Type: Bug > Components: Metrics, Streaming, Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0, 1.3.0 > > Attachments: outRecordsNotreported.png > > > The web frontend does not report any outgoing records in the web frontend. > The amount of data in MB is reported correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5329) Metric list is being cut off in the WebFrontend
[ https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5329: Affects Version/s: 1.3.0 > Metric list is being cut off in the WebFrontend > --- > > Key: FLINK-5329 > URL: https://issues.apache.org/jira/browse/FLINK-5329 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 > Environment: FireFox 50.0.2 >Reporter: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0 > > Attachments: ss_metrics.png > > > The list for selecting metrics is being cut off, see the attached screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files
[ https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5432: Affects Version/s: 1.3.0 > ContinuousFileMonitoringFunction is not monitoring nested files > --- > > Key: FLINK-5432 > URL: https://issues.apache.org/jira/browse/FLINK-5432 > Project: Flink > Issue Type: Bug > Components: filesystem-connector >Affects Versions: 1.2.0, 1.3.0 >Reporter: Yassine Marzougui >Assignee: Yassine Marzougui > Fix For: 1.2.0, 1.3.0 > > > The {{ContinuousFileMonitoringFunction}} does not monitor nested files even > if the inputformat has NestedFileEnumeration set to true. This can be fixed > by enabling a recursive scan of the directories in the {{listEligibleFiles}} > method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files
[ https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5432. --- Resolution: Fixed master: 9945904e2251e7c0e218e2766bf07778d1307277 1.2: 28c18e22127a85f773e7504a0e9d188bad9334e2 > ContinuousFileMonitoringFunction is not monitoring nested files > --- > > Key: FLINK-5432 > URL: https://issues.apache.org/jira/browse/FLINK-5432 > Project: Flink > Issue Type: Bug > Components: filesystem-connector >Affects Versions: 1.2.0, 1.3.0 >Reporter: Yassine Marzougui >Assignee: Yassine Marzougui > Fix For: 1.2.0, 1.3.0 > > > The {{ContinuousFileMonitoringFunction}} does not monitor nested files even > if the inputformat has NestedFileEnumeration set to true. This can be fixed > by enabling a recursive scan of the directories in the {{listEligibleFiles}} > method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5150) WebUI metric-related resource leak
[ https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-5150. --- Resolution: Fixed master: cbd933bd1def789d1978be5349005dd7d9de3082 1.2: 79b68263f52690697ceec0528023d35feed0bd89 > WebUI metric-related resource leak > -- > > Key: FLINK-5150 > URL: https://issues.apache.org/jira/browse/FLINK-5150 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.0, 1.3.0 > > > The WebUI maintains a list of selected metrics for all jobs and vertices. > When a metric is selected in the metric screen it is added to this list, and > removed if it is unselected. > The contents of this list are stored in the browser's localStorage. This > allows a user to setup a metric screen, move to another page, and return to > the original screen completely intact. > However, if the metrics are never *unselected* by the user they will remain > in this list. They will also still be in this list if the WebUI can't even > display the corresponding job page anymore, if for example the history size > limit was exceeded. They will even survive a browser restart, since they are > not stored in a session-based storage. > Furthermore, the WebUI still tries to update these metricsd, adding > additional overhead to the WebBackend and potentially network. > In other words, if you _ever_ checked out metrics tab for some job, chances > are that the next time you start the WebInterface it will still try to update > the metrics for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5150) WebUI metric-related resource leak
[ https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5150: Fix Version/s: 1.3.0 > WebUI metric-related resource leak > -- > > Key: FLINK-5150 > URL: https://issues.apache.org/jira/browse/FLINK-5150 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.0, 1.3.0 > > > The WebUI maintains a list of selected metrics for all jobs and vertices. > When a metric is selected in the metric screen it is added to this list, and > removed if it is unselected. > The contents of this list are stored in the browser's localStorage. This > allows a user to setup a metric screen, move to another page, and return to > the original screen completely intact. > However, if the metrics are never *unselected* by the user they will remain > in this list. They will also still be in this list if the WebUI can't even > display the corresponding job page anymore, if for example the history size > limit was exceeded. They will even survive a browser restart, since they are > not stored in a session-based storage. > Furthermore, the WebUI still tries to update these metricsd, adding > additional overhead to the WebBackend and potentially network. > In other words, if you _ever_ checked out metrics tab for some job, chances > are that the next time you start the WebInterface it will still try to update > the metrics for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5321) FlinkMiniCluster does not start Jobmanager MetricQueryService
[ https://issues.apache.org/jira/browse/FLINK-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831232#comment-15831232 ] ASF GitHub Bot commented on FLINK-5321: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2991 > FlinkMiniCluster does not start Jobmanager MetricQueryService > - > > Key: FLINK-5321 > URL: https://issues.apache.org/jira/browse/FLINK-5321 > Project: Flink > Issue Type: Bug > Components: JobManager, Metrics >Affects Versions: 1.2.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.2.0 > > > The JobManager MetricQueryService is never started when using the > LocalFlinkMiniCluster. It lacks the call to > MetricRegistry#startQueryService(). > As a result jobmanager metrics aren't reporter to the web frontend, and it > causes repeated logging of exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4920) Add a Scala Function Gauge
[ https://issues.apache.org/jira/browse/FLINK-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831234#comment-15831234 ] ASF GitHub Bot commented on FLINK-4920: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3080 > Add a Scala Function Gauge > -- > > Key: FLINK-4920 > URL: https://issues.apache.org/jira/browse/FLINK-4920 > Project: Flink > Issue Type: Improvement > Components: Metrics, Scala API >Reporter: Stephan Ewen >Assignee: Pattarawat Chormai > Labels: easyfix, starter > > A useful metrics utility for the Scala API would be to add a Gauge that > obtains its value by calling a Scala Function0. > That way, one can add Gauges in Scala programs using Scala lambda notation or > function references. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5329) Metric list is being cut off in the WebFrontend
[ https://issues.apache.org/jira/browse/FLINK-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831227#comment-15831227 ] ASF GitHub Bot commented on FLINK-5329: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3109 > Metric list is being cut off in the WebFrontend > --- > > Key: FLINK-5329 > URL: https://issues.apache.org/jira/browse/FLINK-5329 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0 > Environment: FireFox 50.0.2 >Reporter: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0 > > Attachments: ss_metrics.png > > > The list for selecting metrics is being cut off, see the attached screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5424) Improve Restart Strategy Logging
[ https://issues.apache.org/jira/browse/FLINK-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831230#comment-15831230 ] ASF GitHub Bot commented on FLINK-5424: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3079 > Improve Restart Strategy Logging > > > Key: FLINK-5424 > URL: https://issues.apache.org/jira/browse/FLINK-5424 > Project: Flink > Issue Type: Improvement > Components: Core >Reporter: Shannon Carey >Assignee: Shannon Carey >Priority: Minor > > I'll be submitting a PR which includes some minor improvements to logging > related to restart strategies. > Specifically, I added a toString so that the log contains better info about > failure-rate restart strategy, and I added an explanation in the log when the > restart strategy is responsible for preventing job restart (currently, > there's no indication that the restart strategy had anything to do with it). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5150) WebUI metric-related resource leak
[ https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5150: Affects Version/s: (was: 1.1.3) > WebUI metric-related resource leak > -- > > Key: FLINK-5150 > URL: https://issues.apache.org/jira/browse/FLINK-5150 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.0 > > > The WebUI maintains a list of selected metrics for all jobs and vertices. > When a metric is selected in the metric screen it is added to this list, and > removed if it is unselected. > The contents of this list are stored in the browser's localStorage. This > allows a user to setup a metric screen, move to another page, and return to > the original screen completely intact. > However, if the metrics are never *unselected* by the user they will remain > in this list. They will also still be in this list if the WebUI can't even > display the corresponding job page anymore, if for example the history size > limit was exceeded. They will even survive a browser restart, since they are > not stored in a session-based storage. > Furthermore, the WebUI still tries to update these metricsd, adding > additional overhead to the WebBackend and potentially network. > In other words, if you _ever_ checked out metrics tab for some job, chances > are that the next time you start the WebInterface it will still try to update > the metrics for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5432) ContinuousFileMonitoringFunction is not monitoring nested files
[ https://issues.apache.org/jira/browse/FLINK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831226#comment-15831226 ] ASF GitHub Bot commented on FLINK-5432: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3090 > ContinuousFileMonitoringFunction is not monitoring nested files > --- > > Key: FLINK-5432 > URL: https://issues.apache.org/jira/browse/FLINK-5432 > Project: Flink > Issue Type: Bug > Components: filesystem-connector >Affects Versions: 1.2.0 >Reporter: Yassine Marzougui >Assignee: Yassine Marzougui > Fix For: 1.2.0, 1.3.0 > > > The {{ContinuousFileMonitoringFunction}} does not monitor nested files even > if the inputformat has NestedFileEnumeration set to true. This can be fixed > by enabling a recursive scan of the directories in the {{listEligibleFiles}} > method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5150) WebUI metric-related resource leak
[ https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831225#comment-15831225 ] ASF GitHub Bot commented on FLINK-5150: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3104 > WebUI metric-related resource leak > -- > > Key: FLINK-5150 > URL: https://issues.apache.org/jira/browse/FLINK-5150 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.0 > > > The WebUI maintains a list of selected metrics for all jobs and vertices. > When a metric is selected in the metric screen it is added to this list, and > removed if it is unselected. > The contents of this list are stored in the browser's localStorage. This > allows a user to setup a metric screen, move to another page, and return to > the original screen completely intact. > However, if the metrics are never *unselected* by the user they will remain > in this list. They will also still be in this list if the WebUI can't even > display the corresponding job page anymore, if for example the history size > limit was exceeded. They will even survive a browser restart, since they are > not stored in a session-based storage. > Furthermore, the WebUI still tries to update these metricsd, adding > additional overhead to the WebBackend and potentially network. > In other words, if you _ever_ checked out metrics tab for some job, chances > are that the next time you start the WebInterface it will still try to update > the metrics for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3068: [FLINK-5380] Fix task metrics reuse for single-ope...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3068 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3092: [FLINK-5419] Make full TM metrics available throug...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3092 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3071: [FLINK-5417][DOCUMENTATION]correct the wrong confi...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3071 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2939: [FLINK-5113] Ports all functions in the tests to t...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2939 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5417) Fix the wrong config file name
[ https://issues.apache.org/jira/browse/FLINK-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831231#comment-15831231 ] ASF GitHub Bot commented on FLINK-5417: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3071 > Fix the wrong config file name > --- > > Key: FLINK-5417 > URL: https://issues.apache.org/jira/browse/FLINK-5417 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: Tao Wang >Priority: Trivial > > As the config file name is conf/flink-conf.yaml, the usage > "conf/flink-config.yaml" in document is wrong and easy to confuse user. We > should correct them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5150) WebUI metric-related resource leak
[ https://issues.apache.org/jira/browse/FLINK-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-5150: Affects Version/s: 1.3.0 1.2.0 > WebUI metric-related resource leak > -- > > Key: FLINK-5150 > URL: https://issues.apache.org/jira/browse/FLINK-5150 > Project: Flink > Issue Type: Bug > Components: Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.2.0 > > > The WebUI maintains a list of selected metrics for all jobs and vertices. > When a metric is selected in the metric screen it is added to this list, and > removed if it is unselected. > The contents of this list are stored in the browser's localStorage. This > allows a user to setup a metric screen, move to another page, and return to > the original screen completely intact. > However, if the metrics are never *unselected* by the user they will remain > in this list. They will also still be in this list if the WebUI can't even > display the corresponding job page anymore, if for example the history size > limit was exceeded. They will even survive a browser restart, since they are > not stored in a session-based storage. > Furthermore, the WebUI still tries to update these metricsd, adding > additional overhead to the WebBackend and potentially network. > In other words, if you _ever_ checked out metrics tab for some job, chances > are that the next time you start the WebInterface it will still try to update > the metrics for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3080: [FLINK-4920] Add a Scala Function Gauge
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3080 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5113) Make all Testing Functions implement CheckpointedFunction Interface.
[ https://issues.apache.org/jira/browse/FLINK-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831233#comment-15831233 ] ASF GitHub Bot commented on FLINK-5113: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2939 > Make all Testing Functions implement CheckpointedFunction Interface. > > > Key: FLINK-5113 > URL: https://issues.apache.org/jira/browse/FLINK-5113 > Project: Flink > Issue Type: Improvement > Components: DataStream API >Affects Versions: 1.2.0 >Reporter: Kostas Kloudas >Assignee: Kostas Kloudas > Fix For: 1.2.0 > > > Currently stateful functions implement the (old) Checkpointed interface. > This is issue aims at porting all these function to the new > CheckpointedFunction interface, so that they can leverage the new > capabilities by it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5380) Number of outgoing records not reported in web interface
[ https://issues.apache.org/jira/browse/FLINK-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831228#comment-15831228 ] ASF GitHub Bot commented on FLINK-5380: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3068 > Number of outgoing records not reported in web interface > > > Key: FLINK-5380 > URL: https://issues.apache.org/jira/browse/FLINK-5380 > Project: Flink > Issue Type: Bug > Components: Metrics, Streaming, Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.2.0, 1.3.0 > > Attachments: outRecordsNotreported.png > > > The web frontend does not report any outgoing records in the web frontend. > The amount of data in MB is reported correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2991: [FLINK-5321] [metrics] LocalFlinkMiniCluster start...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2991 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5419) Taskmanager metrics not accessible via REST
[ https://issues.apache.org/jira/browse/FLINK-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831229#comment-15831229 ] ASF GitHub Bot commented on FLINK-5419: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3092 > Taskmanager metrics not accessible via REST > --- > > Key: FLINK-5419 > URL: https://issues.apache.org/jira/browse/FLINK-5419 > Project: Flink > Issue Type: Bug > Components: Metrics, Webfrontend >Affects Versions: 1.2.0, 1.3.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.2.0, 1.3.0 > > > There is currently a URL clash between the TaskManagersHandler and > TaskManagerMetricsHandler, with both being routed to > {code} > /taskmanagers/:taskmanagerid/metrics > {code} > As a result it is not possible to query the full set of metrics for a > taskmanager, but only the hard-coded subset that is displayed on the metrics > tab on the taskmanager page. > This is a side-effect of 6d53bbc4b92e651786ecc8c2c6dfeb8e450a16a3 making the > URL's more consistent. The TaskManager page in the web-interface has 3 tabs: > Metrics, Log and Stdout. > The URLs for these tabs are > {code} > /taskmanager//metrics > /taskmanager//log > /taskmanager//stdout > {code} > which correspond to the REST URL's used. Previously, the metrics tab used > {code}/taskmanager/{code} > However, 70704de0c82cbb7b143dd696221e11999feb3600 then exposed the metrics > gathered by the metrics system through the REST API. The assumption was that > general information for the taskmanagers are retrieved via > /taskmanager/, similar to how the job-related URL's are structured, > which sadly isn't the case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3137: [hotfix] [doc] Fix several broken "Linking with Fl...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3137 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3109: [FLINK-5329] Fix metric list being cut off
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3109 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3090: [FLINK-5432] Fix nested files enumeration in Conti...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3090 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3079: [FLINK-5424] Improve Restart Strategy Logging
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3079 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3010: [minor] Fix String formats in FlinkDistributionOve...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3010 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3104: [FLINK-5150] [webui] Store metrics in sessionStora...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3104 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/3149#discussion_r97015379 --- Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java --- @@ -0,0 +1,322 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.io.InputFormat; +import org.apache.flink.api.common.io.LocatableInputSplitAssigner; +import org.apache.flink.api.common.io.RichInputFormat; +import org.apache.flink.api.common.io.statistics.BaseStatistics; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.ResultTypeQueryable; +import org.apache.flink.api.java.typeutils.RowTypeInfo; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.core.io.InputSplitAssigner; +import org.apache.flink.types.Row; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.ClusterConnection; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.HRegionLocator; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +/** + * {@link InputFormat} subclass that wraps the access for HTables. Returns the result as {@link Row} + */ +public class HBaseTableSourceInputFormat extends RichInputFormatimplements ResultTypeQueryable { + + private static final long serialVersionUID = 1L; + + private static final Logger LOG = LoggerFactory.getLogger(HBaseTableSourceInputFormat.class); + private String tableName; + private TypeInformation[] fieldTypeInfos; + private String[] fieldNames; + private transient Table table; + private transient Scan scan; + private transient Connection conn; + private ResultScanner resultScanner = null; + + private byte[] lastRow; + private int scannedRows; + private boolean endReached = false; + private org.apache.hadoop.conf.Configuration conf; + private static final String COLON = ":"; + + public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) { + this.conf = conf; + this.tableName = tableName; + this.fieldNames = fieldNames; + this.fieldTypeInfos = fieldTypeInfos; + } + + @Override + public void configure(Configuration parameters) { + LOG.info("Initializing HBaseConfiguration"); + connectToTable(); + if(table != null) { + scan = createScanner(); + } + } + + private Scan createScanner() { + Scan scan = new Scan(); + for(String field : fieldNames) { + // select only the fields in the 'selectedFields' + String[] famCol = field.split(COLON); + scan.addColumn(Bytes.toBytes(famCol[0]), Bytes.toBytes(famCol[1])); + } + return scan; + } + + private void connectToTable() { + //use files found in the classpath + if(this.conf == null) { + this.conf =
[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/3149#discussion_r97015321 --- Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSource.java --- @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.DataSet; +import org.apache.flink.api.java.ExecutionEnvironment; +import org.apache.flink.api.java.typeutils.RowTypeInfo; +import org.apache.flink.table.sources.BatchTableSource; +import org.apache.flink.table.sources.ProjectableTableSource; +import org.apache.flink.types.Row; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.conf.Configuration; + +/** + * Creates a table source that helps to scan data from an hbase table + * + * Note : the colNames are specified along with a familyName and they are seperated by a ':' + * For eg, cf1:q1 - where cf1 is the familyName and q1 is the qualifier name + */ +public class HBaseTableSource implements BatchTableSource, ProjectableTableSource { + + private Configuration conf; + private String tableName; + private byte[] rowKey; + private String[] colNames; + private TypeInformation[] colTypes; + + public HBaseTableSource(Configuration conf, String tableName, byte[] rowKey, String[] colNames, + TypeInformation[] colTypes) { + this.conf = conf; + this.tableName = Preconditions.checkNotNull(tableName, "Table name"); + this.rowKey = Preconditions.checkNotNull(rowKey, "Rowkey"); --- End diff -- Yes. That is true but do we always want full table scan? Actually in HBase it is better we specify start and end key. So how do we specify that? I have not used this rowKey now but I thought it is better to be used? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-2168) Add HBaseTableSource
[ https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831223#comment-15831223 ] ASF GitHub Bot commented on FLINK-2168: --- Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/3149#discussion_r97015379 --- Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java --- @@ -0,0 +1,322 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.io.InputFormat; +import org.apache.flink.api.common.io.LocatableInputSplitAssigner; +import org.apache.flink.api.common.io.RichInputFormat; +import org.apache.flink.api.common.io.statistics.BaseStatistics; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.ResultTypeQueryable; +import org.apache.flink.api.java.typeutils.RowTypeInfo; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.core.io.InputSplitAssigner; +import org.apache.flink.types.Row; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.ClusterConnection; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.HRegionLocator; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +/** + * {@link InputFormat} subclass that wraps the access for HTables. Returns the result as {@link Row} + */ +public class HBaseTableSourceInputFormat extends RichInputFormatimplements ResultTypeQueryable { + + private static final long serialVersionUID = 1L; + + private static final Logger LOG = LoggerFactory.getLogger(HBaseTableSourceInputFormat.class); + private String tableName; + private TypeInformation[] fieldTypeInfos; + private String[] fieldNames; + private transient Table table; + private transient Scan scan; + private transient Connection conn; + private ResultScanner resultScanner = null; + + private byte[] lastRow; + private int scannedRows; + private boolean endReached = false; + private org.apache.hadoop.conf.Configuration conf; + private static final String COLON = ":"; + + public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) { + this.conf = conf; + this.tableName = tableName; + this.fieldNames = fieldNames; + this.fieldTypeInfos = fieldTypeInfos; + } + + @Override + public void configure(Configuration parameters) { + LOG.info("Initializing HBaseConfiguration"); + connectToTable(); + if(table != null) { + scan = createScanner(); + } + } + + private Scan createScanner() { + Scan scan = new Scan(); + for(String field : fieldNames) { + // select only the fields in the 'selectedFields' + String[] famCol = field.split(COLON); + scan.addColumn(Bytes.toBytes(famCol[0]), Bytes.toBytes(famCol[1]));
[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/3149#discussion_r97015267 --- Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java --- @@ -0,0 +1,322 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.io.InputFormat; +import org.apache.flink.api.common.io.LocatableInputSplitAssigner; +import org.apache.flink.api.common.io.RichInputFormat; +import org.apache.flink.api.common.io.statistics.BaseStatistics; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.ResultTypeQueryable; +import org.apache.flink.api.java.typeutils.RowTypeInfo; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.core.io.InputSplitAssigner; +import org.apache.flink.types.Row; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.ClusterConnection; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.HRegionLocator; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +/** + * {@link InputFormat} subclass that wraps the access for HTables. Returns the result as {@link Row} + */ +public class HBaseTableSourceInputFormat extends RichInputFormatimplements ResultTypeQueryable { + + private static final long serialVersionUID = 1L; + + private static final Logger LOG = LoggerFactory.getLogger(HBaseTableSourceInputFormat.class); + private String tableName; + private TypeInformation[] fieldTypeInfos; + private String[] fieldNames; + private transient Table table; + private transient Scan scan; + private transient Connection conn; + private ResultScanner resultScanner = null; + + private byte[] lastRow; + private int scannedRows; + private boolean endReached = false; + private org.apache.hadoop.conf.Configuration conf; + private static final String COLON = ":"; + + public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) { + this.conf = conf; + this.tableName = tableName; + this.fieldNames = fieldNames; + this.fieldTypeInfos = fieldTypeInfos; + } + + @Override + public void configure(Configuration parameters) { + LOG.info("Initializing HBaseConfiguration"); + connectToTable(); + if(table != null) { + scan = createScanner(); + } + } + + private Scan createScanner() { + Scan scan = new Scan(); + for(String field : fieldNames) { + // select only the fields in the 'selectedFields' + String[] famCol = field.split(COLON); + scan.addColumn(Bytes.toBytes(famCol[0]), Bytes.toBytes(famCol[1])); + } + return scan; + } + + private void connectToTable() { + //use files found in the classpath + if(this.conf == null) { + this.conf =
[jira] [Commented] (FLINK-2168) Add HBaseTableSource
[ https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831221#comment-15831221 ] ASF GitHub Bot commented on FLINK-2168: --- Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/3149#discussion_r97015321 --- Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSource.java --- @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.DataSet; +import org.apache.flink.api.java.ExecutionEnvironment; +import org.apache.flink.api.java.typeutils.RowTypeInfo; +import org.apache.flink.table.sources.BatchTableSource; +import org.apache.flink.table.sources.ProjectableTableSource; +import org.apache.flink.types.Row; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.conf.Configuration; + +/** + * Creates a table source that helps to scan data from an hbase table + * + * Note : the colNames are specified along with a familyName and they are seperated by a ':' + * For eg, cf1:q1 - where cf1 is the familyName and q1 is the qualifier name + */ +public class HBaseTableSource implements BatchTableSource, ProjectableTableSource { + + private Configuration conf; + private String tableName; + private byte[] rowKey; + private String[] colNames; + private TypeInformation[] colTypes; + + public HBaseTableSource(Configuration conf, String tableName, byte[] rowKey, String[] colNames, + TypeInformation[] colTypes) { + this.conf = conf; + this.tableName = Preconditions.checkNotNull(tableName, "Table name"); + this.rowKey = Preconditions.checkNotNull(rowKey, "Rowkey"); --- End diff -- Yes. That is true but do we always want full table scan? Actually in HBase it is better we specify start and end key. So how do we specify that? I have not used this rowKey now but I thought it is better to be used? > Add HBaseTableSource > > > Key: FLINK-2168 > URL: https://issues.apache.org/jira/browse/FLINK-2168 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Affects Versions: 0.9 >Reporter: Fabian Hueske >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Labels: starter > > Add a {{HBaseTableSource}} to read data from a HBase table. The > {{HBaseTableSource}} should implement the {{ProjectableTableSource}} > (FLINK-3848) and {{FilterableTableSource}} (FLINK-3849) interfaces. > The implementation can be based on Flink's {{TableInputFormat}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2168) Add HBaseTableSource
[ https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831218#comment-15831218 ] ASF GitHub Bot commented on FLINK-2168: --- Github user ramkrish86 commented on a diff in the pull request: https://github.com/apache/flink/pull/3149#discussion_r97015267 --- Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSourceInputFormat.java --- @@ -0,0 +1,322 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.io.InputFormat; +import org.apache.flink.api.common.io.LocatableInputSplitAssigner; +import org.apache.flink.api.common.io.RichInputFormat; +import org.apache.flink.api.common.io.statistics.BaseStatistics; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.ResultTypeQueryable; +import org.apache.flink.api.java.typeutils.RowTypeInfo; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.core.io.InputSplitAssigner; +import org.apache.flink.types.Row; +import org.apache.hadoop.hbase.HBaseConfiguration; +import org.apache.hadoop.hbase.TableName; +import org.apache.hadoop.hbase.TableNotFoundException; +import org.apache.hadoop.hbase.client.Scan; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.ClusterConnection; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.ResultScanner; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.HRegionLocator; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +/** + * {@link InputFormat} subclass that wraps the access for HTables. Returns the result as {@link Row} + */ +public class HBaseTableSourceInputFormat extends RichInputFormatimplements ResultTypeQueryable { + + private static final long serialVersionUID = 1L; + + private static final Logger LOG = LoggerFactory.getLogger(HBaseTableSourceInputFormat.class); + private String tableName; + private TypeInformation[] fieldTypeInfos; + private String[] fieldNames; + private transient Table table; + private transient Scan scan; + private transient Connection conn; + private ResultScanner resultScanner = null; + + private byte[] lastRow; + private int scannedRows; + private boolean endReached = false; + private org.apache.hadoop.conf.Configuration conf; + private static final String COLON = ":"; + + public HBaseTableSourceInputFormat(org.apache.hadoop.conf.Configuration conf, String tableName, String[] fieldNames, TypeInformation[] fieldTypeInfos) { + this.conf = conf; + this.tableName = tableName; + this.fieldNames = fieldNames; + this.fieldTypeInfos = fieldTypeInfos; + } + + @Override + public void configure(Configuration parameters) { + LOG.info("Initializing HBaseConfiguration"); + connectToTable(); + if(table != null) { + scan = createScanner(); + } + } + + private Scan createScanner() { + Scan scan = new Scan(); + for(String field : fieldNames) { + // select only the fields in the 'selectedFields' + String[] famCol = field.split(COLON); + scan.addColumn(Bytes.toBytes(famCol[0]), Bytes.toBytes(famCol[1]));
[GitHub] flink issue #3150: [FLINK-4693][tableApi] Add session group-windows for batc...
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3150 @fhueske I have updated the PR according to your comments.The change information is as followsï¼ 1. Fix "remember and read input objects across method calls". 2. Amend some code format. thanks again. SunJincheng. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4693) Add session group-windows for batch tables
[ https://issues.apache.org/jira/browse/FLINK-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831151#comment-15831151 ] ASF GitHub Bot commented on FLINK-4693: --- Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3150 @fhueske I have updated the PR according to your comments.The change information is as follows: 1. Fix "remember and read input objects across method calls". 2. Amend some code format. thanks again. SunJincheng. > Add session group-windows for batch tables > --- > > Key: FLINK-4693 > URL: https://issues.apache.org/jira/browse/FLINK-4693 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL >Reporter: Timo Walther >Assignee: sunjincheng > > Add Session group-windows for batch tables as described in > [FLIP-11|https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations]. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5584) Support Sliding-count row-window on streaming sql
[ https://issues.apache.org/jira/browse/FLINK-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831133#comment-15831133 ] ASF GitHub Bot commented on FLINK-5584: --- Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3175 Hi, @hongyuhong , thank your for your job. Agree with @wuchong 's comments. I add same database OVER example for you: ``` Example data: select * from PeopleInfo ID Name Gender Score 6 LiHuan Man 80 7 LiHuan Man 90 8 LiMing Man 56 9 LiMing Woman 60 10 WangHua Woman 80 ``` ``` --Simple case SELECT name, gender, count(name) OVER () AS num FROM PeopleInfo namegender num LiHuan Man 5 LiHuan Man 5 LiMing Man 5 LiMing Woman 5 WangHuaWoman5 ``` ``` --With ORDER BY case SELECT name,gender,score ROW_NUMBER() OVER (ORDER BY score ASC) AS num FROM PeopleInfo namegender score num LiMing Man 56 1 LiMing Woman 60 2 WangHua Woman 80 3 LiHuan Man 80 4 LiHuan Man 90 5 ``` ``` --With both PARTITION BY and ORDER BY case SELECT [name],gender,score, ROW_NUMBER() OVER(PARTITION BY Gender ORDER BY score ASC) as num FROM PeopleInfo; namegender score num LiMing Man 56 1 LiHuan Man 80 2 LiHuan Man 90 3 LiMing Woman 60 1 WangHua Woman 80 2 ``` ``` --With ROWS PRECEDING and CURRENT ROW case SELECT name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY id ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) as sum FROM PeopleInfo name gender score sum LiHuan Man 80 80 LiHuan Man 90 170 LiMing Man 56 226 LiMing Woman 60 60 WangHua Woman 80 140 SELECT name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER BY id ASC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW ) as sum FROM PeopleInfo name genderscore sum LiHuan Man 80 80 LiHuan Man 90 170 LiMing Man 56 146 LiMing Woman 60 60 WangHua Woman 80 140 ``` ``` --With ROWS FOLLOWING case SELECT id, name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER BY id ASC ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING ) as sum FROM dbo.PeopleInfo id name genderscore sum 6 LiHuan Man 80 170 7 LiHuan Man 90 226 8 LiMing Man 56 146 9 LiMing Woman 60 140 10 WangHua Woman 80 140 SELECT id,name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY id ASC ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING ) as sum FROM PeopleInfo id name gender score sum 6 LiHuan Man 80 170 7 LiHuan Man 90 226 8 LiMing Man 56 226 9 LiMing Woman 60 140 10 WangHua Woman 80 140 SELECT id, name, gender, score,sum(score) OVER (PARTITION BY gender ORDER BY id ASC ROWS BETWEEN 1 PRECEDING AND UNBOUNDED FOLLOWING ) as sum FROM PeopleInfo id name gender score sum 8 LiMing Man 56 146 7 LiHuan Man 90 226 6 LiHuan Man 80 226 10 WangHua Woman 80 140 9 LiMing Woman 60 140 ``` Thank you , SunJincheng. > Support Sliding-count row-window on streaming sql > - > >
[GitHub] flink issue #3175: [FLINK-5584]support sliding-count row-window on streaming...
Github user sunjincheng121 commented on the issue: https://github.com/apache/flink/pull/3175 Hi, @hongyuhong , thank your for your job. Agree with @wuchong 's comments. I add same database OVER example for you: ``` Example data: select * from PeopleInfo ID Name Gender Score 6 LiHuan Man 80 7 LiHuan Man 90 8 LiMing Man 56 9 LiMing Woman 60 10 WangHua Woman 80 ``` ``` --Simple case SELECT name, gender, count(name) OVER () AS num FROM PeopleInfo namegender num LiHuan Man 5 LiHuan Man 5 LiMing Man 5 LiMing Woman 5 WangHuaWoman5 ``` ``` --With ORDER BY case SELECT name,gender,score ROW_NUMBER() OVER (ORDER BY score ASC) AS num FROM PeopleInfo namegender score num LiMing Man 56 1 LiMing Woman 60 2 WangHua Woman 80 3 LiHuan Man 80 4 LiHuan Man 90 5 ``` ``` --With both PARTITION BY and ORDER BY case SELECT [name],gender,score, ROW_NUMBER() OVER(PARTITION BY Gender ORDER BY score ASC) as num FROM PeopleInfo; namegender score num LiMing Man 56 1 LiHuan Man 80 2 LiHuan Man 90 3 LiMing Woman 60 1 WangHua Woman 80 2 ``` ``` --With ROWS PRECEDING and CURRENT ROW case SELECT name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY id ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) as sum FROM PeopleInfo name gender score sum LiHuan Man 80 80 LiHuan Man 90 170 LiMing Man 56 226 LiMing Woman 60 60 WangHua Woman 80 140 SELECT name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER BY id ASC ROWS BETWEEN 1 PRECEDING AND CURRENT ROW ) as sum FROM PeopleInfo name genderscore sum LiHuan Man 80 80 LiHuan Man 90 170 LiMing Man 56 146 LiMing Woman 60 60 WangHua Woman 80 140 ``` ``` --With ROWS FOLLOWING case SELECT id, name, gender, score, sum(score) OVER (PARTITION BY Gender ORDER BY id ASC ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING ) as sum FROM dbo.PeopleInfo id name genderscore sum 6 LiHuan Man 80 170 7 LiHuan Man 90 226 8 LiMing Man 56 146 9 LiMing Woman 60 140 10 WangHua Woman 80 140 SELECT id,name, gender, score, sum(score) OVER (PARTITION BY gender ORDER BY id ASC ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING ) as sum FROM PeopleInfo id name gender score sum 6 LiHuan Man 80 170 7 LiHuan Man 90 226 8 LiMing Man 56 226 9 LiMing Woman 60 140 10 WangHua Woman 80 140 SELECT id, name, gender, score,sum(score) OVER (PARTITION BY gender ORDER BY id ASC ROWS BETWEEN 1 PRECEDING AND UNBOUNDED FOLLOWING ) as sum FROM PeopleInfo id name gender score sum 8 LiMing Man 56 146 7 LiHuan Man 90 226 6 LiHuan Man 80 226 10 WangHua Woman 80 140 9 LiMing Woman 60 140 ``` Thank you , SunJincheng. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #3153: [hotfix] [docs] Insert closing parentheses on "Flink Data...
Github user keijiyoshida commented on the issue: https://github.com/apache/flink/pull/3153 Thanks for your checking and merging! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5584) Support Sliding-count row-window on streaming sql
[ https://issues.apache.org/jira/browse/FLINK-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831112#comment-15831112 ] ASF GitHub Bot commented on FLINK-5584: --- Github user wuchong commented on the issue: https://github.com/apache/flink/pull/3175 Hi @hongyuhong , thank your for your job. But it seems that you misunderstand the SQL OVER syntax. The OVER clause defines a window or user-specified set of rows within a query result set. A window function then computes a value for each row in the window. It is similar to Row-Window proposed in [FLIP-11](https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations), but is different with Sliding Row-count window. For example, OVER (ROWS 2 PRECEDING) means that the window of rows that the function operates on is three rows in size, starting with 2 rows preceding until and including the current row. Say we have a table `T1` ``` t a - 1 1 2 5 3 3 4 5 5 4 6 11 ``` and the following SQL will yield: ```sql SELECT t, a, sum(a) OVER (ROWS 2 PRECEDING) FROM T1 ``` ``` t a avg -- 1 1 1 2 5 6 3 3 9 4 5 13 5 4 12 6 11 20 ``` For Row-window, we would need something more complex, especially when we need to order by timestamp. For example, to support event-time count-window row-window, we need to create a custom operator that collects records in a priority queue ordered by timestamp. Once a watermark is received for the upper bound of a window, the priority queue is used to evaluate the window function (based on count) and to purge too old records. I would suggest this PR to wait for FLINK-4679. When FLINK-4679 is fixed, this PR can be easily supported IMO. > Support Sliding-count row-window on streaming sql > - > > Key: FLINK-5584 > URL: https://issues.apache.org/jira/browse/FLINK-5584 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Yuhong Hong > > Calcite has already support sliding-count row-window, the grammar look like: > select sum(amount) over (rows 10 preceding) from Order; > select sum(amount) over (partition by user rows 10 preceding) from Order; > And it will parse the sql as a LogicalWindow relnode, the logical Window > contains aggregate func info and window info, it's similar to Flink > LogicalWIndowAggregate, so we can add an convert rule to directly convert > LogicalWindow into DataStreamAggregate relnode, and if Calcite support more > grammar, we can extend the convert rule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #3175: [FLINK-5584]support sliding-count row-window on streaming...
Github user wuchong commented on the issue: https://github.com/apache/flink/pull/3175 Hi @hongyuhong , thank your for your job. But it seems that you misunderstand the SQL OVER syntax. The OVER clause defines a window or user-specified set of rows within a query result set. A window function then computes a value for each row in the window. It is similar to Row-Window proposed in [FLIP-11](https://cwiki.apache.org/confluence/display/FLINK/FLIP-11%3A+Table+API+Stream+Aggregations), but is different with Sliding Row-count window. For example, OVER (ROWS 2 PRECEDING) means that the window of rows that the function operates on is three rows in size, starting with 2 rows preceding until and including the current row. Say we have a table `T1` ``` t a - 1 1 2 5 3 3 4 5 5 4 6 11 ``` and the following SQL will yield: ```sql SELECT t, a, sum(a) OVER (ROWS 2 PRECEDING) FROM T1 ``` ``` t a avg -- 1 1 1 2 5 6 3 3 9 4 5 13 5 4 12 6 11 20 ``` For Row-window, we would need something more complex, especially when we need to order by timestamp. For example, to support event-time count-window row-window, we need to create a custom operator that collects records in a priority queue ordered by timestamp. Once a watermark is received for the upper bound of a window, the priority queue is used to evaluate the window function (based on count) and to purge too old records. I would suggest this PR to wait for FLINK-4679. When FLINK-4679 is fixed, this PR can be easily supported IMO. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5584) Support Sliding-count row-window on streaming sql
[ https://issues.apache.org/jira/browse/FLINK-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831036#comment-15831036 ] ASF GitHub Bot commented on FLINK-5584: --- GitHub user hongyuhong opened a pull request: https://github.com/apache/flink/pull/3175 [FLINK-5584]support sliding-count row-window on streaming sql Calcite has already support sliding-count row-window, the grammar look like: select sum(amount) over (rows 10 preceding) from Order; select sum(amount) over (partition by user rows 10 preceding) from Order; And it will parse the sql as a LogicalWindow relnode, the logical Window contains aggregate func info and window info, it's similar to Flink LogicalWIndowAggregate, so we can add an convert rule to directly convert LogicalWindow into DataStreamAggregate relnode. 1. Add HepPlanner to do the window optimize, cause valcano planner can not choose the ProjectToWindow optimize as the best. 2. Add DataStreamWindowRule.scala to convert LogicalWindow to DataStreamAggregate. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hongyuhong/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3175.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3175 commit 2417e24ad676474df3c9fce6701024ca88c6a439 Author: hongyuhong 00223286Date: 2017-01-20T02:02:12Z support sliding-count row-window on streaming sql > Support Sliding-count row-window on streaming sql > - > > Key: FLINK-5584 > URL: https://issues.apache.org/jira/browse/FLINK-5584 > Project: Flink > Issue Type: New Feature > Components: Table API & SQL >Reporter: Yuhong Hong > > Calcite has already support sliding-count row-window, the grammar look like: > select sum(amount) over (rows 10 preceding) from Order; > select sum(amount) over (partition by user rows 10 preceding) from Order; > And it will parse the sql as a LogicalWindow relnode, the logical Window > contains aggregate func info and window info, it's similar to Flink > LogicalWIndowAggregate, so we can add an convert rule to directly convert > LogicalWindow into DataStreamAggregate relnode, and if Calcite support more > grammar, we can extend the convert rule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3175: [FLINK-5584]support sliding-count row-window on st...
GitHub user hongyuhong opened a pull request: https://github.com/apache/flink/pull/3175 [FLINK-5584]support sliding-count row-window on streaming sql Calcite has already support sliding-count row-window, the grammar look like: select sum(amount) over (rows 10 preceding) from Order; select sum(amount) over (partition by user rows 10 preceding) from Order; And it will parse the sql as a LogicalWindow relnode, the logical Window contains aggregate func info and window info, it's similar to Flink LogicalWIndowAggregate, so we can add an convert rule to directly convert LogicalWindow into DataStreamAggregate relnode. 1. Add HepPlanner to do the window optimize, cause valcano planner can not choose the ProjectToWindow optimize as the best. 2. Add DataStreamWindowRule.scala to convert LogicalWindow to DataStreamAggregate. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hongyuhong/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3175.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3175 commit 2417e24ad676474df3c9fce6701024ca88c6a439 Author: hongyuhong 00223286Date: 2017-01-20T02:02:12Z support sliding-count row-window on streaming sql --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-5584) Support Sliding-count row-window on streaming sql
Yuhong Hong created FLINK-5584: -- Summary: Support Sliding-count row-window on streaming sql Key: FLINK-5584 URL: https://issues.apache.org/jira/browse/FLINK-5584 Project: Flink Issue Type: New Feature Components: Table API & SQL Reporter: Yuhong Hong Calcite has already support sliding-count row-window, the grammar look like: select sum(amount) over (rows 10 preceding) from Order; select sum(amount) over (partition by user rows 10 preceding) from Order; And it will parse the sql as a LogicalWindow relnode, the logical Window contains aggregate func info and window info, it's similar to Flink LogicalWIndowAggregate, so we can add an convert rule to directly convert LogicalWindow into DataStreamAggregate relnode, and if Calcite support more grammar, we can extend the convert rule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3141: [FLINK-5520] [table] Disable outer joins with non-...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3141 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5520) Disable outer joins with non-equality predicates
[ https://issues.apache.org/jira/browse/FLINK-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830760#comment-15830760 ] ASF GitHub Bot commented on FLINK-5520: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3141 > Disable outer joins with non-equality predicates > > > Key: FLINK-5520 > URL: https://issues.apache.org/jira/browse/FLINK-5520 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: lincoln.lee >Priority: Blocker > Fix For: 1.2.0 > > > Outer joins with non-equality predicates (and at least one equality > predicate) compute incorrect results. > Since this is not a very common requirement, I propose to disable this > feature for the 1.2.0 release and correctly implement it for a later version. > The fix should add checks in the Table API validation phase (to get a good > error message) and in the DataSetJoinRule to prevent translation of SQL > queries with non-equality predicates on outer joins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3095: [FLINK-5452] [table] Fix SortITCase which will fai...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3095 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5452) Make table unit tests pass under cluster mode
[ https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830759#comment-15830759 ] ASF GitHub Bot commented on FLINK-5452: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3095 > Make table unit tests pass under cluster mode > - > > Key: FLINK-5452 > URL: https://issues.apache.org/jira/browse/FLINK-5452 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Kurt Young >Assignee: Kurt Young > > Currently if we change the test execution mode to > {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will > fail. Need to figure out whether it's the case design problem or there are > some bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3153: [hotfix] [docs] Insert closing parentheses on "Fli...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3153 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-3679) DeserializationSchema should handle zero or more outputs for every input
[ https://issues.apache.org/jira/browse/FLINK-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned FLINK-3679: - Assignee: Haohui Mai > DeserializationSchema should handle zero or more outputs for every input > > > Key: FLINK-3679 > URL: https://issues.apache.org/jira/browse/FLINK-3679 > Project: Flink > Issue Type: Bug > Components: DataStream API, Kafka Connector >Reporter: Jamie Grier >Assignee: Haohui Mai > > There are a couple of issues with the DeserializationSchema API that I think > should be improved. This request has come to me via an existing Flink user. > The main issue is simply that the API assumes that there is a one-to-one > mapping between input and outputs. In reality there are scenarios where one > input message (say from Kafka) might actually map to zero or more logical > elements in the pipeline. > Particularly important here is the case where you receive a message from a > source (such as Kafka) and say the raw bytes don't deserialize properly. > Right now the only recourse is to throw IOException and therefore fail the > job. > This is definitely not good since bad data is a reality and failing the job > is not the right option. If the job fails we'll just end up replaying the > bad data and the whole thing will start again. > Instead in this case it would be best if the user could just return the empty > set. > The other case is where one input message should logically be multiple output > messages. This case is probably less important since there are other ways to > do this but in general it might be good to make the > DeserializationSchema.deserialize() method return a collection rather than a > single element. > Maybe we need to support a DeserializationSchema variant that has semantics > more like that of FlatMap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-5557) Fix link in library methods
[ https://issues.apache.org/jira/browse/FLINK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Hogan closed FLINK-5557. - Resolution: Fixed Fixed in master: 4833e74e7a8b514e8dfa243f3f196ce443cf1c50 1.2: 92e39350045f22942773c1605ca788a547c35f21 > Fix link in library methods > --- > > Key: FLINK-5557 > URL: https://issues.apache.org/jira/browse/FLINK-5557 > Project: Flink > Issue Type: Improvement > Components: Gelly >Affects Versions: 1.2.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.2.0 > > > The link to "Towards real-time community detection in large networks" is > padded with unnecessary and seemingly malformed text. > https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/library_methods.html#community-detection -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5557) Fix link in library methods
[ https://issues.apache.org/jira/browse/FLINK-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830546#comment-15830546 ] ASF GitHub Bot commented on FLINK-5557: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3163 > Fix link in library methods > --- > > Key: FLINK-5557 > URL: https://issues.apache.org/jira/browse/FLINK-5557 > Project: Flink > Issue Type: Improvement > Components: Gelly >Affects Versions: 1.2.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Trivial > Fix For: 1.2.0 > > > The link to "Towards real-time community detection in large networks" is > padded with unnecessary and seemingly malformed text. > https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/gelly/library_methods.html#community-detection -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3163: [FLINK-5557] [gelly] Fix link in library methods
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3163 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-5579) Kerberos not working for Kafka connector using ticket cache
[ https://issues.apache.org/jira/browse/FLINK-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-5579. -- Resolution: Not A Bug > Kerberos not working for Kafka connector using ticket cache > --- > > Key: FLINK-5579 > URL: https://issues.apache.org/jira/browse/FLINK-5579 > Project: Flink > Issue Type: Bug > Components: Security, YARN >Reporter: Tzu-Li (Gordon) Tai >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > > Setup: security using Kerberos ticket cache, single Flink job deployed on YARN > The Kerberos ticket cache doesn't seem to be picked up / sent to TaskManager > containers when using the Kafka connector when deployed on YARN (when > deployed using standalone, this works normally). > {code} > Caused by: org.apache.kafka.common.KafkaException: > javax.security.auth.login.LoginException: Unable to obtain Princpal Name for > authentication > at > org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74) > at > org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60) > at > org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79) > at > org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:271) > ... 23 more > Caused by: javax.security.auth.login.LoginException: Unable to obtain > Princpal Name for authentication > at > com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:804) > at > com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:675) > at > com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:588) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) > at > javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) > at > javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) > at > javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) > at java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) > at > javax.security.auth.login.LoginContext.login(LoginContext.java:595) > at > org.apache.kafka.common.security.kerberos.Login.login(Login.java:298) > at > org.apache.kafka.common.security.kerberos.Login.(Login.java:104) > at > org.apache.kafka.common.security.kerberos.LoginManager.(LoginManager.java:44) > at > org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85) > at > org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55) > ... 26 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5579) Kerberos not working for Kafka connector using ticket cache
[ https://issues.apache.org/jira/browse/FLINK-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830523#comment-15830523 ] Tzu-Li (Gordon) Tai commented on FLINK-5579: This actually isn't a bug. With YARN, Kerberos security for Kafka only works with keytabs. Please see FLINK-3929. On the other hand, we should probably make this more clear in the documentation. > Kerberos not working for Kafka connector using ticket cache > --- > > Key: FLINK-5579 > URL: https://issues.apache.org/jira/browse/FLINK-5579 > Project: Flink > Issue Type: Bug > Components: Security, YARN >Reporter: Tzu-Li (Gordon) Tai >Assignee: Tzu-Li (Gordon) Tai >Priority: Critical > > Setup: security using Kerberos ticket cache, single Flink job deployed on YARN > The Kerberos ticket cache doesn't seem to be picked up / sent to TaskManager > containers when using the Kafka connector when deployed on YARN (when > deployed using standalone, this works normally). > {code} > Caused by: org.apache.kafka.common.KafkaException: > javax.security.auth.login.LoginException: Unable to obtain Princpal Name for > authentication > at > org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74) > at > org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60) > at > org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79) > at > org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:271) > ... 23 more > Caused by: javax.security.auth.login.LoginException: Unable to obtain > Princpal Name for authentication > at > com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:804) > at > com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:675) > at > com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:588) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) > at > javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) > at > javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) > at > javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) > at java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) > at > javax.security.auth.login.LoginContext.login(LoginContext.java:595) > at > org.apache.kafka.common.security.kerberos.Login.login(Login.java:298) > at > org.apache.kafka.common.security.kerberos.Login.(Login.java:104) > at > org.apache.kafka.common.security.kerberos.LoginManager.(LoginManager.java:44) > at > org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85) > at > org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55) > ... 26 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-5583) Support flexible error handling in the Kafka consumer
Haohui Mai created FLINK-5583: - Summary: Support flexible error handling in the Kafka consumer Key: FLINK-5583 URL: https://issues.apache.org/jira/browse/FLINK-5583 Project: Flink Issue Type: Improvement Reporter: Haohui Mai Assignee: Haohui Mai We found that it is valuable to allow the applications to handle errors and exceptions in the Kafka consumer in order to build a robust application in production. The context is the following: (1) We have schematized, Avro records flowing through Kafka. (2) The decoder implements the DeserializationSchema to decode the records. (3) Occasionally there are corrupted records (e.g., schema issues). The streaming pipeline might want to bail out (which is the current behavior) or to skip the corrupted records depending on the applications. Two options are available: (1) Have a variant of DeserializationSchema to return a FlatMap like structure as suggested in FLINK-3679. (2) Allow the applications to catch and handle the exception by exposing some APIs that are similar to the {{ExceptionProxy}}. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5459) Add documentation how to debug classloading and event time issues
[ https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-5459: -- Description: {{docs/monitoring/debugging_classloading.md}} was: {{docs/monitoring/debugging_classloading.md}} {{docs/monitoring/debugging_event_time.md}} > Add documentation how to debug classloading and event time issues > - > > Key: FLINK-5459 > URL: https://issues.apache.org/jira/browse/FLINK-5459 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Stephan Ewen > Fix For: 1.2.0, 1.3.0 > > > {{docs/monitoring/debugging_classloading.md}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5459) Add documentation how to debug classloading
[ https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-5459: -- Summary: Add documentation how to debug classloading (was: Add documentation how to debug classloading and event time issues) > Add documentation how to debug classloading > --- > > Key: FLINK-5459 > URL: https://issues.apache.org/jira/browse/FLINK-5459 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Stephan Ewen > Fix For: 1.2.0, 1.3.0 > > > {{docs/monitoring/debugging_classloading.md}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-5459) Add documentation how to debug classloading issues
[ https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-5459: -- Summary: Add documentation how to debug classloading issues (was: Add documentation how to debug classloading) > Add documentation how to debug classloading issues > -- > > Key: FLINK-5459 > URL: https://issues.apache.org/jira/browse/FLINK-5459 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Stephan Ewen > Fix For: 1.2.0, 1.3.0 > > > {{docs/monitoring/debugging_classloading.md}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-5555) Add documentation about debugging watermarks
[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger resolved FLINK-. --- Resolution: Fixed Fix Version/s: 1.3.0 Resolved for 1.3 in master: http://git-wip-us.apache.org/repos/asf/flink/commit/b86a3d76 Resolved for 1.2 in 7f3c8ff811de620cb9f272d7b6a179bdb9aa33db > Add documentation about debugging watermarks > > > Key: FLINK- > URL: https://issues.apache.org/jira/browse/FLINK- > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.2.0 >Reporter: Robert Metzger >Assignee: Robert Metzger > Fix For: 1.2.0, 1.3.0 > > > This was a frequent question on the mailing list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5459) Add documentation how to debug classloading and event time issues
[ https://issues.apache.org/jira/browse/FLINK-5459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830408#comment-15830408 ] Robert Metzger commented on FLINK-5459: --- Event time debugging has been addressed in FLINK- already. I'll edit this JIRA. > Add documentation how to debug classloading and event time issues > - > > Key: FLINK-5459 > URL: https://issues.apache.org/jira/browse/FLINK-5459 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Stephan Ewen > Fix For: 1.2.0, 1.3.0 > > > {{docs/monitoring/debugging_classloading.md}} > {{docs/monitoring/debugging_event_time.md}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5555) Add documentation about debugging watermarks
[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830405#comment-15830405 ] ASF GitHub Bot commented on FLINK-: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3170 > Add documentation about debugging watermarks > > > Key: FLINK- > URL: https://issues.apache.org/jira/browse/FLINK- > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.2.0 >Reporter: Robert Metzger >Assignee: Robert Metzger > Fix For: 1.2.0 > > > This was a frequent question on the mailing list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #3170: [FLINK-5555] Document how to debug event time usin...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3170 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5555) Add documentation about debugging watermarks
[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830401#comment-15830401 ] ASF GitHub Bot commented on FLINK-: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3170 Thanks a lot for the quick review. I'll merge it. > Add documentation about debugging watermarks > > > Key: FLINK- > URL: https://issues.apache.org/jira/browse/FLINK- > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.2.0 >Reporter: Robert Metzger >Assignee: Robert Metzger > Fix For: 1.2.0 > > > This was a frequent question on the mailing list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #3170: [FLINK-5555] Document how to debug event time using water...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3170 Thanks a lot for the quick review. I'll merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5355) Handle AmazonKinesisException gracefully in Kinesis Streaming Connector
[ https://issues.apache.org/jira/browse/FLINK-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830326#comment-15830326 ] ASF GitHub Bot commented on FLINK-5355: --- Github user skidder commented on the issue: https://github.com/apache/flink/pull/3078 Thank you @tzulitai for the feedback on the styling! I've pushed a commit that addresses your comments. > Handle AmazonKinesisException gracefully in Kinesis Streaming Connector > --- > > Key: FLINK-5355 > URL: https://issues.apache.org/jira/browse/FLINK-5355 > Project: Flink > Issue Type: Improvement > Components: Kinesis Connector >Affects Versions: 1.2.0, 1.1.3 >Reporter: Scott Kidder >Assignee: Scott Kidder > > My Flink job that consumes from a Kinesis stream must be restarted at least > once daily due to an uncaught AmazonKinesisException when reading from > Kinesis. The complete stacktrace looks like: > {noformat} > com.amazonaws.services.kinesis.model.AmazonKinesisException: null (Service: > AmazonKinesis; Status Code: 500; Error Code: InternalFailure; Request ID: > dc1b7a1a-1b97-1a32-8cd5-79a896a55223) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) > at > com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) > at > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1747) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1723) > at > com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:858) > at > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getRecords(KinesisProxy.java:193) > at > org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.getRecords(ShardConsumer.java:268) > at > org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.run(ShardConsumer.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {noformat} > It's interesting that the Kinesis endpoint returned a 500 status code, but > that's outside the scope of this issue. > I think we can handle this exception in the same manner as a > ProvisionedThroughputException: performing an exponential backoff and > retrying a finite number of times before throwing an exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #3078: [FLINK-5355] Handle AmazonKinesisException gracefully in ...
Github user skidder commented on the issue: https://github.com/apache/flink/pull/3078 Thank you @tzulitai for the feedback on the styling! I've pushed a commit that addresses your comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-5452) Make table unit tests pass under cluster mode
[ https://issues.apache.org/jira/browse/FLINK-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830287#comment-15830287 ] ASF GitHub Bot commented on FLINK-5452: --- Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3095 Thanks for fixing the tests @KurtYoung! +1 to merge > Make table unit tests pass under cluster mode > - > > Key: FLINK-5452 > URL: https://issues.apache.org/jira/browse/FLINK-5452 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Kurt Young >Assignee: Kurt Young > > Currently if we change the test execution mode to > {{TestExecutionMode.CLUSTER}} in {{TableProgramsTestBase}}, some cases will > fail. Need to figure out whether it's the case design problem or there are > some bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #3095: [FLINK-5452] [table] Fix SortITCase which will fail under...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3095 Thanks for fixing the tests @KurtYoung! +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---