[jira] [Commented] (FLINK-2055) Implement Streaming HBaseSink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518160#comment-15518160
 ] 

ASF GitHub Bot commented on FLINK-2055:
---

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2332
  
Thanks for very detailed and helpful comments, let me work on it :-)


> Implement Streaming HBaseSink
> -
>
> Key: FLINK-2055
> URL: https://issues.apache.org/jira/browse/FLINK-2055
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming, Streaming Connectors
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Erli Ding
>
> As per : 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Write-Stream-to-HBase-td1300.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2332: [FLINK-2055] Implement Streaming HBaseSink

[delding]

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2332
  
Thanks for very detailed and helpful comments, let me work on it :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4218) Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." causes task restarting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517475#comment-15517475
 ] 

ASF GitHub Bot commented on FLINK-4218:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/2544#discussion_r80319776
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/StateObject.java ---
@@ -47,7 +49,7 @@
 * If the the size is not known, return {@code 0}.
 *
 * @return Size of the state in bytes.
-* @throws Exception If the operation fails during size retrieval.
+* @throws IOException If the operation fails during size retrieval.
 */
-   long getStateSize() throws Exception;
+   long getStateSize() throws IOException;
--- End diff --

I think with the change in `StreamStateHandle`, even throwing IOException 
becomes obsolete now for all existing implementations. We might remove it.


> Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." 
> causes task restarting
> --
>
> Key: FLINK-4218
> URL: https://issues.apache.org/jira/browse/FLINK-4218
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sergii Koshel
>
> Sporadically see exception as below. And restart of task because of it.
> {code:title=Exception|borderStyle=solid}
> java.lang.RuntimeException: Error triggering a checkpoint as the result of 
> receiving checkpoint barrier
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:785)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:775)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: No such file or directory: 
> s3:///flink/checkpoints/ece317c26960464ba5de75f3bbc38cb2/chk-8810/96eebbeb-de14-45c7-8ebb-e7cde978d6d3
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
>   at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
>   at 
> org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
>   at 
> org.apache.flink.runtime.state.filesystem.FileStreamStateHandle.getStateSize(FileStreamStateHandle.java:58)
>   at 
> org.apache.flink.runtime.state.AbstractStateBackend$DataInputViewHandle.getStateSize(AbstractStateBackend.java:482)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:77)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:604)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:779)
>   ... 8 more
> {code}
> File actually exists on S3. 
> I suppose it is related to some race conditions with S3 but would be good to 
> retry a few times before stop task execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517350#comment-15517350
 ] 

ASF GitHub Bot commented on FLINK-4624:
---

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/2527#discussion_r80311778
  
--- Diff: 
flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/Summarization.java
 ---
@@ -226,11 +247,15 @@ public void setGroupRepresentativeId(K 
groupRepresentativeId) {
}
 
public VGV getVertexGroupValue() {
-   return f2;
+   return f2.isLeft() ? f2.left() : null;
}
 
public void setVertexGroupValue(VGV vertexGroupValue) {
-   f2 = vertexGroupValue;
+   if (vertexGroupValue == null) {
+   f2 = new 
Either.Right<>(NullValue.getInstance());
--- End diff --

I am tracing `VertexGroupReducer.reduce` calling 
`createGroupRepresentativeTuple` which calls `setVertexGroupValue` which 
creates a new `Either.Left` or `Either.Right`. Then `VertexGroupReducer.reduce` 
calls `reset` which creates a new `Either.Right`.


> Gelly's summarization algorithm cannot deal with null vertex group values
> -
>
> Key: FLINK-4624
> URL: https://issues.apache.org/jira/browse/FLINK-4624
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly
>Reporter: Till Rohrmann
>Assignee: Martin Junghanns
> Fix For: 1.2.0
>
>
> Gelly's {{Summarization}} algorithm cannot handle null values in the 
> `VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
> value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
> handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517294#comment-15517294
 ] 

ASF GitHub Bot commented on FLINK-4624:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
`ToNullValue` is now committed in a new `translator` package in Gelly. I 
think that one will be generally very useful.

I created FLINK-4673 so we can look at fixing the type extraction with the 
Either type. Using the recently added `TypeInfoFactory` it was quite simple and 
I will create the PR after tests pass (I also removed the explicit code from 
`TypeExtractor`).


> Gelly's summarization algorithm cannot deal with null vertex group values
> -
>
> Key: FLINK-4624
> URL: https://issues.apache.org/jira/browse/FLINK-4624
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly
>Reporter: Till Rohrmann
>Assignee: Martin Junghanns
> Fix For: 1.2.0
>
>
> Gelly's {{Summarization}} algorithm cannot handle null values in the 
> `VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
> value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
> handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4673) TypeFactory for Either type

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-4673:
-

 Summary: TypeFactory for Either type
 Key: FLINK-4673
 URL: https://issues.apache.org/jira/browse/FLINK-4673
 Project: Flink
  Issue Type: Improvement
  Components: Core
Affects Versions: 1.2.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Minor


I was able to resolve the requirement to specify an explicit 
{{TypeInformation}} in the pull request for FLINK-4624 by creating a 
{{TypeInfoFactory}} for the {{Either}} type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4668) Fix positive random int generation

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan updated FLINK-4668:
--
Issue Type: Improvement  (was: Bug)

> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
> Fix For: 1.2.0
>
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4668) Fix positive random int generation

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-4668.
-
Resolution: Implemented

Implemented in 40c978b0443d9cb3ee66b6e0e63ab61416deac25

> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Improvement
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
> Fix For: 1.2.0
>
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4668) Fix positive random int generation

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan updated FLINK-4668:
--
Fix Version/s: 1.2.0

> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
> Fix For: 1.2.0
>
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4664) Add translator to NullValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517244#comment-15517244
 ] 

ASF GitHub Bot commented on FLINK-4664:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2536


> Add translator to NullValue
> ---
>
> Key: FLINK-4664
> URL: https://issues.apache.org/jira/browse/FLINK-4664
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.2.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.2.0
>
>
> Existing translators convert from LongValue (the output label type of graph 
> generators) to IntValue, StringValue, and an offset LongValue. Translators 
> can also be used to convert vertex or edge values. This translator will be 
> appropriate for translating these vertex or edge values to NullValue when the 
> values are not used in an algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2539: [FLINK-4668] Fix positive random int generation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2539


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4668) Fix positive random int generation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517092#comment-15517092
 ] 

ASF GitHub Bot commented on FLINK-4668:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2539
  
I'll merge this ...


> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4482) numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-4482.
-
   Resolution: Fixed
 Assignee: Ted Yu
Fix Version/s: 1.2.0

Fixed via 5d0358af46e0f3683224986a7718adca88f504db

Thank you for the contribution!

> numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock
> --
>
> Key: FLINK-4482
> URL: https://issues.apache.org/jira/browse/FLINK-4482
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 1.2.0
>
>
> In CheckpointCoordinator#stopCheckpointScheduler() :
> {code}
> synchronized (lock) {
> ...
>   numUnsuccessfulCheckpointsTriggers = 0;
> {code}
> triggerLock is not held in the above operation.
> See comment for triggerLock earlier in triggerCheckpoint():
> {code}
> // we lock with a special lock to make sure that trigger requests do not 
> overtake each other.
> // this is not done with the coordinator-wide lock, because the 
> 'checkpointIdCounter'
> // may issue blocking operations. Using a different lock than teh 
> coordinator-wide lock,
> // we avoid blocking the processing of 'acknowledge/decline' messages 
> during that time.
> synchronized (triggerLock) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4482) numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-4482.
---

> numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock
> --
>
> Key: FLINK-4482
> URL: https://issues.apache.org/jira/browse/FLINK-4482
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 1.2.0
>
>
> In CheckpointCoordinator#stopCheckpointScheduler() :
> {code}
> synchronized (lock) {
> ...
>   numUnsuccessfulCheckpointsTriggers = 0;
> {code}
> triggerLock is not held in the above operation.
> See comment for triggerLock earlier in triggerCheckpoint():
> {code}
> // we lock with a special lock to make sure that trigger requests do not 
> overtake each other.
> // this is not done with the coordinator-wide lock, because the 
> 'checkpointIdCounter'
> // may issue blocking operations. Using a different lock than teh 
> coordinator-wide lock,
> // we avoid blocking the processing of 'acknowledge/decline' messages 
> during that time.
> synchronized (triggerLock) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516983#comment-15516983
 ] 

ASF GitHub Bot commented on FLINK-3930:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2425
  
Thanks @StephanEwen 


> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
>  Issue Type: New Feature
>  Components: Security
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Service-level authorization is the initial authorization mechanism to ensure 
> clients (or servers) connecting to the Flink cluster are authorized to do so. 
>   The purpose is to prevent a cluster from being used by an unauthorized 
> user, whether to execute jobs, disrupt cluster functionality, or gain access 
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2425: FLINK-3930 Added shared secret based authorization for Fl...

[vijikarthi]

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2425
  
Thanks @StephanEwen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2425: FLINK-3930 Added shared secret based authorization for Fl...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2425
  
I think we are waiting for @rmetzger to follow up. He is out of office for 
a few days...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3930) Implement Service-Level Authorization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516974#comment-15516974
 ] 

ASF GitHub Bot commented on FLINK-3930:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2425
  
I think we are waiting for @rmetzger to follow up. He is out of office for 
a few days...


> Implement Service-Level Authorization
> -
>
> Key: FLINK-3930
> URL: https://issues.apache.org/jira/browse/FLINK-3930
> Project: Flink
>  Issue Type: New Feature
>  Components: Security
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Service-level authorization is the initial authorization mechanism to ensure 
> clients (or servers) connecting to the Flink cluster are authorized to do so. 
>   The purpose is to prevent a cluster from being used by an unauthorized 
> user, whether to execute jobs, disrupt cluster functionality, or gain access 
> to secrets stored within the cluster.
> Implement service-level authorization as described in the design doc.
> - Introduce a shared secret cookie
> - Enable Akka security cookie
> - Implement data transfer authentication
> - Secure the web dashboard



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4668) Fix positive random int generation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516971#comment-15516971
 ] 

ASF GitHub Bot commented on FLINK-4668:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2539
  
Yes, looks good to me, too.
(I retract my earlier statement, I was apparently a bit tired and mixed up 
original version and edit).


> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2539: [FLINK-4668] Fix positive random int generation

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2539
  
Yes, looks good to me, too.
(I retract my earlier statement, I was apparently a bit tired and mixed up 
original version and edit).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2425: FLINK-3930 Added shared secret based authorization for Fl...

[vijikarthi]

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2425
  
@rmetzger @StephanEwen are you guys waiting for any inputs from my side?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4632) when yarn nodemanager lost, flink hung

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516956#comment-15516956
 ] 

Stephan Ewen commented on FLINK-4632:
-

Okay, that should have a fix in the next days.
Would you be able to try out a snapshot version to validate that the fix works? 
That would be great!

> when yarn nodemanager lost,  flink hung
> ---
>
> Key: FLINK-4632
> URL: https://issues.apache.org/jira/browse/FLINK-4632
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, Streaming
>Affects Versions: 1.2.0, 1.1.2
> Environment: cdh5.5.1  jdk1.7 flink1.1.2  1.2-snapshot   kafka0.8.2
>Reporter: 刘喆
>
> When run flink streaming on yarn,  using kafka as source,  it runs well when 
> start. But after long run, for example  8 hours, dealing 60,000,000+ 
> messages, it hung: no messages consumed,   one taskmanager is CANCELING, the 
> exception show:
> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: 
> connection timeout
>   at 
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.exceptionCaught(PartitionRequestClientHandler.java:152)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:835)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: 连接超时
>   at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>   at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>   at 
> io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
>   at 
> io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
>   ... 6 more
> after applyhttps://issues.apache.org/jira/browse/FLINK-4625   
> it show:
> java.lang.Exception: TaskManager was lost/killed: 
> ResourceID{resourceId='container_1471620986643_744852_01_001400'} @ 
> 38.slave.adh (dataPort=45349)
>   at 
> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:162)
>   at 
> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533)
>   at 
> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:138)
>   at 
> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167)
>   at 
> 

[jira] [Commented] (FLINK-4482) numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516950#comment-15516950
 ] 

ASF GitHub Bot commented on FLINK-4482:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2421


> numUnsuccessfulCheckpointsTriggers is accessed without holding triggerLock
> --
>
> Key: FLINK-4482
> URL: https://issues.apache.org/jira/browse/FLINK-4482
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> In CheckpointCoordinator#stopCheckpointScheduler() :
> {code}
> synchronized (lock) {
> ...
>   numUnsuccessfulCheckpointsTriggers = 0;
> {code}
> triggerLock is not held in the above operation.
> See comment for triggerLock earlier in triggerCheckpoint():
> {code}
> // we lock with a special lock to make sure that trigger requests do not 
> overtake each other.
> // this is not done with the coordinator-wide lock, because the 
> 'checkpointIdCounter'
> // may issue blocking operations. Using a different lock than teh 
> coordinator-wide lock,
> // we avoid blocking the processing of 'acknowledge/decline' messages 
> during that time.
> synchronized (triggerLock) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2421: FLINK-4482 numUnsuccessfulCheckpointsTriggers is a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2421


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4672) TaskManager accidentally decorates Kill messages

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-4672:
---

 Summary: TaskManager accidentally decorates Kill messages
 Key: FLINK-4672
 URL: https://issues.apache.org/jira/browse/FLINK-4672
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.1.2, 1.2.0
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.2.0, 1.1.3






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4668) Fix positive random int generation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516928#comment-15516928
 ] 

ASF GitHub Bot commented on FLINK-4668:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2539
  
These are temporary files automatically deleted by the client. To handle 
paths with leading dashes one can prefix with a directory (`vi ./-44.txt`) or 
place after a double dash (`vi -- -44.txt`).

This looks to me like a safe change (I'd first question why we're prefixing 
with a random string when Java is doing the same in `File.createTempFile`). On 
the other hand, this isn't fixing an error and there are limitless ways to 
refactor the existing codebase. And fortunately the anti-pattern is well-known.


> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2539: [FLINK-4668] Fix positive random int generation

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2539
  
These are temporary files automatically deleted by the client. To handle 
paths with leading dashes one can prefix with a directory (`vi ./-44.txt`) or 
place after a double dash (`vi -- -44.txt`).

This looks to me like a safe change (I'd first question why we're prefixing 
with a random string when Java is doing the same in `File.createTempFile`). On 
the other hand, this isn't fixing an error and there are limitless ways to 
refactor the existing codebase. And fortunately the anti-pattern is well-known.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4671) Table API can not be built

[Timo Walther (JIRA)]

Timo Walther created FLINK-4671:
---

 Summary: Table API can not be built
 Key: FLINK-4671
 URL: https://issues.apache.org/jira/browse/FLINK-4671
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: Timo Walther


Running {{mvn clean verify}} in {{flink-table}} results in a build failure.

{code}
[ERROR] Failed to execute goal on project flink-table_2.10: Could not resolve 
dependencies for project org.apache.flink:flink-table_2.10:jar:1.2-SNAPSHOT: 
Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced -> [Help 1]
{code}

However, the master can be built successfully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4549) Test and document implicitly supported SQL functions

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-4549.
-
   Resolution: Fixed
Fix Version/s: 1.2.0

Fixed in 9a1bc021aed0a3eec8c6eabb843d15b8c2b0b43f.

> Test and document implicitly supported SQL functions
> 
>
> Key: FLINK-4549
> URL: https://issues.apache.org/jira/browse/FLINK-4549
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.2.0
>
>
> Calcite supports many SQL functions by translating them into {{RexNode}} s. 
> However, SQL functions like {{NULLIF}}, {{OVERLAPS}} are neither tested nor 
> document although supported.
> These functions should be tested and added to the documentation. We could 
> adopt parts from the Calcite documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4549) Test and document implicitly supported SQL functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516778#comment-15516778
 ] 

ASF GitHub Bot commented on FLINK-4549:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2500


> Test and document implicitly supported SQL functions
> 
>
> Key: FLINK-4549
> URL: https://issues.apache.org/jira/browse/FLINK-4549
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
> Fix For: 1.2.0
>
>
> Calcite supports many SQL functions by translating them into {{RexNode}} s. 
> However, SQL functions like {{NULLIF}}, {{OVERLAPS}} are neither tested nor 
> document although supported.
> These functions should be tested and added to the documentation. We could 
> adopt parts from the Calcite documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2500: [FLINK-4549] [table] Test and document implicitly ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2500


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4550) Clearly define SQL operator table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516700#comment-15516700
 ] 

ASF GitHub Bot commented on FLINK-4550:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2502


> Clearly define SQL operator table
> -
>
> Key: FLINK-4550
> URL: https://issues.apache.org/jira/browse/FLINK-4550
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>  Labels: starter
>
> Currently, we use {{SqlStdOperatorTable.instance()}} for setting all 
> supported operations. However, not all of them are actually supported. 
> {{FunctionCatalog}} should only return those operators that are tested and 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2502: [FLINK-4550] [table] Clearly define SQL operator t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2502


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2539: [FLINK-4668] Fix positive random int generation

[apivovarov]

Github user apivovarov commented on the issue:

https://github.com/apache/flink/pull/2539
  
it's not even convenient to work with files started with `-`, e.g.
```$ vi -44.txt
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Aug  1 2016 19:37:21)
Unknown option argument: "-44.txt"
More info with: "vim -h"```

```$ rm -rf "-4.txt"
rm: illegal option -- 4
usage: rm [-f | -i] [-dPRrvW] file ...
   unlink file```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4668) Fix positive random int generation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516689#comment-15516689
 ] 

ASF GitHub Bot commented on FLINK-4668:
---

Github user apivovarov commented on the issue:

https://github.com/apache/flink/pull/2539
  
it's not even convenient to work with files started with `-`, e.g.
```$ vi -44.txt
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Aug  1 2016 19:37:21)
Unknown option argument: "-44.txt"
More info with: "vim -h"```

```$ rm -rf "-4.txt"
rm: illegal option -- 4
usage: rm [-f | -i] [-dPRrvW] file ...
   unlink file```


> Fix positive random int generation
> --
>
> Key: FLINK-4668
> URL: https://issues.apache.org/jira/browse/FLINK-4668
> Project: Flink
>  Issue Type: Bug
>  Components: Client
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> According to java spec
> {code}Math.abs(Integer.MIN_VALUE) == Integer.MIN_VALUE{code}
> So, {code}Math.abs(rnd.nextInt()){code} might return negative value
> To generate positive random int value we can use 
> {code}rnd.nextInt(Integer.MAX_VALUE){code}
> Integer.MAX_VALUE will be excluded btw



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4590) Some Table API tests are failing when debug lvl is set to DEBUG

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516659#comment-15516659
 ] 

ASF GitHub Bot commented on FLINK-4590:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2504#discussion_r80259892
  
--- Diff: 
flink-libraries/flink-table/src/test/resources/log4j-test.properties ---
@@ -0,0 +1,27 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+# Set root logger level to OFF to not flood build logs
+# set manually to INFO for debugging purposes
+log4j.rootLogger=OFF, testlogger
--- End diff --

Ah OK. The PR description didn't say that the missing cost functions were 
causing the failing tests. 


> Some Table API tests are failing when debug lvl is set to DEBUG
> ---
>
> Key: FLINK-4590
> URL: https://issues.apache.org/jira/browse/FLINK-4590
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Timo Walther
>
> For debugging another issue, I've set the log level on travis to DEBUG.
> After that, the Table API tests started failing
> {code}
> Failed tests: 
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
> {code}
> Probably Calcite is executing additional assertions depending on the debug 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4590) Some Table API tests are failing when debug lvl is set to DEBUG

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516662#comment-15516662
 ] 

ASF GitHub Bot commented on FLINK-4590:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2504
  
+1 to merge


> Some Table API tests are failing when debug lvl is set to DEBUG
> ---
>
> Key: FLINK-4590
> URL: https://issues.apache.org/jira/browse/FLINK-4590
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Timo Walther
>
> For debugging another issue, I've set the log level on travis to DEBUG.
> After that, the Table API tests started failing
> {code}
> Failed tests: 
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
> {code}
> Probably Calcite is executing additional assertions depending on the debug 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4620) Automatically creating savepoints

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516660#comment-15516660
 ] 

Niels Basjes commented on FLINK-4620:
-

Done.

> Automatically creating savepoints
> -
>
> Key: FLINK-4620
> URL: https://issues.apache.org/jira/browse/FLINK-4620
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Affects Versions: 1.1.2
>Reporter: Niels Basjes
>
> In the current versions of Flink you can run an external command and then a 
> savepoint is persisted in a durable location.
> Feature request: Make this a lot more automatic and easy to use.
> _Proposed workflow_
> # In my application I do something like this:
> {code}
> StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setStateBackend(new FsStateBackend("hdfs:///tmp/applicationState"));
> env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
> env.enableAutomaticSavePoints(30);
> env.enableAutomaticSavePointCleaner(10);
> {code}
> # When I start the application for the first time the state backend is 
> 'empty'. 
> I expect the system to start in a clean state.
> After 10 minutes (30ms) a savepoint is created and stored.
> # When I stop and start the topology again it will automatically restore the 
> last available savepoint.
> Things to think about:
> * Note that this feature still means the manual version is useful!!
> * What to do on startup if the state is incompatible with the topology? Fail 
> the startup?
> * How many automatic savepoints to we keep? Only the last one?
> * Perhaps the API should allow multiple automatic savepoints at different 
> intervals in different locations.
> {code}
> // Make every 10 minutes and keep the last 10
> env.enableAutomaticSavePoints(30, new 
> FsStateBackend("hdfs:///tmp/applicationState"), 10);
> // Make every 24 hours and keep the last 30
> // Useful for being able to reproduce a problem a few days later
> env.enableAutomaticSavePoints(8640, new 
> FsStateBackend("hdfs:///tmp/applicationDailyStateSnapshot"), 30);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2504: [FLINK-4590] [table] Some Table API tests are fail...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2504#discussion_r80259892
  
--- Diff: 
flink-libraries/flink-table/src/test/resources/log4j-test.properties ---
@@ -0,0 +1,27 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+# Set root logger level to OFF to not flood build logs
+# set manually to INFO for debugging purposes
+log4j.rootLogger=OFF, testlogger
--- End diff --

Ah OK. The PR description didn't say that the missing cost functions were 
causing the failing tests. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2504: [FLINK-4590] [table] Some Table API tests are failing whe...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2504
  
+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4511) Schedule periodic savepoints

[Niels Basjes (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516657#comment-15516657
 ] 

Niels Basjes commented on FLINK-4511:
-

As requested by [~aljoscha] my comments copied from FLINK-4620

In the current versions of Flink you can run an external command and then a 
savepoint is persisted in a durable location.

Feature request: Make this a lot more automatic and easy to use.

_Proposed workflow_
# In my application I do something like this:
{code}
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
env.setStateBackend(new FsStateBackend("hdfs:///tmp/applicationState"));
env.enableCheckpointing(5000, CheckpointingMode.EXACTLY_ONCE);
env.enableAutomaticSavePoints(30);
env.enableAutomaticSavePointCleaner(10);
{code}

# When I start the application for the first time the state backend is 'empty'. 
I expect the system to start in a clean state.
After 10 minutes (30ms) a savepoint is created and stored.
# When I stop and start the topology again it will automatically restore the 
last available savepoint.

Things to think about:
* Note that this feature still means the manual version is useful!!
* What to do on startup if the state is incompatible with the topology? Fail 
the startup?
* How many automatic savepoints to we keep? Only the last one?
* Perhaps the API should allow multiple automatic savepoints at different 
intervals in different locations.
{code}
// Make every 10 minutes and keep the last 10
env.enableAutomaticSavePoints(30, new 
FsStateBackend("hdfs:///tmp/applicationState"), 10);

// Make every 24 hours and keep the last 30
// Useful for being able to reproduce a problem a few days later
env.enableAutomaticSavePoints(8640, new 
FsStateBackend("hdfs:///tmp/applicationDailyStateSnapshot"), 30);
{code}



> Schedule periodic savepoints
> 
>
> Key: FLINK-4511
> URL: https://issues.apache.org/jira/browse/FLINK-4511
> Project: Flink
>  Issue Type: Sub-task
>  Components: State Backends, Checkpointing
>Reporter: Ufuk Celebi
>
> Allow triggering of periodic savepoints, which are kept in a bounded queue 
> (like completed checkpoints currently, but separate).
> If there is no periodic checkpointing enabled, only periodic savepoints 
> should be schedulded.
> If periodic checkpointing is enabled, the periodic savepoints should not be 
> scheduled independently, but instead the checkpoint scheduler should trigger 
> a savepoint instead. This will ensure that no unexpected interference between 
> checkpoints and savepoints happens. For this, I would restrict the savepoint 
> interval to be a multiple of the checkpointing interval (if enabled).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4550) Clearly define SQL operator table

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516646#comment-15516646
 ] 

ASF GitHub Bot commented on FLINK-4550:
---

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2502
  
Thanks @fhueske. Merging...


> Clearly define SQL operator table
> -
>
> Key: FLINK-4550
> URL: https://issues.apache.org/jira/browse/FLINK-4550
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>  Labels: starter
>
> Currently, we use {{SqlStdOperatorTable.instance()}} for setting all 
> supported operations. However, not all of them are actually supported. 
> {{FunctionCatalog}} should only return those operators that are tested and 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4613) Extend ALS to handle implicit feedback datasets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516644#comment-15516644
 ] 

ASF GitHub Bot commented on FLINK-4613:
---

Github user gaborhermann commented on the issue:

https://github.com/apache/flink/pull/2542
  
We did not measure performance against Spark or other implementations yet. 
Those would reflect the performance of Flink ALS implementation, as there is 
not much difference between the implicit and explicit implementations.

Instead, we compared the implicit case with the explicit case in the Flink 
implementation on the same datasets, to make sure the implicit case does not 
decrease the performance significantly. (Of course, we expected the implicit 
case to be slower due to the extra precomputation and broadcasting of `Xt * X`.)

```
expl  impl
100 8885   9196
10007879  11282
1   8839   9220
10  7102  10998
100 7543  10680
```

The numbers in the left column indicate the size of the training set (I'm 
not sure about the measure, but @jfeher can tell about it). The numbers are the 
training time in milliseconds in the explicit and implicit case respectively. 
We did the measurements on a small cluster of 3 nodes.

It seems, there is a large constant overhead, but it's not significantly 
slower in the implicit case.
We could do further, more thorough measurements if needed, but maybe that 
would be another issue. Benchmarking more and optimizing both the original ALS 
algorithm and the specific `Xt * X` computation in the implicit case could be a 
separate PR.

What are your thoughts on this?


> Extend ALS to handle implicit feedback datasets
> ---
>
> Key: FLINK-4613
> URL: https://issues.apache.org/jira/browse/FLINK-4613
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Gábor Hermann
>Assignee: Gábor Hermann
>
> The Alternating Least Squares implementation should be extended to handle 
> _implicit feedback_ datasets. These datasets do not contain explicit ratings 
> by users, they are rather built by collecting user behavior (e.g. user 
> listened to artist X for Y minutes), and they require a slightly different 
> optimization objective. See details by [Hu et 
> al|http://dx.doi.org/10.1109/ICDM.2008.22].
> We do not need to modify much in the original ALS algorithm. See [Spark ALS 
> implementation|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala],
>  which could be a basis for this extension. Only the updating factor part is 
> modified, and most of the changes are in the local parts of the algorithm 
> (i.e. UDFs). In fact, the only modification that is not local, is 
> precomputing a matrix product Y^T * Y and broadcasting it to all the nodes, 
> which we can do with broadcast DataSets. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2542: [FLINK-4613] Extend ALS to handle implicit feedback datas...

[gaborhermann]

Github user gaborhermann commented on the issue:

https://github.com/apache/flink/pull/2542
  
We did not measure performance against Spark or other implementations yet. 
Those would reflect the performance of Flink ALS implementation, as there is 
not much difference between the implicit and explicit implementations.

Instead, we compared the implicit case with the explicit case in the Flink 
implementation on the same datasets, to make sure the implicit case does not 
decrease the performance significantly. (Of course, we expected the implicit 
case to be slower due to the extra precomputation and broadcasting of `Xt * X`.)

```
expl  impl
100 8885   9196
10007879  11282
1   8839   9220
10  7102  10998
100 7543  10680
```

The numbers in the left column indicate the size of the training set (I'm 
not sure about the measure, but @jfeher can tell about it). The numbers are the 
training time in milliseconds in the explicit and implicit case respectively. 
We did the measurements on a small cluster of 3 nodes.

It seems, there is a large constant overhead, but it's not significantly 
slower in the implicit case.
We could do further, more thorough measurements if needed, but maybe that 
would be another issue. Benchmarking more and optimizing both the original ALS 
algorithm and the specific `Xt * X` computation in the implicit case could be a 
separate PR.

What are your thoughts on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2502: [FLINK-4550] [table] Clearly define SQL operator table

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/2502
  
Thanks @fhueske. Merging...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4590) Some Table API tests are failing when debug lvl is set to DEBUG

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516635#comment-15516635
 ] 

ASF GitHub Bot commented on FLINK-4590:
---

Github user twalthr commented on a diff in the pull request:

https://github.com/apache/flink/pull/2504#discussion_r80258489
  
--- Diff: 
flink-libraries/flink-table/src/test/resources/log4j-test.properties ---
@@ -0,0 +1,27 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+# Set root logger level to OFF to not flood build logs
+# set manually to INFO for debugging purposes
+log4j.rootLogger=OFF, testlogger
--- End diff --

As I explained in my PR description: "This PR fixes the estimated costs for 
minus and union DataSet RelNodes". This was the root cause for the failing 
Calcite assumptions. I just added the log4j file to easily enable assertions in 
future.


> Some Table API tests are failing when debug lvl is set to DEBUG
> ---
>
> Key: FLINK-4590
> URL: https://issues.apache.org/jira/browse/FLINK-4590
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Timo Walther
>
> For debugging another issue, I've set the log level on travis to DEBUG.
> After that, the Table API tests started failing
> {code}
> Failed tests: 
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
> {code}
> Probably Calcite is executing additional assertions depending on the debug 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2504: [FLINK-4590] [table] Some Table API tests are fail...

[twalthr]

Github user twalthr commented on a diff in the pull request:

https://github.com/apache/flink/pull/2504#discussion_r80258489
  
--- Diff: 
flink-libraries/flink-table/src/test/resources/log4j-test.properties ---
@@ -0,0 +1,27 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+# Set root logger level to OFF to not flood build logs
+# set manually to INFO for debugging purposes
+log4j.rootLogger=OFF, testlogger
--- End diff --

As I explained in my PR description: "This PR fixes the estimated costs for 
minus and union DataSet RelNodes". This was the root cause for the failing 
Calcite assumptions. I just added the log4j file to easily enable assertions in 
future.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4241) Cryptic expression parser exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516619#comment-15516619
 ] 

ASF GitHub Bot commented on FLINK-4241:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2529
  
+1 to merge


> Cryptic expression parser exceptions
> 
>
> Key: FLINK-4241
> URL: https://issues.apache.org/jira/browse/FLINK-4241
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Timo Walther
>
> The exceptions thrown when giving wrong SQL syntax to Flink's SQL parser is 
> very cryptic and should be improved. For example, the following code snippet:
> {code}
> inputTable.filter("a == 0");
> {code}
> gives the following exception:
> {code}
> Exception in thread "main" 
> org.apache.flink.api.table.ExpressionParserException: Could not parse 
> expression: [1.4] failure: `-' expected but `=' found
> a == 0
>^
>   at 
> org.apache.flink.api.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:355)
>   at org.apache.flink.api.table.Table.filter(table.scala:161)
>   at 
> com.dataartisans.streaming.SimpleTableAPIJob.main(SimpleTableAPIJob.java:32)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> {code}
> From this description it is very hard to understand that {{==}} is not a 
> valid operator.
> Another example is:
> {code}
> inputTable.select("*");
> {code}
> which gives
> {code}
> Exception in thread "main" 
> org.apache.flink.api.table.ExpressionParserException: Could not parse 
> expression: Base Failure
>   at 
> org.apache.flink.api.table.expressions.ExpressionParser$.parseExpressionList(ExpressionParser.scala:342)
>   at org.apache.flink.api.table.Table.select(table.scala:103)
>   at 
> com.dataartisans.streaming.SimpleTableAPIJob.main(SimpleTableAPIJob.java:33)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> {code}
> I think it would considerably improve user experience if we print more 
> helpful parsing exceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2529: [FLINK-4241] [table] Cryptic expression parser exceptions

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2529
  
+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4252) Table program cannot be compiled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516590#comment-15516590
 ] 

ASF GitHub Bot commented on FLINK-4252:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2507
  
+1 to merge


> Table program cannot be compiled
> 
>
> Key: FLINK-4252
> URL: https://issues.apache.org/jira/browse/FLINK-4252
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
> Environment: OS X EI Captain
> scala 2.11.7
> jdk 8
>Reporter: Renkai Ge
>Assignee: Timo Walther
> Attachments: TestMain.scala
>
>
> I'm trying the table apis.
> I got some errors like this
> My code is in the attachments
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The program 
> execution failed: Job execution failed.
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:413)
>   at 
> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:92)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:389)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:376)
>   at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:61)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at org.apache.flink.api.java.DataSet.collect(DataSet.java:410)
>   at org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
>   at org.apache.flink.api.scala.DataSet.print(DataSet.scala:1672)
>   at TestMain$.main(TestMain.scala:31)
>   at TestMain.main(TestMain.scala)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331)
>   at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
>   at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:853)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:799)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:799)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.Exception: The user defined 'open(Configuration)' method 
> in class org.apache.flink.api.table.runtime.FlatMapRunner caused an 
> exception: Table program cannot be compiled. This is a bug. Please file an 
> issue.
>   at 
> org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1337)
>   at 
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.openTask(ChainedFlatMapDriver.java:47)
>   at 
> org.apache.flink.runtime.operators.BatchTask.openChainedTasks(BatchTask.java:1377)
>   at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:471)
>   at 
> 

[GitHub] flink issue #2507: [FLINK-4252] [table] Validate input and output classes of...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/2507
  
+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4590) Some Table API tests are failing when debug lvl is set to DEBUG

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516579#comment-15516579
 ] 

ASF GitHub Bot commented on FLINK-4590:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2504#discussion_r80252345
  
--- Diff: 
flink-libraries/flink-table/src/test/resources/log4j-test.properties ---
@@ -0,0 +1,27 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+# Set root logger level to OFF to not flood build logs
+# set manually to INFO for debugging purposes
+log4j.rootLogger=OFF, testlogger
--- End diff --

I assume this is the actual fix for the referenced bug, right?


> Some Table API tests are failing when debug lvl is set to DEBUG
> ---
>
> Key: FLINK-4590
> URL: https://issues.apache.org/jira/browse/FLINK-4590
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Robert Metzger
>Assignee: Timo Walther
>
> For debugging another issue, I've set the log level on travis to DEBUG.
> After that, the Table API tests started failing
> {code}
> Failed tests: 
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusAll:156 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinusDifferentFieldNames:204 Internal error: Error 
> occurred while applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
>   SetOperatorsITCase.testMinus:175 Internal error: Error occurred while 
> applying rule DataSetScanRule
> {code}
> Probably Calcite is executing additional assertions depending on the debug 
> level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2504: [FLINK-4590] [table] Some Table API tests are fail...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2504#discussion_r80252345
  
--- Diff: 
flink-libraries/flink-table/src/test/resources/log4j-test.properties ---
@@ -0,0 +1,27 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+
+# Set root logger level to OFF to not flood build logs
+# set manually to INFO for debugging purposes
+log4j.rootLogger=OFF, testlogger
--- End diff --

I assume this is the actual fix for the referenced bug, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4613) Extend ALS to handle implicit feedback datasets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516571#comment-15516571
 ] 

ASF GitHub Bot commented on FLINK-4613:
---

Github user gaborhermann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2542#discussion_r80253321
  
--- Diff: docs/dev/libs/ml/als.md ---
@@ -49,6 +49,18 @@ By applying this step alternately to the matrices $U$ 
and $V$, we can iterativel
 
 The matrix $R$ is given in its sparse representation as a tuple of $(i, j, 
r)$ where $i$ denotes the row index, $j$ the column index and $r$ is the matrix 
value at position $(i,j)$.
 
+An alternative model can be used for _implicit feedback_ datasets.
+These datasets only contain implicit feedback from the user
+in contrast to datasets with explicit feedback like movie ratings.
+For example users watch videos on a website and the website monitors which 
user
+viewed which video, so the users only provide their preference implicitly.
+In these cases the feedback should not be treated as a
+rating, but rather an evidence that the user prefers that item.
+Thus, for implicit feedback datasets there is a slightly different
+minimalization problem to solve (see [Hu et 
al.](http://dx.doi.org/10.1109/ICDM.2008.22) for details).
+Flink supports both explicit and implicit ALS,
+and the choice between the two can be set in the parameters.
+
--- End diff --

Okay, I added
"The implementation is based on the Apache Spark implementation of implicit 
ALS."
and referred to the relevant file in the Spark codebase.



> Extend ALS to handle implicit feedback datasets
> ---
>
> Key: FLINK-4613
> URL: https://issues.apache.org/jira/browse/FLINK-4613
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Gábor Hermann
>Assignee: Gábor Hermann
>
> The Alternating Least Squares implementation should be extended to handle 
> _implicit feedback_ datasets. These datasets do not contain explicit ratings 
> by users, they are rather built by collecting user behavior (e.g. user 
> listened to artist X for Y minutes), and they require a slightly different 
> optimization objective. See details by [Hu et 
> al|http://dx.doi.org/10.1109/ICDM.2008.22].
> We do not need to modify much in the original ALS algorithm. See [Spark ALS 
> implementation|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala],
>  which could be a basis for this extension. Only the updating factor part is 
> modified, and most of the changes are in the local parts of the algorithm 
> (i.e. UDFs). In fact, the only modification that is not local, is 
> precomputing a matrix product Y^T * Y and broadcasting it to all the nodes, 
> which we can do with broadcast DataSets. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2542: [FLINK-4613] Extend ALS to handle implicit feedbac...

[gaborhermann]

Github user gaborhermann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2542#discussion_r80253321
  
--- Diff: docs/dev/libs/ml/als.md ---
@@ -49,6 +49,18 @@ By applying this step alternately to the matrices $U$ 
and $V$, we can iterativel
 
 The matrix $R$ is given in its sparse representation as a tuple of $(i, j, 
r)$ where $i$ denotes the row index, $j$ the column index and $r$ is the matrix 
value at position $(i,j)$.
 
+An alternative model can be used for _implicit feedback_ datasets.
+These datasets only contain implicit feedback from the user
+in contrast to datasets with explicit feedback like movie ratings.
+For example users watch videos on a website and the website monitors which 
user
+viewed which video, so the users only provide their preference implicitly.
+In these cases the feedback should not be treated as a
+rating, but rather an evidence that the user prefers that item.
+Thus, for implicit feedback datasets there is a slightly different
+minimalization problem to solve (see [Hu et 
al.](http://dx.doi.org/10.1109/ICDM.2008.22) for details).
+Flink supports both explicit and implicit ALS,
+and the choice between the two can be set in the parameters.
+
--- End diff --

Okay, I added
"The implementation is based on the Apache Spark implementation of implicit 
ALS."
and referred to the relevant file in the Spark codebase.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4544) TaskManager metrics are vulnerable to custom JMX bean installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516540#comment-15516540
 ] 

ASF GitHub Bot commented on FLINK-4544:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2445
  
@StephanEwen I moved the code into a separate class.


> TaskManager metrics are vulnerable to custom JMX bean installation
> --
>
> Key: FLINK-4544
> URL: https://issues.apache.org/jira/browse/FLINK-4544
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.2
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.1.3
>
>
> The TaskManager's CPU load magic may fail when JMX providers are overwritten.
> The TaskManager logic checks if the class 
> {{com.sun.management.OperatingSystemMXBean}} is available. If yes, it assumes 
> that the {{ManagementFactory.getOperatingSystemMXBean()}} is of that type. 
> That is not necessarily the case.
> This is visible in the Cassandra tests, as Cassandra overrides the JMX 
> provider - every heartbeat causes an exception that is logged (See below), 
> flooding the log, killing the heartbeat message.
> I would also suggest to move the entire metrics code out of the 
> {{TaskManager}} class into a dedicated class {{TaskManagerJvmMetrics}}. That 
> one can, with a static method, install the metrics into the TaskManager's 
> metric group.
> Sample stack trace when default platform beans are overridden:
> {code}
> 23914 [flink-akka.actor.default-dispatcher-3] WARN  
> org.apache.flink.runtime.taskmanager.TaskManager  - Error retrieving CPU Load 
> through OperatingSystemMXBean
> java.lang.IllegalArgumentException: object is not an instance of declaring 
> class
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
>   at scala.Option.map(Option.scala:145)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2351)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2348)
>   at 
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:32)
>   at 
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:20)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
>   at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2444)
>   at 
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
>   at 
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
>   at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:186)
>   at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
>   at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3631)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3022)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.sendHeartbeatToJobManager(TaskManager.scala:1278)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:309)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.testingUtils.TestingTaskManagerLike$$anonfun$handleTestingMessage$1.applyOrElse(TestingTaskManagerLike.scala:65)
>   at 

[GitHub] flink issue #2445: [FLINK-4544] Refactor old CPU metric initialization

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2445
  
@StephanEwen I moved the code into a separate class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4632) when yarn nodemanager lost, flink hung

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLINK-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516530#comment-15516530
 ] 

刘喆 commented on FLINK-4632:
---

It is (2).
I make a new idle hadoop yarn cluster, and use the source from github September 
22,  and when the application running well, so it's order must be :   kill jvm( 
yarn peemption or unhealthy),  cancel application.
But when I run it on busy yarn cluster, the problem comes up again.

> when yarn nodemanager lost,  flink hung
> ---
>
> Key: FLINK-4632
> URL: https://issues.apache.org/jira/browse/FLINK-4632
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, Streaming
>Affects Versions: 1.2.0, 1.1.2
> Environment: cdh5.5.1  jdk1.7 flink1.1.2  1.2-snapshot   kafka0.8.2
>Reporter: 刘喆
>
> When run flink streaming on yarn,  using kafka as source,  it runs well when 
> start. But after long run, for example  8 hours, dealing 60,000,000+ 
> messages, it hung: no messages consumed,   one taskmanager is CANCELING, the 
> exception show:
> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: 
> connection timeout
>   at 
> org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.exceptionCaught(PartitionRequestClientHandler.java:152)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
>   at 
> io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:835)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: 连接超时
>   at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>   at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>   at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>   at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>   at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>   at 
> io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
>   at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
>   at 
> io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
>   at 
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
>   ... 6 more
> after applyhttps://issues.apache.org/jira/browse/FLINK-4625   
> it show:
> java.lang.Exception: TaskManager was lost/killed: 
> ResourceID{resourceId='container_1471620986643_744852_01_001400'} @ 
> 38.slave.adh (dataPort=45349)
>   at 
> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:162)
>   at 
> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533)
>   at 
> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:138)
>   at 
> 

[GitHub] flink issue #2542: [FLINK-4613] Extend ALS to handle implicit feedback datas...

[mbalassi]

Github user mbalassi commented on the issue:

https://github.com/apache/flink/pull/2542
  
@gaborhermann @jfeher Could you share results where you benchmarked the 
algorithm against other implementations, please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4613) Extend ALS to handle implicit feedback datasets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516517#comment-15516517
 ] 

ASF GitHub Bot commented on FLINK-4613:
---

Github user mbalassi commented on the issue:

https://github.com/apache/flink/pull/2542
  
@gaborhermann @jfeher Could you share results where you benchmarked the 
algorithm against other implementations, please?


> Extend ALS to handle implicit feedback datasets
> ---
>
> Key: FLINK-4613
> URL: https://issues.apache.org/jira/browse/FLINK-4613
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Gábor Hermann
>Assignee: Gábor Hermann
>
> The Alternating Least Squares implementation should be extended to handle 
> _implicit feedback_ datasets. These datasets do not contain explicit ratings 
> by users, they are rather built by collecting user behavior (e.g. user 
> listened to artist X for Y minutes), and they require a slightly different 
> optimization objective. See details by [Hu et 
> al|http://dx.doi.org/10.1109/ICDM.2008.22].
> We do not need to modify much in the original ALS algorithm. See [Spark ALS 
> implementation|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala],
>  which could be a basis for this extension. Only the updating factor part is 
> modified, and most of the changes are in the local parts of the algorithm 
> (i.e. UDFs). In fact, the only modification that is not local, is 
> precomputing a matrix product Y^T * Y and broadcasting it to all the nodes, 
> which we can do with broadcast DataSets. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4613) Extend ALS to handle implicit feedback datasets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516515#comment-15516515
 ] 

ASF GitHub Bot commented on FLINK-4613:
---

Github user mbalassi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2542#discussion_r80248596
  
--- Diff: docs/dev/libs/ml/als.md ---
@@ -49,6 +49,18 @@ By applying this step alternately to the matrices $U$ 
and $V$, we can iterativel
 
 The matrix $R$ is given in its sparse representation as a tuple of $(i, j, 
r)$ where $i$ denotes the row index, $j$ the column index and $r$ is the matrix 
value at position $(i,j)$.
 
+An alternative model can be used for _implicit feedback_ datasets.
+These datasets only contain implicit feedback from the user
+in contrast to datasets with explicit feedback like movie ratings.
+For example users watch videos on a website and the website monitors which 
user
+viewed which video, so the users only provide their preference implicitly.
+In these cases the feedback should not be treated as a
+rating, but rather an evidence that the user prefers that item.
+Thus, for implicit feedback datasets there is a slightly different
+minimalization problem to solve (see [Hu et 
al.](http://dx.doi.org/10.1109/ICDM.2008.22) for details).
+Flink supports both explicit and implicit ALS,
+and the choice between the two can be set in the parameters.
+
--- End diff --

Let us mention, that the implementation was motivated by the Spark implicit 
ALS implementation. 


> Extend ALS to handle implicit feedback datasets
> ---
>
> Key: FLINK-4613
> URL: https://issues.apache.org/jira/browse/FLINK-4613
> Project: Flink
>  Issue Type: New Feature
>  Components: Machine Learning Library
>Reporter: Gábor Hermann
>Assignee: Gábor Hermann
>
> The Alternating Least Squares implementation should be extended to handle 
> _implicit feedback_ datasets. These datasets do not contain explicit ratings 
> by users, they are rather built by collecting user behavior (e.g. user 
> listened to artist X for Y minutes), and they require a slightly different 
> optimization objective. See details by [Hu et 
> al|http://dx.doi.org/10.1109/ICDM.2008.22].
> We do not need to modify much in the original ALS algorithm. See [Spark ALS 
> implementation|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala],
>  which could be a basis for this extension. Only the updating factor part is 
> modified, and most of the changes are in the local parts of the algorithm 
> (i.e. UDFs). In fact, the only modification that is not local, is 
> precomputing a matrix product Y^T * Y and broadcasting it to all the nodes, 
> which we can do with broadcast DataSets. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2542: [FLINK-4613] Extend ALS to handle implicit feedbac...

[mbalassi]

Github user mbalassi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2542#discussion_r80248596
  
--- Diff: docs/dev/libs/ml/als.md ---
@@ -49,6 +49,18 @@ By applying this step alternately to the matrices $U$ 
and $V$, we can iterativel
 
 The matrix $R$ is given in its sparse representation as a tuple of $(i, j, 
r)$ where $i$ denotes the row index, $j$ the column index and $r$ is the matrix 
value at position $(i,j)$.
 
+An alternative model can be used for _implicit feedback_ datasets.
+These datasets only contain implicit feedback from the user
+in contrast to datasets with explicit feedback like movie ratings.
+For example users watch videos on a website and the website monitors which 
user
+viewed which video, so the users only provide their preference implicitly.
+In these cases the feedback should not be treated as a
+rating, but rather an evidence that the user prefers that item.
+Thus, for implicit feedback datasets there is a slightly different
+minimalization problem to solve (see [Hu et 
al.](http://dx.doi.org/10.1109/ICDM.2008.22) for details).
+Flink supports both explicit and implicit ALS,
+and the choice between the two can be set in the parameters.
+
--- End diff --

Let us mention, that the implementation was motivated by the Spark implicit 
ALS implementation. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4218) Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." causes task restarting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516443#comment-15516443
 ] 

ASF GitHub Bot commented on FLINK-4218:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2544

[FLINK-4218] [checkpoints] Do not rely on FileSystem to determine state 
sizes

This prevents failures on eventually consistent S3, where the operations 
for keys (=entries in the parent directory/bucket) are not guaranteed to be 
immediately consistent (visible) after a blob was written.

Not relying on any operation on keys (= requesting `FileStatus`) should 
mitigate the problem.

This also changes the exception signature from `getStateSize()` from 
`Exception` to `IOException`, which fits more natural with the exception 
signatures of some other I/O methods.

Related issue: We may still want to have retries on `FileStatus` operations 
on S3, for other parts of the system (like FileOutputFormats)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink state_size_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2544.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2544


commit 7ce2de7e14b9a1fb24c27dc674f689f08abdf7cc
Author: Stephan Ewen 
Date:   2016-09-23T13:16:27Z

[FLINK-4218] [checkpoints] Do not rely on FileSystem to determing state 
sizes

This prevents failures on eventually consistent S3, where the operations for
keys (=entries in the parent directory/bucket) are not guaranteed to be 
immediately
 consistent (visible) after a blob was written.




> Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." 
> causes task restarting
> --
>
> Key: FLINK-4218
> URL: https://issues.apache.org/jira/browse/FLINK-4218
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sergii Koshel
>
> Sporadically see exception as below. And restart of task because of it.
> {code:title=Exception|borderStyle=solid}
> java.lang.RuntimeException: Error triggering a checkpoint as the result of 
> receiving checkpoint barrier
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:785)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:775)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: No such file or directory: 
> s3:///flink/checkpoints/ece317c26960464ba5de75f3bbc38cb2/chk-8810/96eebbeb-de14-45c7-8ebb-e7cde978d6d3
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
>   at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
>   at 
> org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
>   at 
> org.apache.flink.runtime.state.filesystem.FileStreamStateHandle.getStateSize(FileStreamStateHandle.java:58)
>   at 
> org.apache.flink.runtime.state.AbstractStateBackend$DataInputViewHandle.getStateSize(AbstractStateBackend.java:482)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:77)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:604)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:779)
>   ... 8 more
> {code}
> File actually exists on S3. 
> I suppose it is related to some race conditions with S3 but would be good to 
> retry a few times before stop task execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2544: [FLINK-4218] [checkpoints] Do not rely on FileSyst...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2544

[FLINK-4218] [checkpoints] Do not rely on FileSystem to determine state 
sizes

This prevents failures on eventually consistent S3, where the operations 
for keys (=entries in the parent directory/bucket) are not guaranteed to be 
immediately consistent (visible) after a blob was written.

Not relying on any operation on keys (= requesting `FileStatus`) should 
mitigate the problem.

This also changes the exception signature from `getStateSize()` from 
`Exception` to `IOException`, which fits more natural with the exception 
signatures of some other I/O methods.

Related issue: We may still want to have retries on `FileStatus` operations 
on S3, for other parts of the system (like FileOutputFormats)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink state_size_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2544.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2544


commit 7ce2de7e14b9a1fb24c27dc674f689f08abdf7cc
Author: Stephan Ewen 
Date:   2016-09-23T13:16:27Z

[FLINK-4218] [checkpoints] Do not rely on FileSystem to determing state 
sizes

This prevents failures on eventually consistent S3, where the operations for
keys (=entries in the parent directory/bucket) are not guaranteed to be 
immediately
 consistent (visible) after a blob was written.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-4494) Expose the TimeServiceProvider from the Task to each Operator.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-4494.
-
Resolution: Fixed

> Expose the TimeServiceProvider from the Task to each Operator.
> --
>
> Key: FLINK-4494
> URL: https://issues.apache.org/jira/browse/FLINK-4494
> Project: Flink
>  Issue Type: Bug
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>
> This change aims at simplifying the {{StreamTask}} class by exposing directly 
> the {{TimeServiceProvider}} to the operators being executed. 
> This implies removing the {{registerTimer()}} and 
> {{getCurrentProcessingTime()}} methods from the {{StreamTask}}. Now, to 
> register a timer and query the time, each operator will be able to get the 
> {{TimeServiceProvider}} and call the corresponding methods directly on it.
> In addition, this will simplify many of the tests which now implement their 
> own time providers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.

[Kostas Kloudas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostas Kloudas closed FLINK-4496.
-
Resolution: Fixed

> Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.
> 
>
> Key: FLINK-4496
> URL: https://issues.apache.org/jira/browse/FLINK-4496
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516456#comment-15516456
 ] 

ASF GitHub Bot commented on FLINK-4496:
---

Github user kl0u closed the pull request at:

https://github.com/apache/flink/pull/2434


> Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.
> 
>
> Key: FLINK-4496
> URL: https://issues.apache.org/jira/browse/FLINK-4496
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2434: [FLINK-4496] Refactor the TimeServiceProvider to t...

[kl0u]

Github user kl0u closed the pull request at:

https://github.com/apache/flink/pull/2434


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4494) Expose the TimeServiceProvider from the Task to each Operator.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516455#comment-15516455
 ] 

ASF GitHub Bot commented on FLINK-4494:
---

Github user kl0u closed the pull request at:

https://github.com/apache/flink/pull/2532


> Expose the TimeServiceProvider from the Task to each Operator.
> --
>
> Key: FLINK-4494
> URL: https://issues.apache.org/jira/browse/FLINK-4494
> Project: Flink
>  Issue Type: Bug
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>
> This change aims at simplifying the {{StreamTask}} class by exposing directly 
> the {{TimeServiceProvider}} to the operators being executed. 
> This implies removing the {{registerTimer()}} and 
> {{getCurrentProcessingTime()}} methods from the {{StreamTask}}. Now, to 
> register a timer and query the time, each operator will be able to get the 
> {{TimeServiceProvider}} and call the corresponding methods directly on it.
> In addition, this will simplify many of the tests which now implement their 
> own time providers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4664) Add translator to NullValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516451#comment-15516451
 ] 

ASF GitHub Bot commented on FLINK-4664:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2536
  
I added a second commit to move the translators into their own subpackage, 
as well as additional tests.


> Add translator to NullValue
> ---
>
> Key: FLINK-4664
> URL: https://issues.apache.org/jira/browse/FLINK-4664
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.2.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.2.0
>
>
> Existing translators convert from LongValue (the output label type of graph 
> generators) to IntValue, StringValue, and an offset LongValue. Translators 
> can also be used to convert vertex or edge values. This translator will be 
> appropriate for translating these vertex or edge values to NullValue when the 
> values are not used in an algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2532: [FLINK-4494] Expose the TimeServiceProvider from t...

[kl0u]

Github user kl0u closed the pull request at:

https://github.com/apache/flink/pull/2532


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516425#comment-15516425
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80238976
  
--- Diff: 
flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/MetricGroup.java
 ---
@@ -189,4 +189,15 @@
 * @return fully qualified metric name
 */
String getMetricIdentifier(String metricName, CharacterFilter filter);
+
+   /**
+* Returns the fully qualified metric name for reporter with certain 
number position in group, for example
+* {@code "host-7.taskmanager-2.window_word_count.my-mapper.metricName"}
+*
+* @param metricName metric name
+* @param filter character filter which is applied to the scope 
components if not null.
+* @param indexReporter position number of reporter in group reporters
+* @return fully qualified metric name
+*/
+   String getMetricIdentifier(String metricName, CharacterFilter 
filter,int indexReporter);
--- End diff --

missing space after second comma.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516418#comment-15516418
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80239515
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -99,10 +99,17 @@ public MetricRegistry(Configuration config) {
DelegatingConfiguration reporterConfig = new 
DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + 
namedReporter + ".");
final String className = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);
if (className == null) {
-   LOG.error("No reporter class set for 
reporter " + namedReporter + ". Metrics might not be exposed/reported.");
+   LOG.error("No reporter class set for 
reporter {}. Metrics might not be exposed/reported.",namedReporter);
--- End diff --

missing space after comma.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2517: [FLINK-4564] [metrics] Delimiter should be configu...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80242008
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -99,10 +99,17 @@ public MetricRegistry(Configuration config) {
DelegatingConfiguration reporterConfig = new 
DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + 
namedReporter + ".");
final String className = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);
if (className == null) {
-   LOG.error("No reporter class set for 
reporter " + namedReporter + ". Metrics might not be exposed/reported.");
+   LOG.error("No reporter class set for 
reporter {}. Metrics might not be exposed/reported.",namedReporter);
continue;
}
 
+   String delimiterForReporter = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER,defaultDelimiter);
+   if(delimiterForReporter.length()!=1){
+   LOG.warn("Failed to parse delimiter for 
reporter {}, using global delimiter.",namedReporter);
--- End diff --

and possibly the what was parsed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516421#comment-15516421
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80239675
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -99,10 +99,17 @@ public MetricRegistry(Configuration config) {
DelegatingConfiguration reporterConfig = new 
DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + 
namedReporter + ".");
final String className = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);
if (className == null) {
-   LOG.error("No reporter class set for 
reporter " + namedReporter + ". Metrics might not be exposed/reported.");
+   LOG.error("No reporter class set for 
reporter {}. Metrics might not be exposed/reported.",namedReporter);
continue;
}
 
+   String delimiterForReporter = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER,defaultDelimiter);
+   if(delimiterForReporter.length()!=1){
+   LOG.warn("Failed to parse delimiter for 
reporter {}, using global delimiter.",namedReporter);
--- End diff --

let's include the global delimiter in the message as well.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516434#comment-15516434
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80241834
  
--- Diff: 
flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/MetricGroup.java
 ---
@@ -189,4 +189,15 @@
 * @return fully qualified metric name
 */
String getMetricIdentifier(String metricName, CharacterFilter filter);
+
+   /**
+* Returns the fully qualified metric name for reporter with certain 
number position in group, for example
+* {@code "host-7.taskmanager-2.window_word_count.my-mapper.metricName"}
+*
+* @param metricName metric name
+* @param filter character filter which is applied to the scope 
components if not null.
+* @param indexReporter position number of reporter in group reporters
+* @return fully qualified metric name
+*/
+   String getMetricIdentifier(String metricName, CharacterFilter 
filter,int indexReporter);
--- End diff --

This method should be part of `AbstractMetricGroup`. The signature of the 
`MetricRegistry#register()` may be changed if necessary.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2517: [FLINK-4564] [metrics] Delimiter should be configu...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80241061
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java
 ---
@@ -168,21 +168,40 @@ public String getMetricIdentifier(String metricName) {
 * @param filter character filter which is applied to the scope 
components if not null.
 * @return fully qualified metric name
 */
-   public String getMetricIdentifier(String metricName, CharacterFilter 
filter) {
-   if (scopeString == null) {
-   if (filter != null) {
-   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
-   } else {
-   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
-   }
+   public String getMetricIdentifier(String metricName,CharacterFilter 
filter) {
+   if (filter != null){
+   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
+   } else {
+   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
}
-
-   if (filter != null) {
+   if (filter != null){
return scopeString + registry.getDelimiter() + 
filter.filterCharacters(metricName);
} else {
return scopeString + registry.getDelimiter() + 
metricName;
}
}
+
+   /**
+* Returns the fully qualified metric name for reporter with certain 
number position in group, for example
+* {@code "host-7.taskmanager-2.window_word_count.my-mapper.metricName"}
+*
+* @param metricName metric name
+* @param filter character filter which is applied to the scope 
components if not null.
+* @param indexReporter position number of reporter in group reporters
+* @return fully qualified metric name
+*/
+   public String getMetricIdentifier(String metricName, CharacterFilter 
filter, int indexReporter) {
+   if (filter != null){
--- End diff --

missing space before ```{```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516437#comment-15516437
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80241187
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java
 ---
@@ -168,21 +168,40 @@ public String getMetricIdentifier(String metricName) {
 * @param filter character filter which is applied to the scope 
components if not null.
 * @return fully qualified metric name
 */
-   public String getMetricIdentifier(String metricName, CharacterFilter 
filter) {
-   if (scopeString == null) {
-   if (filter != null) {
-   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
-   } else {
-   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
-   }
+   public String getMetricIdentifier(String metricName,CharacterFilter 
filter) {
+   if (filter != null){
+   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
+   } else {
+   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
}
-
-   if (filter != null) {
+   if (filter != null){
return scopeString + registry.getDelimiter() + 
filter.filterCharacters(metricName);
} else {
return scopeString + registry.getDelimiter() + 
metricName;
}
}
+
+   /**
+* Returns the fully qualified metric name for reporter with certain 
number position in group, for example
+* {@code "host-7.taskmanager-2.window_word_count.my-mapper.metricName"}
+*
+* @param metricName metric name
+* @param filter character filter which is applied to the scope 
components if not null.
+* @param indexReporter position number of reporter in group reporters
--- End diff --

"index of the reporter whose delimiter should be used"


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516431#comment-15516431
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80240626
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -219,9 +235,19 @@ public ScopeFormats getScopeFormats() {
public void register(Metric metric, String metricName, MetricGroup 
group) {
try {
if (reporters != null) {
-   for (MetricReporter reporter : reporters) {
-   if (reporter != null) {
-   
reporter.notifyOfAddedMetric(metric, metricName, group);
+   for (int i= 0; i [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516436#comment-15516436
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80242008
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -99,10 +99,17 @@ public MetricRegistry(Configuration config) {
DelegatingConfiguration reporterConfig = new 
DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + 
namedReporter + ".");
final String className = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);
if (className == null) {
-   LOG.error("No reporter class set for 
reporter " + namedReporter + ". Metrics might not be exposed/reported.");
+   LOG.error("No reporter class set for 
reporter {}. Metrics might not be exposed/reported.",namedReporter);
continue;
}
 
+   String delimiterForReporter = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER,defaultDelimiter);
+   if(delimiterForReporter.length()!=1){
+   LOG.warn("Failed to parse delimiter for 
reporter {}, using global delimiter.",namedReporter);
--- End diff --

and possibly the what was parsed.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516422#comment-15516422
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80239440
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -72,14 +74,12 @@ public MetricRegistry(Configuration config) {
}
this.scopeFormats = scopeFormats;
 
-   char delim;
-   try {
-   delim = 
config.getString(ConfigConstants.METRICS_SCOPE_DELIMITER, ".").charAt(0);
-   } catch (Exception e) {
-   LOG.warn("Failed to parse delimiter, using default 
delimiter.", e);
-   delim = '.';
+   String defaultDelimiter = 
config.getString(ConfigConstants.METRICS_SCOPE_DELIMITER, ".");
+   if(defaultDelimiter.length()!=1){
--- End diff --

these changes are unnecessary, renaming ```delimiter``` to 
```globalDelimiter``` should be sufficient.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516420#comment-15516420
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80239606
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -99,10 +99,17 @@ public MetricRegistry(Configuration config) {
DelegatingConfiguration reporterConfig = new 
DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + 
namedReporter + ".");
final String className = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);
if (className == null) {
-   LOG.error("No reporter class set for 
reporter " + namedReporter + ". Metrics might not be exposed/reported.");
+   LOG.error("No reporter class set for 
reporter {}. Metrics might not be exposed/reported.",namedReporter);
continue;
}
 
+   String delimiterForReporter = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER,defaultDelimiter);
--- End diff --

missing spaces after commas, if and around operands.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516423#comment-15516423
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80240197
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -99,10 +99,17 @@ public MetricRegistry(Configuration config) {
DelegatingConfiguration reporterConfig = new 
DelegatingConfiguration(config, ConfigConstants.METRICS_REPORTER_PREFIX + 
namedReporter + ".");
final String className = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_CLASS_SUFFIX, null);
if (className == null) {
-   LOG.error("No reporter class set for 
reporter " + namedReporter + ". Metrics might not be exposed/reported.");
+   LOG.error("No reporter class set for 
reporter {}. Metrics might not be exposed/reported.",namedReporter);
continue;
}
 
+   String delimiterForReporter = 
reporterConfig.getString(ConfigConstants.METRICS_REPORTER_SCOPE_DELIMITER,defaultDelimiter);
+   if(delimiterForReporter.length()!=1){
+   LOG.warn("Failed to parse delimiter for 
reporter {}, using global delimiter.",namedReporter);
+   delimiterForReporter=".";
+   }
+   
this.delimiters.add(delimiterForReporter.charAt(0));
+
--- End diff --

Please add the configured delimiter to the info log message when a reporter 
was instantiated. (near the end of the constructor)


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4494) Expose the TimeServiceProvider from the Task to each Operator.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516453#comment-15516453
 ] 

ASF GitHub Bot commented on FLINK-4494:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/2532
  
Yes. Thanks a lot @aljoscha !



> Expose the TimeServiceProvider from the Task to each Operator.
> --
>
> Key: FLINK-4494
> URL: https://issues.apache.org/jira/browse/FLINK-4494
> Project: Flink
>  Issue Type: Bug
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>
> This change aims at simplifying the {{StreamTask}} class by exposing directly 
> the {{TimeServiceProvider}} to the operators being executed. 
> This implies removing the {{registerTimer()}} and 
> {{getCurrentProcessingTime()}} methods from the {{StreamTask}}. Now, to 
> register a timer and query the time, each operator will be able to get the 
> {{TimeServiceProvider}} and call the corresponding methods directly on it.
> In addition, this will simplify many of the tests which now implement their 
> own time providers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2532: [FLINK-4494] Expose the TimeServiceProvider from the Task...

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/2532
  
Yes. Thanks a lot @aljoscha !



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2500: [FLINK-4549] [table] Test and document implicitly ...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2500#discussion_r80238320
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/CodeGenerator.scala
 ---
@@ -657,13 +659,17 @@ class CodeGenerator(
 }
   }
 
-  override def visitCorrelVariable(correlVariable: RexCorrelVariable): 
GeneratedExpression = ???
+  override def visitCorrelVariable(correlVariable: RexCorrelVariable): 
GeneratedExpression =
+throw new CodeGenException("The 'CorrelVariable' feature is currently 
not supported.")
--- End diff --

Same here and in the following added exceptions. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4549) Test and document implicitly supported SQL functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516449#comment-15516449
 ] 

ASF GitHub Bot commented on FLINK-4549:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2500#discussion_r80238320
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/CodeGenerator.scala
 ---
@@ -657,13 +659,17 @@ class CodeGenerator(
 }
   }
 
-  override def visitCorrelVariable(correlVariable: RexCorrelVariable): 
GeneratedExpression = ???
+  override def visitCorrelVariable(correlVariable: RexCorrelVariable): 
GeneratedExpression =
+throw new CodeGenException("The 'CorrelVariable' feature is currently 
not supported.")
--- End diff --

Same here and in the following added exceptions. 


> Test and document implicitly supported SQL functions
> 
>
> Key: FLINK-4549
> URL: https://issues.apache.org/jira/browse/FLINK-4549
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Calcite supports many SQL functions by translating them into {{RexNode}} s. 
> However, SQL functions like {{NULLIF}}, {{OVERLAPS}} are neither tested nor 
> document although supported.
> These functions should be tested and added to the documentation. We could 
> adopt parts from the Calcite documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4549) Test and document implicitly supported SQL functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516450#comment-15516450
 ] 

ASF GitHub Bot commented on FLINK-4549:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2500#discussion_r80243251
  
--- Diff: docs/dev/table_api.md ---
@@ -2442,6 +3141,18 @@ object TimestampModifier extends ScalarFunction {
 
 
 
+### Limitations
+
+Not supported at all yet:
--- End diff --

"The following operations are not supported yet:"


> Test and document implicitly supported SQL functions
> 
>
> Key: FLINK-4549
> URL: https://issues.apache.org/jira/browse/FLINK-4549
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Calcite supports many SQL functions by translating them into {{RexNode}} s. 
> However, SQL functions like {{NULLIF}}, {{OVERLAPS}} are neither tested nor 
> document although supported.
> These functions should be tested and added to the documentation. We could 
> adopt parts from the Calcite documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2500: [FLINK-4549] [table] Test and document implicitly ...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2500#discussion_r80243251
  
--- Diff: docs/dev/table_api.md ---
@@ -2442,6 +3141,18 @@ object TimestampModifier extends ScalarFunction {
 
 
 
+### Limitations
+
+Not supported at all yet:
--- End diff --

"The following operations are not supported yet:"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4549) Test and document implicitly supported SQL functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516448#comment-15516448
 ] 

ASF GitHub Bot commented on FLINK-4549:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2500#discussion_r80237743
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/CodeGenerator.scala
 ---
@@ -545,7 +545,9 @@ class CodeGenerator(
 generateInputAccess(input._1, input._2, index)
   }
 
-  override def visitFieldAccess(rexFieldAccess: RexFieldAccess): 
GeneratedExpression = ???
+  override def visitFieldAccess(rexFieldAccess: RexFieldAccess): 
GeneratedExpression =
+throw new CodeGenException("The 'FieldAccess' feature is currently not 
supported.")
--- End diff --

This error message might not be very helpful. It does not tell a user what 
caused the problem and how it can be fixed. Would `"Nested field access is not 
support yet"` be better?


> Test and document implicitly supported SQL functions
> 
>
> Key: FLINK-4549
> URL: https://issues.apache.org/jira/browse/FLINK-4549
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Calcite supports many SQL functions by translating them into {{RexNode}} s. 
> However, SQL functions like {{NULLIF}}, {{OVERLAPS}} are neither tested nor 
> document although supported.
> These functions should be tested and added to the documentation. We could 
> adopt parts from the Calcite documentation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2536: [FLINK-4664] [gelly] Add translator to NullValue

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2536
  
I added a second commit to move the translators into their own subpackage, 
as well as additional tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2500: [FLINK-4549] [table] Test and document implicitly ...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/2500#discussion_r80237743
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/CodeGenerator.scala
 ---
@@ -545,7 +545,9 @@ class CodeGenerator(
 generateInputAccess(input._1, input._2, index)
   }
 
-  override def visitFieldAccess(rexFieldAccess: RexFieldAccess): 
GeneratedExpression = ???
+  override def visitFieldAccess(rexFieldAccess: RexFieldAccess): 
GeneratedExpression =
+throw new CodeGenException("The 'FieldAccess' feature is currently not 
supported.")
--- End diff --

This error message might not be very helpful. It does not tell a user what 
caused the problem and how it can be fixed. Would `"Nested field access is not 
support yet"` be better?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4218) Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." causes task restarting

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516445#comment-15516445
 ] 

ASF GitHub Bot commented on FLINK-4218:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2544
  
@StefanRRichter Maybe interesting for you to review.


> Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." 
> causes task restarting
> --
>
> Key: FLINK-4218
> URL: https://issues.apache.org/jira/browse/FLINK-4218
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sergii Koshel
>
> Sporadically see exception as below. And restart of task because of it.
> {code:title=Exception|borderStyle=solid}
> java.lang.RuntimeException: Error triggering a checkpoint as the result of 
> receiving checkpoint barrier
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:785)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:775)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: No such file or directory: 
> s3:///flink/checkpoints/ece317c26960464ba5de75f3bbc38cb2/chk-8810/96eebbeb-de14-45c7-8ebb-e7cde978d6d3
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
>   at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
>   at 
> org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
>   at 
> org.apache.flink.runtime.state.filesystem.FileStreamStateHandle.getStateSize(FileStreamStateHandle.java:58)
>   at 
> org.apache.flink.runtime.state.AbstractStateBackend$DataInputViewHandle.getStateSize(AbstractStateBackend.java:482)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:77)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:604)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:779)
>   ... 8 more
> {code}
> File actually exists on S3. 
> I suppose it is related to some race conditions with S3 but would be good to 
> retry a few times before stop task execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2544: [FLINK-4218] [checkpoints] Do not rely on FileSystem to d...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2544
  
@StefanRRichter Maybe interesting for you to review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516435#comment-15516435
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80240523
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -219,9 +235,19 @@ public ScopeFormats getScopeFormats() {
public void register(Metric metric, String metricName, MetricGroup 
group) {
try {
if (reporters != null) {
-   for (MetricReporter reporter : reporters) {
-   if (reporter != null) {
-   
reporter.notifyOfAddedMetric(metric, metricName, group);
+   for (int i= 0; i [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516419#comment-15516419
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80238884
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java
 ---
@@ -168,21 +168,40 @@ public String getMetricIdentifier(String metricName) {
 * @param filter character filter which is applied to the scope 
components if not null.
 * @return fully qualified metric name
 */
-   public String getMetricIdentifier(String metricName, CharacterFilter 
filter) {
-   if (scopeString == null) {
-   if (filter != null) {
-   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
-   } else {
-   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
-   }
+   public String getMetricIdentifier(String metricName,CharacterFilter 
filter) {
--- End diff --

these changes should be reverted.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516426#comment-15516426
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80239024
  
--- Diff: 
flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/UnregisteredMetricsGroup.java
 ---
@@ -114,4 +114,9 @@ public String getMetricIdentifier(String metricName) {
public String getMetricIdentifier(String metricName, CharacterFilter 
filter) {
return metricName;
}
+
+   @Override
+   public String getMetricIdentifier(String metricName, CharacterFilter 
filter,int indexReporter ) {
--- End diff --

missing space after second comma.


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516424#comment-15516424
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80240342
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -163,7 +170,16 @@ public void startQueryService(ActorSystem actorSystem) 
{
}
 
public char getDelimiter() {
-   return this.delimiter;
+   return this.globalDelimiter;
+   }
+
+   public char getDelimiter(int reporterIndex) {
+   try {
+   return delimiters.get(reporterIndex);
+   }catch (IndexOutOfBoundsException e){
+   LOG.warn("Delimiter for index {} not found return 
global delimiter",reporterIndex);
--- End diff --

missing space after comma. Message should be "Delimiter for index {} not 
found, returning global delimiter."


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516430#comment-15516430
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80240685
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricRegistry.java
 ---
@@ -219,9 +235,19 @@ public ScopeFormats getScopeFormats() {
public void register(Metric metric, String metricName, MetricGroup 
group) {
try {
if (reporters != null) {
-   for (MetricReporter reporter : reporters) {
-   if (reporter != null) {
-   
reporter.notifyOfAddedMetric(metric, metricName, group);
+   for (int i= 0; i [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4564) [metrics] Delimiter should be configured per reporter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516428#comment-15516428
 ] 

ASF GitHub Bot commented on FLINK-4564:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2517#discussion_r80241027
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/AbstractMetricGroup.java
 ---
@@ -168,21 +168,40 @@ public String getMetricIdentifier(String metricName) {
 * @param filter character filter which is applied to the scope 
components if not null.
 * @return fully qualified metric name
 */
-   public String getMetricIdentifier(String metricName, CharacterFilter 
filter) {
-   if (scopeString == null) {
-   if (filter != null) {
-   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
-   } else {
-   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
-   }
+   public String getMetricIdentifier(String metricName,CharacterFilter 
filter) {
+   if (filter != null){
+   scopeString = ScopeFormat.concat(filter, 
registry.getDelimiter(), scopeComponents);
+   } else {
+   scopeString = 
ScopeFormat.concat(registry.getDelimiter(), scopeComponents);
}
-
-   if (filter != null) {
+   if (filter != null){
return scopeString + registry.getDelimiter() + 
filter.filterCharacters(metricName);
} else {
return scopeString + registry.getDelimiter() + 
metricName;
}
}
+
+   /**
+* Returns the fully qualified metric name for reporter with certain 
number position in group, for example
+* {@code "host-7.taskmanager-2.window_word_count.my-mapper.metricName"}
+*
+* @param metricName metric name
+* @param filter character filter which is applied to the scope 
components if not null.
+* @param indexReporter position number of reporter in group reporters
+* @return fully qualified metric name
+*/
+   public String getMetricIdentifier(String metricName, CharacterFilter 
filter, int indexReporter) {
--- End diff --

argument should be called ```reporterIndex```


> [metrics] Delimiter should be configured per reporter
> -
>
> Key: FLINK-4564
> URL: https://issues.apache.org/jira/browse/FLINK-4564
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Anton Mushin
>
> Currently, the delimiter used or the scope string is based on a configuration 
> setting shared by all reporters. However, different reporters may have 
> different requirements in regards to the delimiter, as such we should allow 
> reporters to use a different delimiter.
> We can keep the current setting as a global setting that is used if no 
> specific setting was set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)