[jira] [Updated] (FLINK-4666) Make constants to be final in ParameterTool

[Alexander Pivovarov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Pivovarov updated FLINK-4666:
---
Summary: Make constants to be final in ParameterTool  (was: Make constants 
final in ParameterTool)

> Make constants to be final in ParameterTool
> ---
>
> Key: FLINK-4666
> URL: https://issues.apache.org/jira/browse/FLINK-4666
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> NO_VALUE_KEY and DEFAULT_UNDEFINED in ParameterTool should be final



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4666) Make constants to be final in ParameterTool

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515504#comment-15515504
 ] 

ASF GitHub Bot commented on FLINK-4666:
---

GitHub user apivovarov opened a pull request:

https://github.com/apache/flink/pull/2538

[FLINK-4666] Make constants to be final in ParameterTool

https://issues.apache.org/jira/browse/FLINK-4666

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apivovarov/flink FLINK-4666

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2538


commit 11fdcddb932c5474b506f9338592f142fdf1cee3
Author: Alexander Pivovarov 
Date:   2016-09-23T05:54:29Z

[FLINK-4666] Make constants to be final in ParameterTool




> Make constants to be final in ParameterTool
> ---
>
> Key: FLINK-4666
> URL: https://issues.apache.org/jira/browse/FLINK-4666
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> NO_VALUE_KEY and DEFAULT_UNDEFINED in ParameterTool should be final



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2538: [FLINK-4666] Make constants to be final in Paramet...

[apivovarov]

GitHub user apivovarov opened a pull request:

https://github.com/apache/flink/pull/2538

[FLINK-4666] Make constants to be final in ParameterTool

https://issues.apache.org/jira/browse/FLINK-4666

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apivovarov/flink FLINK-4666

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2538


commit 11fdcddb932c5474b506f9338592f142fdf1cee3
Author: Alexander Pivovarov 
Date:   2016-09-23T05:54:29Z

[FLINK-4666] Make constants to be final in ParameterTool




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4666) Make constants final in ParameterTool

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created FLINK-4666:
--

 Summary: Make constants final in ParameterTool
 Key: FLINK-4666
 URL: https://issues.apache.org/jira/browse/FLINK-4666
 Project: Flink
  Issue Type: Bug
  Components: Java API
Reporter: Alexander Pivovarov
Priority: Trivial


NO_VALUE_KEY and DEFAULT_UNDEFINED in ParameterTool should be final



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515452#comment-15515452
 ] 

ASF GitHub Bot commented on FLINK-3322:
---

Github user ramkrish86 commented on the issue:

https://github.com/apache/flink/pull/2495
  
@StephanEwen 
Just trying to explain what my intention was. This is particularly to 
address the sorters.
Solution
We could create memory allocators inside the batch task corresponding to 
every input. So every iteration task will have a memory allocator/creator 
corresponding to each input. It would create an array of memory segments that 
are required for the sorters corresponding to each input and when the task 
keeps looping in an iterative fashion we use the same set of memory segments 
created by this task.
When we receive a termination event only then we do release the segments 
created by this task.
This would ensure that we do create allocate memory per task and that is 
used through the life cycle of the iterative task.
We change the Merge sorters (CombiningUnilateralSortMerger and 
UnilateralSortMerger) such that we pass the MemoryAllocator to it. When the 
sorters start doing the sorting, writing and use large records (if any) we pull 
in the memory segments allocated in the memory allocator. For iterative tasks, 
where we release the segments we just need to put back the segments to the 
memory allocator instead of releasing it back to the memory manager.
When the task receives termination call only then we forcefully close the 
allocators so that all the created segments are released back to the memory 
manager.
So even if preallocation of memory is set to true I think this would work 
and we won’t be requesting new segments from the MemoryManager’s pool and 
instead use the segments that were created initially for the first iteration.
For a normal non-iterative tasks we know that the allocators are created 
for non-iterative tasks. We have a Boolean to indicate if it is an iterative 
task or not. Based on this flag, in the place where we try to release the 
segments we can decide if to release it back to the memory manager or put back 
to the memory allocator only (in case of iterative tasks).

Pls note that I was able to fix the failed test cases that @ggevay  pointed 
out. But I have not updated the PR. I can wait for your feedback and thoughts 
and then proceed with both the PRs - this and #2510 .
Points to note:
Not sure whether this aligns with Steven's future vision of memory 
management.
Impact on Streaming Iterative tasks
Will the amount of memory segments needed for this task be dynamically 
changed? If so the above mechanism cannot work.



> MemoryManager creates too much GC pressure with iterative jobs
> --
>
> Key: FLINK-3322
> URL: https://issues.apache.org/jira/browse/FLINK-3322
> Project: Flink
>  Issue Type: Bug
>  Components: Local Runtime
>Affects Versions: 1.0.0
>Reporter: Gabor Gevay
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 1.0.0
>
> Attachments: FLINK-3322.docx, FLINK-3322_reusingmemoryfordrivers.docx
>
>
> When taskmanager.memory.preallocate is false (the default), released memory 
> segments are not added to a pool, but the GC is expected to take care of 
> them. This puts too much pressure on the GC with iterative jobs, where the 
> operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the 
> JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. 
> (It will generate some lookuptables to /tmp on first run for a few minutes.) 
> (I think the slowdown might also depend somewhat on 
> taskmanager.memory.fraction, because more unused non-managed memory results 
> in rarer GCs.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2495: FLINK-3322 - Make sorters to reuse the memory pages alloc...

[ramkrish86]

Github user ramkrish86 commented on the issue:

https://github.com/apache/flink/pull/2495
  
@StephanEwen 
Just trying to explain what my intention was. This is particularly to 
address the sorters.
Solution
We could create memory allocators inside the batch task corresponding to 
every input. So every iteration task will have a memory allocator/creator 
corresponding to each input. It would create an array of memory segments that 
are required for the sorters corresponding to each input and when the task 
keeps looping in an iterative fashion we use the same set of memory segments 
created by this task.
When we receive a termination event only then we do release the segments 
created by this task.
This would ensure that we do create allocate memory per task and that is 
used through the life cycle of the iterative task.
We change the Merge sorters (CombiningUnilateralSortMerger and 
UnilateralSortMerger) such that we pass the MemoryAllocator to it. When the 
sorters start doing the sorting, writing and use large records (if any) we pull 
in the memory segments allocated in the memory allocator. For iterative tasks, 
where we release the segments we just need to put back the segments to the 
memory allocator instead of releasing it back to the memory manager.
When the task receives termination call only then we forcefully close the 
allocators so that all the created segments are released back to the memory 
manager.
So even if preallocation of memory is set to true I think this would work 
and we won’t be requesting new segments from the MemoryManager’s pool and 
instead use the segments that were created initially for the first iteration.
For a normal non-iterative tasks we know that the allocators are created 
for non-iterative tasks. We have a Boolean to indicate if it is an iterative 
task or not. Based on this flag, in the place where we try to release the 
segments we can decide if to release it back to the memory manager or put back 
to the memory allocator only (in case of iterative tasks).

Pls note that I was able to fix the failed test cases that @ggevay  pointed 
out. But I have not updated the PR. I can wait for your feedback and thoughts 
and then proceed with both the PRs - this and #2510 .
Points to note:
Not sure whether this aligns with Steven's future vision of memory 
management.
Impact on Streaming Iterative tasks
Will the amount of memory segments needed for this task be dynamically 
changed? If so the above mechanism cannot work.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4547) when call connect method in AkkaRpcService using same address and same rpc gateway class, the returned gateways are equal with respect to equals and hashCode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515323#comment-15515323
 ] 

ASF GitHub Bot commented on FLINK-4547:
---

Github user beyond1920 commented on the issue:

https://github.com/apache/flink/pull/2455
  
@tillrohrmann , I rebase the PR. Thanks.


> when call connect method in AkkaRpcService using same address and same rpc 
> gateway class, the returned gateways are equal with respect to equals and 
> hashCode
> -
>
> Key: FLINK-4547
> URL: https://issues.apache.org/jira/browse/FLINK-4547
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: zhangjing
>Assignee: zhangjing
>
> Now every time call connect method in AkkaRpcService class using same address 
> and same rpc gateway class, the return gateway object is totally different 
> with each other which equals and hashcode are not same. 
> Maybe it’s reasonable to have the same result (equals return true, and 
> hashcode is same) when using the same address and same Gateway class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2455: [FLINK-4547] [cluster management] when call connect metho...

[beyond1920]

Github user beyond1920 commented on the issue:

https://github.com/apache/flink/pull/2455
  
@tillrohrmann , I rebase the PR. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4665) Remove boxing/unboxing to parse a primitive

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515279#comment-15515279
 ] 

ASF GitHub Bot commented on FLINK-4665:
---

GitHub user apivovarov opened a pull request:

https://github.com/apache/flink/pull/2537

[FLINK-4665] Remove boxing/unboxing to parse a primitive

https://issues.apache.org/jira/browse/FLINK-4665

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apivovarov/flink FLINK-4665

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2537


commit eeefde68405d27a828d15443b5815a4e53a6b34c
Author: Alexander Pivovarov 
Date:   2016-09-23T03:27:08Z

[FLINK-4665] Remove boxing/unboxing to parse a primitive




> Remove boxing/unboxing to parse a primitive
> ---
>
> Key: FLINK-4665
> URL: https://issues.apache.org/jira/browse/FLINK-4665
> Project: Flink
>  Issue Type: Bug
>  Components: Java API
>Reporter: Alexander Pivovarov
>Priority: Trivial
>
> I found the following issues with boxing/unboxing and Integer
> 1. Current code doing boxing/unboxing to parse a primitive - It is more 
> efficient to just call the static parseXXX method.
> 2. boxing/unboxing to do type cast
> 3. new Integer instead of valueOf - Using new Integer(int) is guaranteed to 
> always result in a new object whereas Integer.valueOf(int) allows caching of 
> values to be done by the compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2537: [FLINK-4665] Remove boxing/unboxing to parse a pri...

[apivovarov]

GitHub user apivovarov opened a pull request:

https://github.com/apache/flink/pull/2537

[FLINK-4665] Remove boxing/unboxing to parse a primitive

https://issues.apache.org/jira/browse/FLINK-4665

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apivovarov/flink FLINK-4665

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2537


commit eeefde68405d27a828d15443b5815a4e53a6b34c
Author: Alexander Pivovarov 
Date:   2016-09-23T03:27:08Z

[FLINK-4665] Remove boxing/unboxing to parse a primitive




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4665) Remove boxing/unboxing to parse a primitive

[Alexander Pivovarov (JIRA)]

Alexander Pivovarov created FLINK-4665:
--

 Summary: Remove boxing/unboxing to parse a primitive
 Key: FLINK-4665
 URL: https://issues.apache.org/jira/browse/FLINK-4665
 Project: Flink
  Issue Type: Bug
  Components: Java API
Reporter: Alexander Pivovarov
Priority: Trivial


I found the following issues with boxing/unboxing and Integer

1. Current code doing boxing/unboxing to parse a primitive - It is more 
efficient to just call the static parseXXX method.

2. boxing/unboxing to do type cast

3. new Integer instead of valueOf - Using new Integer(int) is guaranteed to 
always result in a new object whereas Integer.valueOf(int) allows caching of 
values to be done by the compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4537) ResourceManager registration with JobManager

[zhangjing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangjing resolved FLINK-4537.
--
Resolution: Resolved

> ResourceManager registration with JobManager
> 
>
> Key: FLINK-4537
> URL: https://issues.apache.org/jira/browse/FLINK-4537
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: Maximilian Michels
>Assignee: zhangjing
>
> The ResourceManager keeps tracks of all JobManager's which execute Jobs. When 
> a new JobManager registered, its leadership status is checked through the 
> HighAvailabilityServices. It will then be registered at the ResourceManager 
> using the {{JobID}} provided with the initial registration message.
> ResourceManager should use JobID and LeaderSessionID(notified by 
> HighAvailabilityServices) to identify a a session to JobMaster.
> When JobManager's register at ResourceManager, it takes the following 2 input 
> parameters :
> 1. resourceManagerLeaderId:  the fencing token for the ResourceManager leader 
> which is kept by JobMaster who send the registration
> 2. JobMasterRegistration: contain address, JobID
> ResourceManager need to process the registration event based on the following 
> steps:
> 1. Check whether input resourceManagerLeaderId is as same as the current 
> leadershipSessionId of resourceManager. If not, it means that maybe two or 
> more resourceManager exists at the same time, and current resourceManager is 
> not the proper rm. so it  rejects or ignores the registration.
> 2. Check whether exists a valid JobMaster at the giving address by connecting 
> to the address. Reject the registration from invalid address.(Hidden in the 
> connect logic)
> 3. Keep JobID and JobMasterGateway mapping relationships.
> 4. Start a JobMasterLeaderListener at the given JobID to listen to the 
> leadership of the specified JobMaster.
> 5. Send registration successful ack to the jobMaster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4535) ResourceManager registration with TaskExecutor

[zhangjing (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangjing resolved FLINK-4535.
--
Resolution: Resolved

> ResourceManager registration with TaskExecutor
> --
>
> Key: FLINK-4535
> URL: https://issues.apache.org/jira/browse/FLINK-4535
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management
>Reporter: zhangjing
>Assignee: zhangjing
>
> When TaskExecutor register at ResourceManager, it takes the following 3 input 
> parameters:
> 1. resourceManagerLeaderId:  the fencing token for the ResourceManager leader 
> which is kept by taskExecutor who send the registration
> 2.  taskExecutorAddress: the address of taskExecutor
> 3. resourceID: The resource ID of the TaskExecutor that registers
> ResourceManager need to process the registration event based on the following 
> steps:
> 1. Check whether input resourceManagerLeaderId is as same as the current 
> leadershipSessionId of resourceManager. If not, it means that maybe two or 
> more resourceManager exists at the same time, and current resourceManager is 
> not the proper rm. so it  rejects or ignores the registration.
> 2. Check whether exists a valid taskExecutor at the giving address by 
> connecting to the address. Reject the registration from invalid address.
> 3. Check whether it is a duplicate registration by input resourceId, reject 
> the registration
> 4. Keep resourceID and taskExecutorGateway mapping relationships, And 
> optionally keep resourceID and container mapping relationships in yarn mode.
> 5. Create the connection between resourceManager and taskExecutor, and ensure 
> its healthy based on heartbeat rpc calls between rm and tm ?
> 6. Send registration successful ack to the taskExecutor.
> Discussion:
> Maybe we need import errorCode or several registration decline subclass to 
> distinguish the different causes of decline registration. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4263) SQL's VALUES does not work properly

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515209#comment-15515209
 ] 

Jark Wu commented on FLINK-4263:


I have no idea how to code generate the values input format. So feel free to 
assign it to you [~twalthr], or give me some tips about why code generation can 
solve serialization problem. I will keep watching this issue.

> SQL's VALUES does not work properly
> ---
>
> Key: FLINK-4263
> URL: https://issues.apache.org/jira/browse/FLINK-4263
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Executing the following SQL leads to very strange output:
> {code}
> SELECT  *
> FROM(
> VALUES
> (1, 2),
> (3, 4)
> ) AS q (col1, col2)"
> {code}
> {code}
> org.apache.flink.optimizer.CompilerException: Error translating node 'Data 
> Source "at translateToPlan(DataSetValues.scala:88) 
> (org.apache.flink.api.table.runtime.ValuesInputFormat)" : NONE [[ 
> GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties 
> [ordering=null, grouped=null, unique=null] ]]': Could not write the user code 
> wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
>   at 
> org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:170)
>   at 
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at 
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
>   at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
>   at 
> org.apache.flink.api.scala.batch.sql.SortITCase.testOrderByMultipleFieldsWithSql(SortITCase.scala:56)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> Caused by: 
> org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could 
> not write the user code wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:888)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:281)
>   ... 51 more
> Caused by: java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>   at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)

[jira] [Commented] (FLINK-4630) add netty tcp source support

[shijinkui (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515193#comment-15515193
 ] 

shijinkui commented on FLINK-4630:
--

[~StephanEwen] 
Though flink already have a SocketTextStreamFunction, netty's better 
performance and custom delimiter can be more helpful and flexible. 
SocketTextStreamFunction is read remote source like pull mode, this implements 
is receiver mode, that is accept the push data.
I've finish it except unit test.
What do you think of this issue? 

> add netty tcp source support
> 
>
> Key: FLINK-4630
> URL: https://issues.apache.org/jira/browse/FLINK-4630
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: shijinkui
>
> When source stream get start, listen a provided tcp port, receive stream data 
> from user data source.
> This netty tcp source is keepping alive and end-to-end, that is from business 
> system to flink worker directly. 
> Such source service is needed in produce indeed.
> describe the source in detail below:
> 1.source run as a netty tcp server
> 2.user provide a tcp port, if the port is in used, increace the port 
> number between 1024 to 65535. Source can parallel.
> 3.callback the provided url to report the real port to listen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4630) add netty tcp source support

[shijinkui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shijinkui updated FLINK-4630:
-
Description: 
When source stream get start, listen a provided tcp port, receive stream data 
from user data source.
This netty tcp source is keepping alive and end-to-end, that is from business 
system to flink worker directly. 

Such source service is needed in produce indeed.

describe the source in detail below:

1.  source run as a netty tcp server
2.  user provide a tcp port, if the port is in used, increace the port 
number between 1024 to 65535. Source can parallel.
3.  callback the provided url to report the real port to listen

  was:
When source stream get start, listen a provided tcp port, receive stream data 
from user data source.
This netty tcp source is keepping alive and end-to-end, that is from business 
system to flink worker directly. 

Such source service is needed in produce indeed.

describe the source in detail below:

1.  source run as a netty tcp server
2.  user provide a tcp port, if the port is in used, increace the port 
number between 1024 to 65535
3.  callback the provided url to report the real port to listen


> add netty tcp source support
> 
>
> Key: FLINK-4630
> URL: https://issues.apache.org/jira/browse/FLINK-4630
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: shijinkui
>
> When source stream get start, listen a provided tcp port, receive stream data 
> from user data source.
> This netty tcp source is keepping alive and end-to-end, that is from business 
> system to flink worker directly. 
> Such source service is needed in produce indeed.
> describe the source in detail below:
> 1.source run as a netty tcp server
> 2.user provide a tcp port, if the port is in used, increace the port 
> number between 1024 to 65535. Source can parallel.
> 3.callback the provided url to report the real port to listen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4630) add netty tcp source support

[shijinkui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shijinkui updated FLINK-4630:
-
Description: 
When source stream get start, listen a provided tcp port, receive stream data 
from user data source.
This netty tcp source is keepping alive and end-to-end, that is from business 
system to flink worker directly. 

Such source service is needed in produce indeed.

describe the source in detail below:

1.  source run as a netty tcp server
2.  user provide a tcp port, if the port is in used, increace the port 
number between 1024 to 65535
3.  callback the provided url to report the real port to listen

  was:
When source stream get start, listen a provided tcp port, receive stream data 
sent from the place where origin data generated.
This netty tcp source is end-to-end, that is from business system to flink 
worker directly. 

Such source service is needed in produce indeed.

describe the source in detail below:

1.  source run as a netty tcp server
2.  user provide a tcp port, if the port is in used, increace the port 
number between 1024 to 65535
3.  callback the provided url to report the real port to listen


> add netty tcp source support
> 
>
> Key: FLINK-4630
> URL: https://issues.apache.org/jira/browse/FLINK-4630
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: shijinkui
>
> When source stream get start, listen a provided tcp port, receive stream data 
> from user data source.
> This netty tcp source is keepping alive and end-to-end, that is from business 
> system to flink worker directly. 
> Such source service is needed in produce indeed.
> describe the source in detail below:
> 1.source run as a netty tcp server
> 2.user provide a tcp port, if the port is in used, increace the port 
> number between 1024 to 65535
> 3.callback the provided url to report the real port to listen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2055) Implement Streaming HBaseSink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515022#comment-15515022
 ] 

ASF GitHub Bot commented on FLINK-2055:
---

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2332
  
Hi @ramkrish86 , CassandraTupleWriteAheadSink extends GenericWriteAheadSink 
that deals with the checkpoint mechanism 


> Implement Streaming HBaseSink
> -
>
> Key: FLINK-2055
> URL: https://issues.apache.org/jira/browse/FLINK-2055
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming, Streaming Connectors
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Erli Ding
>
> As per : 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Write-Stream-to-HBase-td1300.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2332: [FLINK-2055] Implement Streaming HBaseSink

[delding]

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2332
  
Hi @ramkrish86 , CassandraTupleWriteAheadSink extends GenericWriteAheadSink 
that deals with the checkpoint mechanism 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT

[shijinkui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shijinkui closed FLINK-4661.

Resolution: Won't Fix

> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
> --
>
> Key: FLINK-4661
> URL: https://issues.apache.org/jira/browse/FLINK-4661
> Project: Flink
>  Issue Type: Bug
>Reporter: shijinkui
>
> [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could 
> not resolve dependencies for project 
> org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find 
> org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in 
> http://localhost:/repository/maven-public/ was cached in the local 
> repository, resolution will not be reattempted until the update interval of 
> nexus-releases has elapsed or updates are forced -> [Help 1]
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests
> I can't find where this tests jar is generated.
> By the way, recently half month, I start to use flink. There is zero time I 
> can compile the Flink project with default setting..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT

[shijinkui (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514995#comment-15514995
 ] 

shijinkui commented on FLINK-4661:
--

hi, [~StephanEwen] [~twalthr]
My local nexus proxy is something wrong.
snapshot can be download from 
https://repository.apache.org/content/groups/snapshots/org/apache/flink/



> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
> --
>
> Key: FLINK-4661
> URL: https://issues.apache.org/jira/browse/FLINK-4661
> Project: Flink
>  Issue Type: Bug
>Reporter: shijinkui
>
> [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could 
> not resolve dependencies for project 
> org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find 
> org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in 
> http://localhost:/repository/maven-public/ was cached in the local 
> repository, resolution will not be reattempted until the update interval of 
> nexus-releases has elapsed or updates are forced -> [Help 1]
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests
> I can't find where this tests jar is generated.
> By the way, recently half month, I start to use flink. There is zero time I 
> can compile the Flink project with default setting..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4664) Add translator to NullValue

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514462#comment-15514462
 ] 

ASF GitHub Bot commented on FLINK-4664:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/2536

[FLINK-4664] [gelly] Add translator to NullValue

This translator is appropriate for translating vertex and edge values to 
NullValue when the values are not used in an algorithm.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
4664_add_translator_to_nullvalue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2536


commit 5d9c7c3dc308a912a84429331afce49539ed8d94
Author: Greg Hogan 
Date:   2016-09-22T20:47:17Z

[FLINK-4664] [gelly] Add translator to NullValue

This translator is appropriate for translating vertex and edge values to
NullValue when the values are not used in an algorithm.




> Add translator to NullValue
> ---
>
> Key: FLINK-4664
> URL: https://issues.apache.org/jira/browse/FLINK-4664
> Project: Flink
>  Issue Type: New Feature
>  Components: Gelly
>Affects Versions: 1.2.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
> Fix For: 1.2.0
>
>
> Existing translators convert from LongValue (the output label type of graph 
> generators) to IntValue, StringValue, and an offset LongValue. Translators 
> can also be used to convert vertex or edge values. This translator will be 
> appropriate for translating these vertex or edge values to NullValue when the 
> values are not used in an algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2536: [FLINK-4664] [gelly] Add translator to NullValue

[greghogan]

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/2536

[FLINK-4664] [gelly] Add translator to NullValue

This translator is appropriate for translating vertex and edge values to 
NullValue when the values are not used in an algorithm.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
4664_add_translator_to_nullvalue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2536


commit 5d9c7c3dc308a912a84429331afce49539ed8d94
Author: Greg Hogan 
Date:   2016-09-22T20:47:17Z

[FLINK-4664] [gelly] Add translator to NullValue

This translator is appropriate for translating vertex and edge values to
NullValue when the values are not used in an algorithm.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4664) Add translator to NullValue

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-4664:
-

 Summary: Add translator to NullValue
 Key: FLINK-4664
 URL: https://issues.apache.org/jira/browse/FLINK-4664
 Project: Flink
  Issue Type: New Feature
  Components: Gelly
Affects Versions: 1.2.0
Reporter: Greg Hogan
Assignee: Greg Hogan
Priority: Minor
 Fix For: 1.2.0


Existing translators convert from LongValue (the output label type of graph 
generators) to IntValue, StringValue, and an offset LongValue. Translators can 
also be used to convert vertex or edge values. This translator will be 
appropriate for translating these vertex or edge values to NullValue when the 
values are not used in an algorithm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3719) WebInterface: Moving the barrier between graph and stats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514265#comment-15514265
 ] 

ASF GitHub Bot commented on FLINK-3719:
---

Github user iampeter commented on the issue:

https://github.com/apache/flink/pull/2467
  
@mushketyk thanks for this feature, very useful indeed.

I'd like to ask you to reconsider some things:
1. First of all, I think it would be good to wrap split.js in an Angular 
directive
2. I think the `style` attribute could be replaced by CSS
3. I think there is a typo in the port numer (80801)

Let me know your thoughts.

Thanks,
Piotr



> WebInterface: Moving the barrier between graph and stats
> 
>
> Key: FLINK-3719
> URL: https://issues.apache.org/jira/browse/FLINK-3719
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Niels Basjes
>Assignee: Ivan Mushketyk
>
> It would be really useful if the separator between the graphical view of a 
> job topology at the top and the textual overview of the counters at the 
> bottom can be moved up/down. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2467: [FLINK-3719][web frontend] Moving the barrier between gra...

[iampeter]

Github user iampeter commented on the issue:

https://github.com/apache/flink/pull/2467
  
@mushketyk thanks for this feature, very useful indeed.

I'd like to ask you to reconsider some things:
1. First of all, I think it would be good to wrap split.js in an Angular 
directive
2. I think the `style` attribute could be replaced by CSS
3. I think there is a typo in the port numer (80801)

Let me know your thoughts.

Thanks,
Piotr



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514143#comment-15514143
 ] 

ASF GitHub Bot commented on FLINK-4624:
---

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/2527#discussion_r80109825
  
--- Diff: 
flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/Summarization.java
 ---
@@ -226,11 +247,15 @@ public void setGroupRepresentativeId(K 
groupRepresentativeId) {
}
 
public VGV getVertexGroupValue() {
-   return f2;
+   return f2.isLeft() ? f2.left() : null;
}
 
public void setVertexGroupValue(VGV vertexGroupValue) {
-   f2 = vertexGroupValue;
+   if (vertexGroupValue == null) {
+   f2 = new 
Either.Right<>(NullValue.getInstance());
--- End diff --

Can these objects be reused?


> Gelly's summarization algorithm cannot deal with null vertex group values
> -
>
> Key: FLINK-4624
> URL: https://issues.apache.org/jira/browse/FLINK-4624
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly
>Reporter: Till Rohrmann
>Assignee: Martin Junghanns
> Fix For: 1.2.0
>
>
> Gelly's {{Summarization}} algorithm cannot handle null values in the 
> `VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
> value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
> handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2527: [FLINK-4624] Allow for null values in Graph Summar...

[greghogan]

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/2527#discussion_r80109825
  
--- Diff: 
flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/library/Summarization.java
 ---
@@ -226,11 +247,15 @@ public void setGroupRepresentativeId(K 
groupRepresentativeId) {
}
 
public VGV getVertexGroupValue() {
-   return f2;
+   return f2.isLeft() ? f2.left() : null;
}
 
public void setVertexGroupValue(VGV vertexGroupValue) {
-   f2 = vertexGroupValue;
+   if (vertexGroupValue == null) {
+   f2 = new 
Either.Right<>(NullValue.getInstance());
--- End diff --

Can these objects be reused?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514124#comment-15514124
 ] 

ASF GitHub Bot commented on FLINK-4624:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
I'm not following why specifying the `TypeInformation` is now required with 
the change to using `Either`. Is the type system failing to handle this 
properly?


> Gelly's summarization algorithm cannot deal with null vertex group values
> -
>
> Key: FLINK-4624
> URL: https://issues.apache.org/jira/browse/FLINK-4624
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly
>Reporter: Till Rohrmann
>Assignee: Martin Junghanns
> Fix For: 1.2.0
>
>
> Gelly's {{Summarization}} algorithm cannot handle null values in the 
> `VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
> value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
> handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2527: [FLINK-4624] Allow for null values in Graph Summarization

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
I'm not following why specifying the `TypeInformation` is now required with 
the change to using `Either`. Is the type system failing to handle this 
properly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4218) Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." causes task restarting

[Zhenzhong Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514093#comment-15514093
 ] 

Zhenzhong Xu commented on FLINK-4218:
-

I think simply have retries potentially with configurable retry strategy would 
solve this problem for now. If everything else fails, we can return a special 
value to indicate unknown size just as you mentioned. This will unblock this 
issue for now.

> Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." 
> causes task restarting
> --
>
> Key: FLINK-4218
> URL: https://issues.apache.org/jira/browse/FLINK-4218
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sergii Koshel
>
> Sporadically see exception as below. And restart of task because of it.
> {code:title=Exception|borderStyle=solid}
> java.lang.RuntimeException: Error triggering a checkpoint as the result of 
> receiving checkpoint barrier
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:785)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:775)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: No such file or directory: 
> s3:///flink/checkpoints/ece317c26960464ba5de75f3bbc38cb2/chk-8810/96eebbeb-de14-45c7-8ebb-e7cde978d6d3
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
>   at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
>   at 
> org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
>   at 
> org.apache.flink.runtime.state.filesystem.FileStreamStateHandle.getStateSize(FileStreamStateHandle.java:58)
>   at 
> org.apache.flink.runtime.state.AbstractStateBackend$DataInputViewHandle.getStateSize(AbstractStateBackend.java:482)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:77)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:604)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:779)
>   ... 8 more
> {code}
> File actually exists on S3. 
> I suppose it is related to some race conditions with S3 but would be good to 
> retry a few times before stop task execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4660) HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in a restarting loop

[Matt Zimmer (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514015#comment-15514015
 ] 

Matt Zimmer commented on FLINK-4660:


[~StephanEwen] We last tested with latest on master as of 9/6.  Should it have 
been there then?

In the mean time, we'll try to test with latest as of today, time allowing.

> HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in 
> a restarting loop
> ---
>
> Key: FLINK-4660
> URL: https://issues.apache.org/jira/browse/FLINK-4660
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: Zhenzhong Xu
>Priority: Critical
> Attachments: Screen Shot 2016-09-20 at 2.49.14 PM.png, Screen Shot 
> 2016-09-20 at 2.49.32 PM.png
>
>
> Flink job with checkpoints enabled and configured to use S3A file system 
> backend, sometimes experiences checkpointing failure due to S3 consistency 
> issue. This behavior is also reported by other people and documented in 
> https://issues.apache.org/jira/browse/FLINK-4218.
> This problem gets magnified by current HadoopFileSystem implementation, which 
> can potentially leak S3 client connections, and eventually get into a 
> restarting loop with “Timeout waiting for a connection from pool” exception 
> thrown from aws client.
> I looked at the code, seems HadoopFileSystem.java never invoke close() method 
> on fs object upon failure, but the FileSystem may be re-initialized every 
> time the job gets restarted.
> A few evidence I observed:
> 1. When I set the connection pool limit to 128, and below commands shows 128 
> connections are stuck in CLOSE_WAIT state.
> !Screen Shot 2016-09-20 at 2.49.14 PM.png|align=left, vspace=5! 
> 2. task manager logs indicates that state backend file system consistently 
> getting initialized upon job restarting.
> !Screen Shot 2016-09-20 at 2.49.32 PM.png!
> 3. Log indicates there is NPE during cleanning up of stream task which was 
> caused by “Timeout waiting for connection from pool” exception when trying to 
> create a directory in S3 bucket.
> 2016-09-02 08:17:50,886 ERROR 
> org.apache.flink.streaming.runtime.tasks.StreamTask - Error during cleanup of 
> stream task
> java.lang.NullPointerException
> at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.cleanup(OneInputStreamTask.java:73)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589)
> at java.lang.Thread.run(Thread.java:745)
> 4.It appears StreamTask from invoking checkpointing operation, to handling 
> failure, there is no logic associated with closing Hadoop File System object 
> (which internally includes S3 aws client object), which resides in 
> HadoopFileSystem.java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513996#comment-15513996
 ] 

ASF GitHub Bot commented on FLINK-4624:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
I'd also like to add a `ToNullValue` translator that would accept any type 
and convert to `NullValue`.


> Gelly's summarization algorithm cannot deal with null vertex group values
> -
>
> Key: FLINK-4624
> URL: https://issues.apache.org/jira/browse/FLINK-4624
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly
>Reporter: Till Rohrmann
>Assignee: Martin Junghanns
> Fix For: 1.2.0
>
>
> Gelly's {{Summarization}} algorithm cannot handle null values in the 
> `VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
> value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
> handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2527: [FLINK-4624] Allow for null values in Graph Summarization

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
I'd also like to add a `ToNullValue` translator that would accept any type 
and convert to `NullValue`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4624) Gelly's summarization algorithm cannot deal with null vertex group values

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513945#comment-15513945
 ] 

ASF GitHub Bot commented on FLINK-4624:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
Would a graph translator simplify the conversion from Long to String? You 
can do `graph.run(new TranslateEdgeValues<...>(new StringToLong())` and write a 
simple `public class StringToLong implements TranslateFunction { 
...`. Or this could be an anonymous class.


> Gelly's summarization algorithm cannot deal with null vertex group values
> -
>
> Key: FLINK-4624
> URL: https://issues.apache.org/jira/browse/FLINK-4624
> Project: Flink
>  Issue Type: Bug
>  Components: Gelly
>Reporter: Till Rohrmann
>Assignee: Martin Junghanns
> Fix For: 1.2.0
>
>
> Gelly's {{Summarization}} algorithm cannot handle null values in the 
> `VertexGroupItem.f2`. This behaviour is hidden by using Strings as a vertex 
> value in the {{SummarizationITCase}}, because the {{StringSerializer}} can 
> handle null values. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2527: [FLINK-4624] Allow for null values in Graph Summarization

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2527
  
Would a graph translator simplify the conversion from Long to String? You 
can do `graph.run(new TranslateEdgeValues<...>(new StringToLong())` and write a 
simple `public class StringToLong implements TranslateFunction { 
...`. Or this could be an anonymous class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4596) RESTART_STRATEGY is not really pluggable

[Monal Daxini (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513734#comment-15513734
 ] 

Monal Daxini commented on FLINK-4596:
-

Hi [~StephanEwen] what are your thoughts on this?

> RESTART_STRATEGY is not really pluggable
> 
>
> Key: FLINK-4596
> URL: https://issues.apache.org/jira/browse/FLINK-4596
> Project: Flink
>  Issue Type: Bug
>Reporter: Nagarjun Guraja
>
> Standalone cluster config accepts an implementation(class) as part of the 
> yaml config file but that does not work either as cluster level restart 
> strategy or streaming job level restart strategy
> CLUSTER LEVEL CAUSE: createRestartStrategyFactory converts configured value 
> of strategyname to lowercase and searches for class name using lowercased 
> string.
> JOB LEVEL CAUSE: Checkpointed streams have specific code to add 
> fixeddelayrestartconfiguration if no RestartConfiguration is specified in  
> the job env. Also, jobs cannot provide their own custom restart strategy 
> implementation and are constrained to pick up one of the three restart 
> strategies provided by flink. 
> FIX: Do not lower case the strategy config value, support a new 
> restartconfiguration to fallback to cluster level restart strategy and 
> support jobs to provide custom implementation of the strategy class itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4263) SQL's VALUES does not work properly

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513450#comment-15513450
 ] 

Timo Walther commented on FLINK-4263:
-

I think they don't need to be basic types even though it is written in the 
Javadoc. At least there is logic for {{ROW}}, {{MULTISET}}. If you don't use 
the code generation you have to make sure that you convert data types correctly 
(time, timestamp, date), which means we have duplicate code.

> SQL's VALUES does not work properly
> ---
>
> Key: FLINK-4263
> URL: https://issues.apache.org/jira/browse/FLINK-4263
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Executing the following SQL leads to very strange output:
> {code}
> SELECT  *
> FROM(
> VALUES
> (1, 2),
> (3, 4)
> ) AS q (col1, col2)"
> {code}
> {code}
> org.apache.flink.optimizer.CompilerException: Error translating node 'Data 
> Source "at translateToPlan(DataSetValues.scala:88) 
> (org.apache.flink.api.table.runtime.ValuesInputFormat)" : NONE [[ 
> GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties 
> [ordering=null, grouped=null, unique=null] ]]': Could not write the user code 
> wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
>   at 
> org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:170)
>   at 
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at 
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
>   at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
>   at 
> org.apache.flink.api.scala.batch.sql.SortITCase.testOrderByMultipleFieldsWithSql(SortITCase.scala:56)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> Caused by: 
> org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could 
> not write the user code wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:888)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:281)
>   ... 51 more
> Caused by: java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>   at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>  

[jira] [Commented] (FLINK-4662) Bump Calcite version up to 1.9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513420#comment-15513420
 ] 

ASF GitHub Bot commented on FLINK-4662:
---

GitHub user wuchong opened a pull request:

https://github.com/apache/flink/pull/2535

[FLINK-4662] Bump Calcite version up to 1.9

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

Bump Calcite version from 1.7 to 1.9 . This PR updates `FlinkPlannerImpl` 
and Interval relate codes because of API breaking.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wuchong/flink calcite-1.9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2535


commit badfbe22abafad03a07f7fd3a6d1d990b0f5c65e
Author: Jark Wu 
Date:   2016-09-22T13:47:37Z

[FLINK-4662] Bump Calcite version up to 1.9




> Bump Calcite version up to 1.9
> --
>
> Key: FLINK-4662
> URL: https://issues.apache.org/jira/browse/FLINK-4662
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Calcite just released the 1.9 version. We should adopt it also in the Table 
> API especially for FLINK-4294.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2535: [FLINK-4662] Bump Calcite version up to 1.9

[wuchong]

GitHub user wuchong opened a pull request:

https://github.com/apache/flink/pull/2535

[FLINK-4662] Bump Calcite version up to 1.9

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed

Bump Calcite version from 1.7 to 1.9 . This PR updates `FlinkPlannerImpl` 
and Interval relate codes because of API breaking.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wuchong/flink calcite-1.9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2535


commit badfbe22abafad03a07f7fd3a6d1d990b0f5c65e
Author: Jark Wu 
Date:   2016-09-22T13:47:37Z

[FLINK-4662] Bump Calcite version up to 1.9




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4263) SQL's VALUES does not work properly

[Jark Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513401#comment-15513401
 ] 

Jark Wu commented on FLINK-4263:


It seems that the data types in {{VALUES}} are always RexLiteral which should 
be basic types and could be serializable ?

> SQL's VALUES does not work properly
> ---
>
> Key: FLINK-4263
> URL: https://issues.apache.org/jira/browse/FLINK-4263
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Executing the following SQL leads to very strange output:
> {code}
> SELECT  *
> FROM(
> VALUES
> (1, 2),
> (3, 4)
> ) AS q (col1, col2)"
> {code}
> {code}
> org.apache.flink.optimizer.CompilerException: Error translating node 'Data 
> Source "at translateToPlan(DataSetValues.scala:88) 
> (org.apache.flink.api.table.runtime.ValuesInputFormat)" : NONE [[ 
> GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties 
> [ordering=null, grouped=null, unique=null] ]]': Could not write the user code 
> wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
>   at 
> org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:170)
>   at 
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at 
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
>   at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
>   at 
> org.apache.flink.api.scala.batch.sql.SortITCase.testOrderByMultipleFieldsWithSql(SortITCase.scala:56)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> Caused by: 
> org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could 
> not write the user code wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:888)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:281)
>   ... 51 more
> Caused by: java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>   at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   

[jira] [Resolved] (FLINK-4628) User class loader unavailable during input split assignment

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-4628.
-
   Resolution: Fixed
Fix Version/s: 1.2.0

Fixed via 345b2529a8acdd59d67e89ea930ec69ad69a55d3

> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.2.0
>
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the 
> executing thread.
> In 2 we only have the system class loader available. If any classes need to 
> be loaded then, we have a problem. Some InputFormats rely on code which 
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the 
> latest master.
> We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4628) User class loader unavailable during input split assignment

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-4628.
---

> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 1.2.0
>
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the 
> executing thread.
> In 2 we only have the system class loader available. If any classes need to 
> be loaded then, we have a problem. Some InputFormats rely on code which 
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the 
> latest master.
> We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen resolved FLINK-4603.
-
Resolution: Fixed

Fixed via 3b8fe95ec728d59e3ffba2901450c56d7cca2b24

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen closed FLINK-4603.
---

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4587) Yet another java.lang.NoSuchFieldError: INSTANCE

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513337#comment-15513337
 ] 

Stephan Ewen commented on FLINK-4587:
-

It depends on which maven version you are using.
Maven 3.0.x woulds directly

If you use Maven 3.3.x, you need to rebuild "flink-dist". It is a problem of 
later versions of Maven and the Maven Shade Plugin 
{code}
mvn clean package -DskipTests
cd flink-dist
mvn clean package -DskipTests
{code}

> Yet another java.lang.NoSuchFieldError: INSTANCE
> 
>
> Key: FLINK-4587
> URL: https://issues.apache.org/jira/browse/FLINK-4587
> Project: Flink
>  Issue Type: Bug
>  Components: Build System, Local Runtime
>Affects Versions: 1.2.0
> Environment: Latest SNAPSHOT
>Reporter: Renkai Ge
> Attachments: diff in mvn clean package.png, flink-explore-src.zip
>
>
> For some reason I need to use org.apache.httpcomponents:httpasyncclient:4.1.2 
> in flink.
> The source file is:
> {code}
> import org.apache.flink.streaming.api.scala._
> import org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory
> /**
>   * Created by renkai on 16/9/7.
>   */
> object Main {
>   def main(args: Array[String]): Unit = {
> val instance = ManagedNHttpClientConnectionFactory.INSTANCE
> println("instance = " + instance)
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val stream = env.fromCollection(1 to 100)
> val result = stream.map { x =>
>   x * 2
> }
> result.print()
> env.execute("xixi")
>   }
> }
> {code}
> and 
> {code}
> name := "flink-explore"
> version := "1.0"
> scalaVersion := "2.11.8"
> crossPaths := false
> libraryDependencies ++= Seq(
>   "org.apache.flink" %% "flink-scala" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
>   "org.apache.flink" %% "flink-connector-kafka-0.8" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
>   "org.apache.flink" %% "flink-streaming-scala" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
>   "org.apache.flink" %% "flink-clients" % "1.2-SNAPSHOT"
> exclude("com.google.code.findbugs", "jsr305"),
>   "org.apache.httpcomponents" % "httpasyncclient" % "4.1.2"
> )
> {code}
> I use `sbt assembly` to get a fat jar.
> If I run the command
> {code}
>  java -cp flink-explore-assembly-1.0.jar Main
> {code}
> I get the result 
> {code}
> instance = 
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory@4909b8da
> log4j:WARN No appenders could be found for logger 
> (org.apache.flink.api.scala.ClosureCleaner$).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Connected to JobManager at Actor[akka://flink/user/jobmanager_1#-1177584915]
> 09/07/2016 12:05:26   Job execution switched to status RUNNING.
> 09/07/2016 12:05:26   Source: Collection Source(1/1) switched to SCHEDULED
> 09/07/2016 12:05:26   Source: Collection Source(1/1) switched to DEPLOYING
> ...
> 09/07/2016 12:05:26   Map -> Sink: Unnamed(20/24) switched to RUNNING
> 09/07/2016 12:05:26   Map -> Sink: Unnamed(19/24) switched to RUNNING
> 15> 30
> 20> 184
> ...
> 19> 182
> 1> 194
> 8> 160
> 09/07/2016 12:05:26   Source: Collection Source(1/1) switched to FINISHED
> ...
> 09/07/2016 12:05:26   Map -> Sink: Unnamed(1/24) switched to FINISHED
> 09/07/2016 12:05:26   Job execution switched to status FINISHED.
> {code}
> Nothing special.
> But if I run the jar by
> {code}
> ./bin/flink run shop-monitor-flink-assembly-1.0.jar
> {code}
> I will get an error
> {code}
> $ ./bin/flink run flink-explore-assembly-1.0.jar
> Cluster configuration: Standalone cluster with JobManager at /127.0.0.1:6123
> Using address 127.0.0.1:6123 to connect to JobManager.
> JobManager web interface address http://127.0.0.1:8081
> Starting execution of program
> 
>  The program finished with the following exception:
> java.lang.NoSuchFieldError: INSTANCE
>   at 
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:53)
>   at 
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:57)
>   at 
> org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory.(DefaultHttpRequestWriterFactory.java:47)
>   at 
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:75)
>   at 
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:83)
>   at 
> org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory.(ManagedNHttpClientConnectionFactory.java:64)
>   at Main$.main(Main.scala:9)
> 

[GitHub] flink issue #2523: [FLINK-4556] Make Queryable State Key-Group Aware

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2523
  
please review @aljoscha or @StephanEwen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4556) Make Queryable State Key-Group Aware

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513330#comment-15513330
 ] 

ASF GitHub Bot commented on FLINK-4556:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2523
  
please review @aljoscha or @StephanEwen 


> Make Queryable State Key-Group Aware
> 
>
> Key: FLINK-4556
> URL: https://issues.apache.org/jira/browse/FLINK-4556
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.2.0
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
>Priority: Blocker
>
> The recent introduction of key-grouped state breaks queryable state because 
> the JobManager does not yet forward the client to the correct TaskManager 
> based on key-group ranges.
> This will either have to be implemented on the JobManager side, i.e. in 
> {{AkkaKvStateLocationLookupService}} or on the {{TaskManager}} when state is 
> registered. The JobManager can know the mapping because it should know the 
> {{parallelism}}/{{maxParallelism}} which it can use to determine where the 
> state for a key-group is stored. The {{TaskManager}} send a 
> {{NotifyKvStateRegistered}} message that already contains a {{keyGroupIndex}} 
> field that is not useful/correct at the moment, though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4554) Add support for array types

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther reassigned FLINK-4554:
---

Assignee: Timo Walther

> Add support for array types
> ---
>
> Key: FLINK-4554
> URL: https://issues.apache.org/jira/browse/FLINK-4554
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Support creating arrays:
> {code}ARRAY[1, 2, 3]{code}
> Access array values:
> {code}myArray[3]{code}
> And operations like:
> {{UNNEST, UNNEST WITH ORDINALITY, CARDINALITY}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4661) Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513311#comment-15513311
 ] 

Timo Walther commented on FLINK-4661:
-

Today [~fhueske] and me also have problems in compiling {{flink-table}}. Maybe 
it is just a Maven central issue. Or have we changed Maven dependencies 
recently?

> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT
> --
>
> Key: FLINK-4661
> URL: https://issues.apache.org/jira/browse/FLINK-4661
> Project: Flink
>  Issue Type: Bug
>Reporter: shijinkui
>
> [ERROR] Failed to execute goal on project flink-streaming-java_2.10: Could 
> not resolve dependencies for project 
> org.apache.flink:flink-streaming-java_2.10:jar:1.2-SNAPSHOT: Failure to find 
> org.apache.flink:flink-runtime_2.10:jar:tests:1.2-SNAPSHOT in 
> http://localhost:/repository/maven-public/ was cached in the local 
> repository, resolution will not be reattempted until the update interval of 
> nexus-releases has elapsed or updates are forced -> [Help 1]
> Failure to find org.apache.flink:flink-runtime_2.10:jar:tests
> I can't find where this tests jar is generated.
> By the way, recently half month, I start to use flink. There is zero time I 
> can compile the Flink project with default setting..



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4263) SQL's VALUES does not work properly

[Timo Walther (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513302#comment-15513302
 ] 

Timo Walther commented on FLINK-4263:
-

I also had a look at it. Replacing {{Seq[Row]}} field by {{Seq[List]}} does 
only solve the current problem. What happens if we have a row of rows or row of 
POJOs. I think maybe we should also code generate the values input format. 
Otherwise we also have to make sure that the contents of the values are always 
serializable no matter which data types may be added in future. [~jark] do you 
wanna still fix this issue? I could also assign it to me.

> SQL's VALUES does not work properly
> ---
>
> Key: FLINK-4263
> URL: https://issues.apache.org/jira/browse/FLINK-4263
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.1.0
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Executing the following SQL leads to very strange output:
> {code}
> SELECT  *
> FROM(
> VALUES
> (1, 2),
> (3, 4)
> ) AS q (col1, col2)"
> {code}
> {code}
> org.apache.flink.optimizer.CompilerException: Error translating node 'Data 
> Source "at translateToPlan(DataSetValues.scala:88) 
> (org.apache.flink.api.table.runtime.ValuesInputFormat)" : NONE [[ 
> GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties 
> [ordering=null, grouped=null, unique=null] ]]': Could not write the user code 
> wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
>   at 
> org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>   at 
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:170)
>   at 
> org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
>   at 
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
>   at 
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
>   at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
>   at 
> org.apache.flink.api.scala.batch.sql.SortITCase.testOrderByMultipleFieldsWithSql(SortITCase.scala:56)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
> Caused by: 
> org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could 
> not write the user code wrapper class 
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : 
> java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at 
> org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:888)
>   at 
> org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:281)
>   ... 51 more
> Caused by: java.io.NotSerializableException: org.apache.flink.api.table.Row
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>   at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>   at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>   at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>   at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>   at 

[jira] [Commented] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513265#comment-15513265
 ] 

ASF GitHub Bot commented on FLINK-4663:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2534
  
+1 to merge.


> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
>Priority: Minor
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2534: [FLINK-4663] Flink JDBCOutputFormat logs wrong WARN messa...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2534
  
+1 to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[Fabian Hueske (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fabian Hueske reopened FLINK-4663:
--

Hi [~the.swapni...@gmail.com], thanks for providing a fix!
We will close the issue once the pull request is merged. 
Thanks, Fabian

> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
>Priority: Minor
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[Swapnil Chougule (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Chougule resolved FLINK-4663.
-
Resolution: Fixed

Resolved. Created PR
https://github.com/apache/flink/pull/2534
Kindly review.

Regards,
Swapnil

> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
>Priority: Minor
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[Swapnil Chougule (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Chougule updated FLINK-4663:

Priority: Minor  (was: Major)

> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
>Priority: Minor
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2534: [FLINK-4663] Flink JDBCOutputFormat logs wrong WAR...

[swapnil-chougule]

GitHub user swapnil-chougule opened a pull request:

https://github.com/apache/flink/pull/2534

[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed

Fixed wrong WARN message logged by JDBCOutputFormat while adding row 
(writing record to prepared statement) as
"Column SQL types array doesn't match arity of passed Row! Check the passed 
array..."
even if there is no mismatch is SQL types array & arity of passed Row. 
[FLINK-4663]

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-4663]")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - No need to change documentation for same.

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swapnil-chougule/flink FLINK-4663

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2534


commit a3a3a76164c717f079887a641598e3b9aa6ce484
Author: swapnil-chougule 
Date:   2016-09-22T12:31:20Z

[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513241#comment-15513241
 ] 

ASF GitHub Bot commented on FLINK-4663:
---

GitHub user swapnil-chougule opened a pull request:

https://github.com/apache/flink/pull/2534

[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed

Fixed wrong WARN message logged by JDBCOutputFormat while adding row 
(writing record to prepared statement) as
"Column SQL types array doesn't match arity of passed Row! Check the passed 
array..."
even if there is no mismatch is SQL types array & arity of passed Row. 
[FLINK-4663]

- [ ] General
  - The pull request references the related JIRA issue ("[FLINK-4663]")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [ ] Documentation
  - No need to change documentation for same.

- [ ] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swapnil-chougule/flink FLINK-4663

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2534


commit a3a3a76164c717f079887a641598e3b9aa6ce484
Author: swapnil-chougule 
Date:   2016-09-22T12:31:20Z

[FLINK-4663] Flink JDBCOutputFormat logs wrong WARN message - Fixed




> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4218) Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." causes task restarting

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513234#comment-15513234
 ] 

Stephan Ewen commented on FLINK-4218:
-

[~zhenzhongxu] Adding you to this conversation.

My assumptions is that the problem is as follows: The file exists and is 
consistently visible (if I understand S3 correctly), but the parent directory's 
metadata is eventual consistent. The operation that fails here is the lookup of 
the file size, which is in most file systems an operation on the parent 
directory, not the file itself. So that would explain why it occasionally fails.

What is the best way to fix this? Simply have a few retries? If it still fails 
after the retries, simply use a special value for unknown file size?
The state size information is used mainly for informational purposes, like in 
the web UI and in metrics.

> Sporadic "java.lang.RuntimeException: Error triggering a checkpoint..." 
> causes task restarting
> --
>
> Key: FLINK-4218
> URL: https://issues.apache.org/jira/browse/FLINK-4218
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Sergii Koshel
>
> Sporadically see exception as below. And restart of task because of it.
> {code:title=Exception|borderStyle=solid}
> java.lang.RuntimeException: Error triggering a checkpoint as the result of 
> receiving checkpoint barrier
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:785)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:775)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(BarrierBuffer.java:203)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierBuffer.getNextNonBlocked(BarrierBuffer.java:129)
>   at 
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
>   at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
>   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: No such file or directory: 
> s3:///flink/checkpoints/ece317c26960464ba5de75f3bbc38cb2/chk-8810/96eebbeb-de14-45c7-8ebb-e7cde978d6d3
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:996)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
>   at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:351)
>   at 
> org.apache.flink.runtime.state.filesystem.AbstractFileStateHandle.getFileSize(AbstractFileStateHandle.java:93)
>   at 
> org.apache.flink.runtime.state.filesystem.FileStreamStateHandle.getStateSize(FileStreamStateHandle.java:58)
>   at 
> org.apache.flink.runtime.state.AbstractStateBackend$DataInputViewHandle.getStateSize(AbstractStateBackend.java:482)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTaskStateList.getStateSize(StreamTaskStateList.java:77)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:604)
>   at 
> org.apache.flink.streaming.runtime.tasks.StreamTask$3.onEvent(StreamTask.java:779)
>   ... 8 more
> {code}
> File actually exists on S3. 
> I suppose it is related to some race conditions with S3 but would be good to 
> retry a few times before stop task execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4660) HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in a restarting loop

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513227#comment-15513227
 ] 

Stephan Ewen commented on FLINK-4660:
-

We looked through this a bit, and the problem may be something else. Flink does 
not close the {{FileSystem}} objects, but it also caches them, so there should 
be only one {{FileSystem}} object per TaskManager.
The connections you see as open may be {{FsDataInputStream}} connections to S3, 
reloading the state. Previous versions of Flink did not ensure that the streams 
were closes in case that the recovery was intercepted by another failure (such 
as File Not Found due to eventual consistency).

The latest version of Flink more thoroughly closes these streams. Can you check 
if that fixes your problem?

For the eventual consistency issue, let's continue the discussion in 
https://issues.apache.org/jira/browse/FLINK-4218

> HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in 
> a restarting loop
> ---
>
> Key: FLINK-4660
> URL: https://issues.apache.org/jira/browse/FLINK-4660
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Reporter: Zhenzhong Xu
>Priority: Critical
> Attachments: Screen Shot 2016-09-20 at 2.49.14 PM.png, Screen Shot 
> 2016-09-20 at 2.49.32 PM.png
>
>
> Flink job with checkpoints enabled and configured to use S3A file system 
> backend, sometimes experiences checkpointing failure due to S3 consistency 
> issue. This behavior is also reported by other people and documented in 
> https://issues.apache.org/jira/browse/FLINK-4218.
> This problem gets magnified by current HadoopFileSystem implementation, which 
> can potentially leak S3 client connections, and eventually get into a 
> restarting loop with “Timeout waiting for a connection from pool” exception 
> thrown from aws client.
> I looked at the code, seems HadoopFileSystem.java never invoke close() method 
> on fs object upon failure, but the FileSystem may be re-initialized every 
> time the job gets restarted.
> A few evidence I observed:
> 1. When I set the connection pool limit to 128, and below commands shows 128 
> connections are stuck in CLOSE_WAIT state.
> !Screen Shot 2016-09-20 at 2.49.14 PM.png|align=left, vspace=5! 
> 2. task manager logs indicates that state backend file system consistently 
> getting initialized upon job restarting.
> !Screen Shot 2016-09-20 at 2.49.32 PM.png!
> 3. Log indicates there is NPE during cleanning up of stream task which was 
> caused by “Timeout waiting for connection from pool” exception when trying to 
> create a directory in S3 bucket.
> 2016-09-02 08:17:50,886 ERROR 
> org.apache.flink.streaming.runtime.tasks.StreamTask - Error during cleanup of 
> stream task
> java.lang.NullPointerException
> at 
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.cleanup(OneInputStreamTask.java:73)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589)
> at java.lang.Thread.run(Thread.java:745)
> 4.It appears StreamTask from invoking checkpointing operation, to handling 
> failure, there is no logic associated with closing Hadoop File System object 
> (which internally includes S3 aws client object), which resides in 
> HadoopFileSystem.java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513199#comment-15513199
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2533


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot resto...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2533


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4628) User class loader unavailable during input split assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513198#comment-15513198
 ] 

ASF GitHub Bot commented on FLINK-4628:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2505


> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the 
> executing thread.
> In 2 we only have the system class loader available. If any classes need to 
> be loaded then, we have a problem. Some InputFormats rely on code which 
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the 
> latest master.
> We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2505: [FLINK-4628] provide user class loader during inpu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2505


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513162#comment-15513162
 ] 

ASF GitHub Bot commented on FLINK-4496:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/2434
  
Thanks @mxm and @aljoscha !
I already have the followup on this open here: 
https://github.com/apache/flink/pull/2532 


> Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.
> 
>
> Key: FLINK-4496
> URL: https://issues.apache.org/jira/browse/FLINK-4496
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2434: [FLINK-4496] Refactor the TimeServiceProvider to take a T...

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/2434
  
Thanks @mxm and @aljoscha !
I already have the followup on this open here: 
https://github.com/apache/flink/pull/2532 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2434: [FLINK-4496] Refactor the TimeServiceProvider to take a T...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2434
  
+1 Looks like a sensible change. Looking forward to fixing the 
`ContinuousFileReaderOperator`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4496) Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513157#comment-15513157
 ] 

ASF GitHub Bot commented on FLINK-4496:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2434
  
+1 Looks like a sensible change. Looking forward to fixing the 
`ContinuousFileReaderOperator`.


> Refactor the TimeServiceProvider to take a Trigerable instead of a Runnable.
> 
>
> Key: FLINK-4496
> URL: https://issues.apache.org/jira/browse/FLINK-4496
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513129#comment-15513129
 ] 

ASF GitHub Bot commented on FLINK-4485:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
This needed another fix because in some tests we use the system class 
loader instead of a class loader instantiated by the BlobLibraryCacheManager. 
If we close that one, we cause tests to fail. The solution is to close only 
`FlinkUserCodeClassLoader`s.


> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this 
> yarn session. It is the exact same job (java code) yet it gets different 
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the 
> parameters are about which data from which tables and the name of the target 
> directory.
> After running several dozen jobs the jobs submission started to fail and we 
> investigated.
> We found that the cause was that on the Yarn node which was hosting the 
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put 
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough 
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID  USER   FD  TYPE DEVICE  SIZE   
> NODE NAME
> java  15034   nbasjes  550r  REG 253,17  66219695
> 245 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003 
> (deleted)
> java  15034   nbasjes  551r  REG 253,17  66219695
> 252 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007 
> (deleted)
> java  15034   nbasjes  552r  REG 253,17  66219695
> 267 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012 
> (deleted)
> java  15034   nbasjes  553r  REG 253,17  66219695
> 250 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005 
> (deleted)
> java  15034   nbasjes  554r  REG 253,17  66219695
> 288 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018 
> (deleted)
> java  15034   nbasjes  555r  REG 253,17  66219695
> 298 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025 
> (deleted)
> java  15034   nbasjes  557r  REG 253,17  66219695
> 254 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008 
> (deleted)
> java  15034   nbasjes  558r  REG 253,17  66219695
> 292 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019 
> (deleted)
> java  15034   nbasjes  559r  REG 253,17  66219695
> 275 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013 
> (deleted)
> java  15034   nbasjes  560r  REG 253,17  66219695
> 159 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002 
> (deleted)
> java  15034   nbasjes  562r  REG 253,17  66219695
> 238 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001 
> (deleted)
> java  15034   nbasjes  568r  REG 253,17  66219695
> 246 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0004 
> (deleted)
> java  15034   nbasjes  569r  REG 253,17  66219695
> 255 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0009 
> (deleted)
> java  15034   nbasjes  571r  REG 253,17  66219695
> 299 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0026 
> (deleted)
> java  15034   nbasjes  572r  REG 253,17  66219695
> 293 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0020 
> (deleted)
> java  15034   nbasjes  574r  REG 253,17  66219695
> 256 

[jira] [Commented] (FLINK-4485) Finished jobs in yarn session fill /tmp filesystem

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513131#comment-15513131
 ] 

ASF GitHub Bot commented on FLINK-4485:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
Merging after tests pass.


> Finished jobs in yarn session fill /tmp filesystem
> --
>
> Key: FLINK-4485
> URL: https://issues.apache.org/jira/browse/FLINK-4485
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.1.0
>Reporter: Niels Basjes
>Assignee: Maximilian Michels
>Priority: Blocker
>
> On a Yarn cluster I start a yarn-session with a few containers and task slots.
> Then I fire a 'large' number of Flink batch jobs in sequence against this 
> yarn session. It is the exact same job (java code) yet it gets different 
> parameters.
> In this scenario it is exporting HBase tables to files in HDFS and the 
> parameters are about which data from which tables and the name of the target 
> directory.
> After running several dozen jobs the jobs submission started to fail and we 
> investigated.
> We found that the cause was that on the Yarn node which was hosting the 
> jobmanager the /tmp file system was full (4GB was 100% full).
> How ever the output of {{du -hcs /tmp}} showed only 200MB in use.
> We found that a very large file (we guess it is the jar of the job) was put 
> in /tmp , used, deleted yet the file handle was not closed by the jobmanager.
> As soon as we killed the jobmanager the disk space was freed.
> The summary of the impact of this is that a yarn-session that receives enough 
> jobs brings down the Yarn node for all users.
> See parts of the output we got from {{lsof}} below.
> {code}
> COMMAND PID  USER   FD  TYPE DEVICE  SIZE   
> NODE NAME
> java  15034   nbasjes  550r  REG 253,17  66219695
> 245 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0003 
> (deleted)
> java  15034   nbasjes  551r  REG 253,17  66219695
> 252 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0007 
> (deleted)
> java  15034   nbasjes  552r  REG 253,17  66219695
> 267 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0012 
> (deleted)
> java  15034   nbasjes  553r  REG 253,17  66219695
> 250 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0005 
> (deleted)
> java  15034   nbasjes  554r  REG 253,17  66219695
> 288 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0018 
> (deleted)
> java  15034   nbasjes  555r  REG 253,17  66219695
> 298 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0025 
> (deleted)
> java  15034   nbasjes  557r  REG 253,17  66219695
> 254 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0008 
> (deleted)
> java  15034   nbasjes  558r  REG 253,17  66219695
> 292 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0019 
> (deleted)
> java  15034   nbasjes  559r  REG 253,17  66219695
> 275 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0013 
> (deleted)
> java  15034   nbasjes  560r  REG 253,17  66219695
> 159 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0002 
> (deleted)
> java  15034   nbasjes  562r  REG 253,17  66219695
> 238 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0001 
> (deleted)
> java  15034   nbasjes  568r  REG 253,17  66219695
> 246 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0004 
> (deleted)
> java  15034   nbasjes  569r  REG 253,17  66219695
> 255 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0009 
> (deleted)
> java  15034   nbasjes  571r  REG 253,17  66219695
> 299 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0026 
> (deleted)
> java  15034   nbasjes  572r  REG 253,17  66219695
> 293 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0020 
> (deleted)
> java  15034   nbasjes  574r  REG 253,17  66219695
> 256 
> /tmp/blobStore-fbe9c4cf-1f85-48cb-aad9-180e8d4ec7ce/incoming/temp-0010 
> (deleted)
> java  15034   nbasjes  575r  REG 253,17  66219695
> 302 
> 

[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
Merging after tests pass.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2499: [FLINK-4485] close and remove user class loader after job...

[mxm]

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2499
  
This needed another fix because in some tests we use the system class 
loader instead of a class loader instantiated by the BlobLibraryCacheManager. 
If we close that one, we cause tests to fail. The solution is to close only 
`FlinkUserCodeClassLoader`s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3580) Reintroduce Date/Time and implement scalar functions for it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513099#comment-15513099
 ] 

ASF GitHub Bot commented on FLINK-3580:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2468


> Reintroduce Date/Time and implement scalar functions for it
> ---
>
> Key: FLINK-3580
> URL: https://issues.apache.org/jira/browse/FLINK-3580
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> This task includes:
> {code}
> DATETIME_PLUS
> EXTRACT_DATE
> FLOOR
> CEIL
> CURRENT_TIME
> CURRENT_TIMESTAMP
> LOCALTIME
> LOCALTIMESTAMP
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2468: [FLINK-3580] [table] Add OVERLAPS function

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2468


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-4662) Bump Calcite version up to 1.9

[Jark Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu reassigned FLINK-4662:
--

Assignee: Jark Wu

> Bump Calcite version up to 1.9
> --
>
> Key: FLINK-4662
> URL: https://issues.apache.org/jira/browse/FLINK-4662
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> Calcite just released the 1.9 version. We should adopt it also in the Table 
> API especially for FLINK-4294.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Márton Balassi updated FLINK-4663:
--
Assignee: Swapnil Chougule

> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
>Assignee: Swapnil Chougule
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[Swapnil Chougule (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512971#comment-15512971
 ] 

Swapnil Chougule commented on FLINK-4663:
-

Hi Team,

Can anybody give me permissions to assign same JIRA to me ?

Thanks,
Swapnil

> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[Swapnil Chougule (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Chougule updated FLINK-4663:

Description: 
Flink JDBCOutputFormat logs wrong WARN message as 
"Column SQL types array doesn't match arity of passed Row! Check the passed 
array..."
even if there is no mismatch is SQL types array & arity of passed Row.

 
(flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)

It logs lot of unnecessary warning messages (one per row passed) in log files.


  was:
Flink JDBCOutputFormat logs wrong WARN message as 
"Column SQL types array doesn't match arity of passed Row! Check the passed 
array..."
even if there is no mismatch is SQL types array & arity of passed Row.

 
(flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)



> Flink JDBCOutputFormat logs wrong WARN message
> --
>
> Key: FLINK-4663
> URL: https://issues.apache.org/jira/browse/FLINK-4663
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats
>Affects Versions: 1.1.1, 1.1.2
> Environment: Across Platform
>Reporter: Swapnil Chougule
> Fix For: 1.1.3
>
>
> Flink JDBCOutputFormat logs wrong WARN message as 
> "Column SQL types array doesn't match arity of passed Row! Check the passed 
> array..."
> even if there is no mismatch is SQL types array & arity of passed Row.
>  
> (flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)
> It logs lot of unnecessary warning messages (one per row passed) in log files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4544) TaskManager metrics are vulnerable to custom JMX bean installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512941#comment-15512941
 ] 

ASF GitHub Bot commented on FLINK-4544:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2445
  
that should be doable, yes.


> TaskManager metrics are vulnerable to custom JMX bean installation
> --
>
> Key: FLINK-4544
> URL: https://issues.apache.org/jira/browse/FLINK-4544
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.2
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.1.3
>
>
> The TaskManager's CPU load magic may fail when JMX providers are overwritten.
> The TaskManager logic checks if the class 
> {{com.sun.management.OperatingSystemMXBean}} is available. If yes, it assumes 
> that the {{ManagementFactory.getOperatingSystemMXBean()}} is of that type. 
> That is not necessarily the case.
> This is visible in the Cassandra tests, as Cassandra overrides the JMX 
> provider - every heartbeat causes an exception that is logged (See below), 
> flooding the log, killing the heartbeat message.
> I would also suggest to move the entire metrics code out of the 
> {{TaskManager}} class into a dedicated class {{TaskManagerJvmMetrics}}. That 
> one can, with a static method, install the metrics into the TaskManager's 
> metric group.
> Sample stack trace when default platform beans are overridden:
> {code}
> 23914 [flink-akka.actor.default-dispatcher-3] WARN  
> org.apache.flink.runtime.taskmanager.TaskManager  - Error retrieving CPU Load 
> through OperatingSystemMXBean
> java.lang.IllegalArgumentException: object is not an instance of declaring 
> class
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
>   at scala.Option.map(Option.scala:145)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2351)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2348)
>   at 
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:32)
>   at 
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:20)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
>   at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2444)
>   at 
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
>   at 
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
>   at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:186)
>   at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
>   at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3631)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3022)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.sendHeartbeatToJobManager(TaskManager.scala:1278)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:309)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.testingUtils.TestingTaskManagerLike$$anonfun$handleTestingMessage$1.applyOrElse(TestingTaskManagerLike.scala:65)
>   at 

[GitHub] flink issue #2445: [FLINK-4544] Refactor old CPU metric initialization

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2445
  
that should be doable, yes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4560) enforcer java version as 1.7

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512936#comment-15512936
 ] 

ASF GitHub Bot commented on FLINK-4560:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2458
  
From my side, this is good to merge.
Unless someone objects, I will merge this later today...


> enforcer java version as 1.7
> 
>
> Key: FLINK-4560
> URL: https://issues.apache.org/jira/browse/FLINK-4560
> Project: Flink
>  Issue Type: Improvement
>Reporter: shijinkui
>
> 1. maven-enforcer-plugin add java version enforce
> 2. maven-enforcer-plugin version upgrade to 1.4.1
> explicit require java version



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2458: [FLINK-4560] enforcer java version as 1.7

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2458
  
From my side, this is good to merge.
Unless someone objects, I will merge this later today...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3719) WebInterface: Moving the barrier between graph and stats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512934#comment-15512934
 ] 

ASF GitHub Bot commented on FLINK-3719:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2467
  
Piotr (who wrote most of the web ui) also wants to leave some comments. 
Let's wait for him.


> WebInterface: Moving the barrier between graph and stats
> 
>
> Key: FLINK-3719
> URL: https://issues.apache.org/jira/browse/FLINK-3719
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Niels Basjes
>Assignee: Ivan Mushketyk
>
> It would be really useful if the separator between the graphical view of a 
> job topology at the top and the textual overview of the counters at the 
> bottom can be moved up/down. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2467: [FLINK-3719][web frontend] Moving the barrier between gra...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2467
  
Piotr (who wrote most of the web ui) also wants to leave some comments. 
Let's wait for him.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3719) WebInterface: Moving the barrier between graph and stats

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512932#comment-15512932
 ] 

ASF GitHub Bot commented on FLINK-3719:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2467
  
The license is MIT, so that is fine.
Can you update the LICENSE file with the dependency?


> WebInterface: Moving the barrier between graph and stats
> 
>
> Key: FLINK-3719
> URL: https://issues.apache.org/jira/browse/FLINK-3719
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Reporter: Niels Basjes
>Assignee: Ivan Mushketyk
>
> It would be really useful if the separator between the graphical view of a 
> job topology at the top and the textual overview of the counters at the 
> bottom can be moved up/down. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4663) Flink JDBCOutputFormat logs wrong WARN message

[Swapnil Chougule (JIRA)]

Swapnil Chougule created FLINK-4663:
---

 Summary: Flink JDBCOutputFormat logs wrong WARN message
 Key: FLINK-4663
 URL: https://issues.apache.org/jira/browse/FLINK-4663
 Project: Flink
  Issue Type: Bug
  Components: Batch Connectors and Input/Output Formats
Affects Versions: 1.1.2, 1.1.1
 Environment: Across Platform
Reporter: Swapnil Chougule
 Fix For: 1.1.3


Flink JDBCOutputFormat logs wrong WARN message as 
"Column SQL types array doesn't match arity of passed Row! Check the passed 
array..."
even if there is no mismatch is SQL types array & arity of passed Row.

 
(flink/flink-batch-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCOutputFormat.java)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4579) Add StateBackendFactory for RocksDB Backend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512931#comment-15512931
 ] 

ASF GitHub Bot commented on FLINK-4579:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2482
  
Good point. I think reflect is good, if it does not add too much work. 
Keeping the modules separate is nice, I think.


> Add StateBackendFactory for RocksDB Backend
> ---
>
> Key: FLINK-4579
> URL: https://issues.apache.org/jira/browse/FLINK-4579
> Project: Flink
>  Issue Type: Improvement
>  Components: State Backends, Checkpointing
>Reporter: Aljoscha Krettek
>Assignee: Jark Wu
>
> Right now, we only have a {{StateBackendFactory}} for the {{FsStateBackend}} 
> which means that users cannot specify to use the RocksDB backend in the flink 
> configuration.
> If we add a factory for rocksdb we should also think about adding the rocksdb 
> backend to the standard distribution lib, otherwise it is only usable if 
> users manually place the rocks jars in the Flink lib folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2467: [FLINK-3719][web frontend] Moving the barrier between gra...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2467
  
The license is MIT, so that is fine.
Can you update the LICENSE file with the dependency?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2482: [FLINK-4579] [StateBackend] Add StateBackendFactory for R...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2482
  
Good point. I think reflect is good, if it does not add too much work. 
Keeping the modules separate is nice, I think.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4628) User class loader unavailable during input split assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512926#comment-15512926
 ] 

ASF GitHub Bot commented on FLINK-4628:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2505
  
Merging this...


> User class loader unavailable during input split assignment
> ---
>
> Key: FLINK-4628
> URL: https://issues.apache.org/jira/browse/FLINK-4628
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> {{InputFormat}}\s runs through two configuration phases in the {{JobManager}}.
> 1. initializeOnMaster which runs the configure() method on the InputFormat
> 2. createInputSplits when the ExecutionJobVertex is created
> In 1 we set the user class loader as the context class loader of the 
> executing thread.
> In 2 we only have the system class loader available. If any classes need to 
> be loaded then, we have a problem. Some InputFormats rely on code which 
> lazily load classes at different points in time.
> In particular, this is a problem with the HBase TableInputFormat in the 
> latest master.
> We should make the user class loader available when creating input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2505: [FLINK-4628] provide user class loader during input split...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2505
  
Merging this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2526: [FLINK-4580] [rpc] Report rpc invocation exceptions to th...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2526
  
Nice!
+1 to merge this


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4580) Check that the RpcEndpoint supports the specified RpcGateway

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512922#comment-15512922
 ] 

ASF GitHub Bot commented on FLINK-4580:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2526
  
Nice!
+1 to merge this


> Check that the RpcEndpoint supports the specified RpcGateway
> 
>
> Key: FLINK-4580
> URL: https://issues.apache.org/jira/browse/FLINK-4580
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> When calling {{RpcService.connect}} the user specifies the type of the 
> {{RpcGateway}}. At the moment, it is not checked whether the {{RpcEndpoint}} 
> actually supports the specified {{RpcGateway}}.
> I think it would be good to add a runtime check that the corresponding 
> {{RpcEndpoint}} supports the specified {{RpcGateway}}. If not, then we can 
> let the connect method fail fast.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2533
  
I see, keeping the serializers for now makes probably sense.
It just seems that there are also user functions in there (like fold, etc) 
- those should probably be removed. May mean that we have to inject them back 
into the state descriptor later.

Orthogonal issue, so +1 for this change.

Merging this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512918#comment-15512918
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2533
  
I see, keeping the serializers for now makes probably sense.
It just seems that there are also user functions in there (like fold, etc) 
- those should probably be removed. May mean that we have to inject them back 
into the state descriptor later.

Orthogonal issue, so +1 for this change.

Merging this...


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2445: [FLINK-4544] Refactor old CPU metric initialization

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2445
  
Can we move this code out of the TaskManager as a whole, into a metrics 
Utility?
We could make it reusable for the JobManager as well, by passing the metric 
group where that should be added.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4544) TaskManager metrics are vulnerable to custom JMX bean installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512915#comment-15512915
 ] 

ASF GitHub Bot commented on FLINK-4544:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2445
  
Can we move this code out of the TaskManager as a whole, into a metrics 
Utility?
We could make it reusable for the JobManager as well, by passing the metric 
group where that should be added.


> TaskManager metrics are vulnerable to custom JMX bean installation
> --
>
> Key: FLINK-4544
> URL: https://issues.apache.org/jira/browse/FLINK-4544
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.2
>Reporter: Stephan Ewen
>Assignee: Chesnay Schepler
> Fix For: 1.2.0, 1.1.3
>
>
> The TaskManager's CPU load magic may fail when JMX providers are overwritten.
> The TaskManager logic checks if the class 
> {{com.sun.management.OperatingSystemMXBean}} is available. If yes, it assumes 
> that the {{ManagementFactory.getOperatingSystemMXBean()}} is of that type. 
> That is not necessarily the case.
> This is visible in the Cassandra tests, as Cassandra overrides the JMX 
> provider - every heartbeat causes an exception that is logged (See below), 
> flooding the log, killing the heartbeat message.
> I would also suggest to move the entire metrics code out of the 
> {{TaskManager}} class into a dedicated class {{TaskManagerJvmMetrics}}. That 
> one can, with a static method, install the metrics into the TaskManager's 
> metric group.
> Sample stack trace when default platform beans are overridden:
> {code}
> 23914 [flink-akka.actor.default-dispatcher-3] WARN  
> org.apache.flink.runtime.taskmanager.TaskManager  - Error retrieving CPU Load 
> through OperatingSystemMXBean
> java.lang.IllegalArgumentException: object is not an instance of declaring 
> class
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3$$anonfun$getValue$2.apply(TaskManager.scala:2351)
>   at scala.Option.map(Option.scala:145)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2351)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anon$3.getValue(TaskManager.scala:2348)
>   at 
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:32)
>   at 
> com.codahale.metrics.json.MetricsModule$GaugeSerializer.serialize(MetricsModule.java:20)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
>   at 
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
>   at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2444)
>   at 
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
>   at 
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
>   at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:186)
>   at 
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
>   at 
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3631)
>   at 
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3022)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager.sendHeartbeatToJobManager(TaskManager.scala:1278)
>   at 
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:309)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> 

[jira] [Commented] (FLINK-4646) Add BipartiteGraph class

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512893#comment-15512893
 ] 

Stephan Ewen commented on FLINK-4646:
-

Ah, I did not see that. Just thought that it would be good to check with some 
committers before embarking on a big implementation project.
Thanks :-)


> Add BipartiteGraph class
> 
>
> Key: FLINK-4646
> URL: https://issues.apache.org/jira/browse/FLINK-4646
> Project: Flink
>  Issue Type: Sub-task
>  Components: Gelly
>Reporter: Ivan Mushketyk
>
> Implement a class to represent a bipartite graph in Flink Gelly. Design 
> discussions can be found in the parent task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4646) Add BipartiteGraph class

[Vasia Kalavri (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512880#comment-15512880
 ] 

Vasia Kalavri commented on FLINK-4646:
--

Hi [~StephanEwen], thanks! We've discussed this in the parent issue, yes.

> Add BipartiteGraph class
> 
>
> Key: FLINK-4646
> URL: https://issues.apache.org/jira/browse/FLINK-4646
> Project: Flink
>  Issue Type: Sub-task
>  Components: Gelly
>Reporter: Ivan Mushketyk
>
> Implement a class to represent a bipartite graph in Flink Gelly. Design 
> discussions can be found in the parent task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2533: [FLINK-4603] Fixes: KeyedStateBackend cannot restore user...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2533
  
@StephanEwen at least in the RocksDB backend we could remove user code 
completely. Right now, the only thing that needs to be serialized is the 
TypeSerializer from the ValueDescriptor. It is used in a check that users can 
not provide a descriptor with a different TypeSerializer than the one that was 
used initially. We might think about removing this to support versioning of 
TypeSerializers, but how can we somehow enforce compatibility between them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---