Re: Akka dissassociated

[Bowen Li]

The amount of resource assigned to JobManager looks fine. How much resource
(CPU and memory) did you allocate for TaskManager?

On Wed, Jul 19, 2017 at 3:07 AM, Stephan Ewen  wrote:

> Hi Greg!
>
> Akka disassociation means that the network connection between the
> JobManager and TaskManager broke.
>
> This can be cause by
>  - actual failures of JobManager / TaskManager (I assume is not the case
> here)
>  - A limit in the number of open file handles
>  - Network flakeyness
>  - I have seen cases where it looks like it was caused by network overload
> where shuffles basically starve/suppress the akka network connections
>
>
> Handling this in akka is not very nice - one does not get an easy way to
> actually deal with the root issues, but the actor systems sort of hides
> these things. This is yet one more reason why I am thinking to move away
> from akka - it may simply not be the best match for what we are doing.
>
>
> Greetings,
> Stephan
>
>
> On Fri, Jul 14, 2017 at 7:11 PM, Greg Hogan  wrote:
>
> > Hi all,
> >
> > I’m having some issues with Akka running on a modest cluster where
> > increasing the parallelism results in disassociation messages.
> >
> > I am running a batch job, Gelly’s TriangleListing (for simplicity) which
> > is join-based. I have not seen this issue running AdamicAdar which is
> > sort-based.
> >
> > I have increased both of the following timeouts and the job takes less
> > than 100 seconds.
> > akka.ask.timeout: 1000 s
> > akka.lookup.timeout: 100 s
> >
> > I have not changed taskmanager.exit-on-fatal-akka-error from the default
> > value of false but the JobManager is dropping all TaskManager
> connections.
> >
> > I can run the TriangleListing job with the same 127 TaskManagers with a
> > smaller parallelism. Dropping from 2286 to around 1000 is often
> successful.
> >
> > CPU and memory should not be a bottleneck for the JobManager (18 cores
> and
> > 18 GB).
> >
> > I would be grateful for solutions, suggestions, or pointers to debugging
> > this issue.
> >
> > Thanks,
> > Greg
> >
> >
> > 2017-07-14 16:50:08,119 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (30/2286) (
> > 5a2e8f0a00530bd2216d7d3ee10688f7) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:08,312 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (26/2286) (
> > c6a91db2d6b6797768596d9f746d316f) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:09,831 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (131/2286) (
> > 2c77b1e4b90b951d3be1e09bf4cf41d2) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:10,057 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (133/2286) (
> > d0c4c4eda4f0c44fe594a1b94eb66c93) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:11,861 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (70/2286) (
> > 69ce8d91fbbad943c277ee92d3c38aaa) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:15,029 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (38/2286) (
> > a72c2dee009342bc4d90ec98427fa717) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:16,583 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (27/2286) (
> > e79ec6229d4afdc6669c1c221a19ad8c) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:19,498 INFO  org.apache.flink.runtime.
> executiongraph.ExecutionGraph
> >   - GroupReduce (Generate triplets) (44/2286) (
> > 53e35ddbd0e02d256620e5310276bea6) switched from RUNNING to FINISHED.
> > 2017-07-14 16:50:21,021 WARN  akka.remote.ReliableDeliverySupervisor
> >   - Association with remote system
> > [akka.tcp://flink@ip-10-0-28-115:40713] has failed, address is now gated
> > for [5000] ms. Reason: [Disassociated]
> > 2017-07-14 16:50:21,097 WARN  akka.remote.ReliableDeliverySupervisor
> >   - Association with remote system
> > [akka.tcp://flink@ip-10-0-21-141:45899] has failed, address is now gated
> > for [5000] ms. Reason: [Disassociated]
> > 2017-07-14 16:50:21,129 WARN  akka.remote.ReliableDeliverySupervisor
> >   - Association with remote system
> > [akka.tcp://flink@ip-10-0-27-236:37471] has failed, address is now gated
> > for [5000] ms. Reason: [Disassociated]
> > 2017-07-14 16:50:21,132 WARN  akka.remote.ReliableDeliverySupervisor
> >   - Association with remote system
> > [akka.tcp://flink@ip-10-0-18-79:45765] has failed, address is now gated
> > for [5000] ms. Reason: [Disassociated]
> > 2017-07-14 16:50:21,140 WARN  akka.remote.ReliableDeliverySupervisor
> >   - Association with remote system
> > 

[apache/flink] [Flink-7218] [JobManager] ExecutionVertex.getPreferredLocationsBasedOnInputs() will always return empty

[周思华]

I really like the new PR template, so i use it instead of the default one on 
github~

You can view, comment on, or merge this pull request online at: #4369 

What is the purpose of the change

The ExecutionVertex.getPreferredLocationsBasedOnInputs will always return 
empty, cause sourceSlot always be null until ExectionVertex has been deployed 
via 'Execution.deployToSlot()'. So allocate resource base on preferred location 
can't work correctly, we need to set the slot info for Execution as soon as 
Execution.allocateSlotForExecution() called successfully.

Brief change log
Added a field assignedFutureSlot in Execution to record the Future 
as soon asExecution.allocateSlotForExecution() called successfully. And the 
assignedFutureSlot will be used in ExectionVertex. 
getPreferredLocationsBasedOnInputs () to get ExecutionVertex's preferred 
locations.
Verifying this change

This change added tests and can be verified as follows:

The test case is under ExecutionGraphSchedulingTest. 
testExecutionVertexGetPreferredLocationsBasedOnInputs(), i have simulated the 
process of the JobGraph deployment and validated the results in this test case.
Does this pull request potentially affect one of the following parts:
Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): 
(no)
The serializers: (don't know)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), 
Checkpointing, Yarn/Mesos, ZooKeeper: (yes):
Documentation
Does this pull request introduce a new feature? (no)

[jira] [Created] (FLINK-7237) Remove DateTimeUtils from Flink once Calcite is upgraded to 1.14

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-7237:
-

 Summary: Remove DateTimeUtils from Flink once Calcite is upgraded 
to 1.14
 Key: FLINK-7237
 URL: https://issues.apache.org/jira/browse/FLINK-7237
 Project: Flink
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7236) Bump up the Calcite version to 1.14

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-7236:
-

 Summary: Bump up the Calcite version to 1.14
 Key: FLINK-7236
 URL: https://issues.apache.org/jira/browse/FLINK-7236
 Project: Flink
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


This is the umbrella task to coordinate tasks to upgrade Calcite to 1.14.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7235) Backport CALCITE-1884 to the Flink repository before Calcite 1.14

[Haohui Mai (JIRA)]

Haohui Mai created FLINK-7235:
-

 Summary: Backport CALCITE-1884 to the Flink repository before 
Calcite 1.14
 Key: FLINK-7235
 URL: https://issues.apache.org/jira/browse/FLINK-7235
 Project: Flink
  Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai


We need to backport CALCITE-1884 in order to unblock upgrading Calcite to 1.13.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Using native library in Flink

[Mike Accola]

Timo/Eron -

Thank you for the responses. To answer a few of your questions:

- For now, I am just running in a simple, local environment 
(start-local.sh)
- I have this entry in the flink-conf.yaml file:  env.java.opts : 
"-Djava.library.path=/myPathWithTheLibrary".  From looking at logs, it 
looks like the JVM is picking up this setting.  (plus if I remove the 
setting, things don't work at all).
- I am loading the library within the processElement() method of my 
ProcessFunction class.

I applied Eron's example of adding the library to my jar file and then 
extracting/loading.  This  seems to work for me.  So at least I have a 
workaround for now (thank you!).  However, it really seems like this is a 
hack that I should not have to do.  I am running on a single system and 
the java.library.path is an absolute path on this system.  I'd love to 
figure out why this is happening and a better way to get around it.

One things I've noted:  In the job manager logs, it appears 
java.library.path is getting set as expected.  But if I do 
System.getProperty("java.library.path") within my processElement method to 
check the property, the results are erratic:  Sometimes I see the value 
from my flink-conf.yaml.  Other times I see something totally different 
that appears to be the jvm default.  More confusing is that seeing these 
different values for java.library.path do NOT seem to correlate to whether 
the library loads successfully or not.  If I run this same application 
twice in succession, am I running in different processes or JVMs?

Please reply if anyone has suggestions on other things to try.

--Mike





From:   Eron Wright 
To: dev@flink.apache.org
Date:   07/18/2017 04:40 PM
Subject:Re: Using native library in Flink



The solution mentioned by Timo works well with a standalone Flink cluster
but might not work with a YARN or Mesos cluster.  An alternative is to 
have
your Java library contain the native library within itself, and to extract
it to a temporary directory before calling `System.loadLibrary(...)`.
Note that you lose the advantages of using the native OS's packaging 
system
(e.g. security patches, dependency management).   The TensorFlow Java
library demonstrates the technique:

https://github.com/tensorflow/tensorflow/blob/v1.2.1/tensorflow/java/src/main/java/org/tensorflow/NativeLibrary.java


-Eron

On Tue, Jul 18, 2017 at 8:02 AM, Timo Walther  wrote:

> Hi Mike,
>
> do you run Flink locally or in a cluster? You have to make sure that VM
> argument -Djava.library.path is set for all Flink JVMs. Job Manager and
> Task Managers might run in separate JVMs. Make also sure that the 
library
> is accessible from all node. I don't know what happens if the file is
> accessed by multiple processes/threads at the same time. It might also
> important where you put the static { ... } loading. It should be in the
> Function, because these classes get deserialized on the TaskManager.
>
> I hope this helps.
>
> Timo
>
>
> Am 17.07.17 um 21:30 schrieb Mike Accola:
>
> I am new Flink user just trying to learn a little bit.  I am trying to
>> incorporate an existing C++ library into a new Flink application.  I am
>> seeing some strange behavior when trying to link in the native (C++)
>> library using java via JNI.
>>   I am running this on Linux (RHEL6)
>>   I can run my application once without error.  Sometimes it will run
>> successfully a 2nd or 3rd time.  However, eventually on a subsequent 
run,
>> I get an exception about the the native library not being found:
>>   java.lang.UnsatisfiedLinkError: no dummy2native in java.library.path
>>  at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
>>  at java.lang.Runtime.loadLibrary0(Runtime.java:870)
>>  at java.lang.System.loadLibrary(System.java:1122)
>>  at com.att.flink.tdata.spss.TinyLoader.loadNative(Dummy2.java:
>> 10)
>>   For debugging purposes for now, my native library does not have any
>> external references.  It really contains 1 method that essentially does
>> nothing.
>>   The behavior seems to indicate that there is some kind of cleanup 
being
>> done that "unloads" the native library.  I suspect this is somehow 
related
>> to Flink's implementation of its library cache manager, but I have not
>> been able to prove this yet.
>>   A few more details:
>>   - I have a c++ library libdummy2native.so that contains a method that
>> can
>> be invoked via JNI.
>> - I have a jar containing a class, called Dummy2.  The Dummy2 
constructor
>> will invoke the JNI method.
>> - The libdummy2native.so library is invoked with System.loadLibrary() 
like
>> this:
>>   static {System.loadLibrary("dummy2native"); }
>> - In my simple Flink application, I have extended the ProcessFunction
>> class.  Within this class, I have overriden processElement method that
>> declares a Dummy2 object.
>> - The Dummy2 class can be called and invoked without error when used 

[jira] [Created] (FLINK-7234) Fix CombineHint documentation

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-7234:
-

 Summary: Fix CombineHint documentation
 Key: FLINK-7234
 URL: https://issues.apache.org/jira/browse/FLINK-7234
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.2.2, 1.4.0, 1.3.2
Reporter: Greg Hogan
Assignee: Greg Hogan


The {{CombineHint}} 
[documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/batch/index.html]
 applies to {{DataSet#reduce}} not {{DataSet#reduceGroup}} and should also be 
note for {{DataSet#distinct}}. It is also set with 
{{.setCombineHint(CombineHint)}} rather than alongside the UDF parameter.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7233) TaskManagerHeapSizeCalculationJavaBashTest failed on Travis

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7233:
---

 Summary: TaskManagerHeapSizeCalculationJavaBashTest failed on 
Travis
 Key: FLINK-7233
 URL: https://issues.apache.org/jira/browse/FLINK-7233
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.4.0
 Environment: https://travis-ci.org/apache/flink/jobs/255289918
Reporter: Chesnay Schepler


{code}
Running org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.926 sec <<< 
FAILURE! - in org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest
compareNetworkBufShellScriptWithJava(org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest)
  Time elapsed: 1.443 sec  <<< ERROR!
org.apache.flink.configuration.IllegalConfigurationException: Invalid 
configuration value for taskmanager.network.memory.min : -2147479274 - Minimum 
memory for network buffers must allow at least one network buffer with respect 
to the memory segment size
at 
org.apache.flink.runtime.taskexecutor.TaskManagerServicesConfiguration.checkConfigParameter(TaskManagerServicesConfiguration.java:459)
at 
org.apache.flink.runtime.taskexecutor.TaskManagerServicesConfiguration.checkNetworkBufferConfig(TaskManagerServicesConfiguration.java:397)
at 
org.apache.flink.runtime.taskexecutor.TaskManagerServices.calculateNetworkBufferMemory(TaskManagerServices.java:427)
at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.getRandomConfig(TaskManagerHeapSizeCalculationJavaBashTest.java:198)
at 
org.apache.flink.dist.TaskManagerHeapSizeCalculationJavaBashTest.compareNetworkBufShellScriptWithJava(TaskManagerHeapSizeCalculationJavaBashTest.java:95)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7232) Update checkstyle docs regarding test inclusion

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7232:
---

 Summary: Update checkstyle docs regarding test inclusion
 Key: FLINK-7232
 URL: https://issues.apache.org/jira/browse/FLINK-7232
 Project: Flink
  Issue Type: Improvement
  Components: Checkstyle, Documentation
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.4.0


The checkstyle setup guide says to restrict checkstyle to sources and exclude 
tests.

We do however also apply checkstyle to tests, and the documentation should 
reflect that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7231) SlotSharingGroups are not always released in time for new restarts

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-7231:
---

 Summary: SlotSharingGroups are not always released in time for new 
restarts
 Key: FLINK-7231
 URL: https://issues.apache.org/jira/browse/FLINK-7231
 Project: Flink
  Issue Type: Bug
  Components: Distributed Coordination
Affects Versions: 1.3.1
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.4.0, 1.3.2


In the case where there are not enough resources to schedule the streaming 
program, a race condition can lead to a sequence of the following errors:

{code}
java.lang.IllegalStateException: SlotSharingGroup cannot clear task assignment, 
group still has allocated resources.
{code}

This eventually recovers, but may involve many fast restart attempts before 
doing so.

The root cause is that slots are not cleared before the next restart attempt.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Re: AVRO Union type support in Flink

[Vishnu Viswanath]

Hi Timo,

I just modified AvroOutputFormatTest to test this and it works fine!. I
don't plan to use it to key by, but it is a good point. Thanks.

Regards,
Vishnu

On Wed, Jul 19, 2017 at 10:57 AM, Timo Walther  wrote:

> We have similar checks in our KafkaAvroTableSource, but I could not find
> such a check in AvroTypeInfo. The field should have a generic type, so you
> can work with it. If you want to use it as key, you might have to use a
> mapper before and convert it into a valid key type.
>
> Timo
>
>
>  Weitergeleitete Nachricht 
> Betreff: Re: AVRO Union type support in Flink
> Datum: Wed, 19 Jul 2017 10:26:24 -0400
> Von: Vishnu Viswanath 
> 
> An: Timo Walther  
>
>
> Hi Timo,
>
> Thanks for checking that. I did not try yet. My current application uses
> Cascading and it has the limitation that Union cannot contain two concrete
> types - link
> ,
> so was wondering if I can use Flink. Will give it a try.
>
> Hi Martin,
> The documentation is here
> 
> I use it to create AVRO files from source files in S3 and write to Kafka.
>
> Thanks,
> Vishnu
>
>
> On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther  wrote:
>
>> Hi Vishnu,
>>
>> I took a look into the code. Actually, we should support it. However,
>> those types might be mapped to Java Objects that will be serialized with
>> our generic Kryo serializer. Have you tested it?
>>
>> Regards,
>> Timo
>>
>>
>> Am 19.07.17 um 06:30 schrieb Martin Eden:
>>
>> Hey Vishnu,
>>
>> For those of us on the list that are not very familiar with Flink and
>> Avro can you give a pointed to the docs you are referring to and how you
>> intend to use it? Just so we gain understanding as well.
>>
>> Thanks,
>> Martin
>>
>> On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath <
>> vishnu.viswanat...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Does Flink support AVRO union types - Documentation says it supports
>>> nullable types: {"name": "type_double_test", "type": ["null", "double"]}
>>>
>>> But my schema has something like : {"name": "union_field", "type":
>>> ["string", "double"]}
>>>
>>> Thanks
>>> Vishnu
>>>
>>>
>>>
>>
>>
>

Re: [DISCUSS] Release testing procedures, Flink 1.3.2

[Aljoscha Krettek]

Hi,

Yes! In my opinion, the most critical issues are these:

 - https://issues.apache.org/jira/browse/FLINK-6964: 
 Fix recovery for 
incremental checkpoints in StandaloneCompletedCheckpointStore
 - https://issues.apache.org/jira/browse/FLINK-7041: 
 Deserialize StateBackend 
from JobCheckpointingSettings with user classloader

The first one makes incremental checkpoints on RocksDB unusable with 
externalised checkpoints. The latter means that you cannot have custom 
configuration of the RocksDB backend.

 - https://issues.apache.org/jira/browse/FLINK-7216: 
 ExecutionGraph can perform 
concurrent global restarts to scheduling
 - https://issues.apache.org/jira/browse/FLINK-7153: 
 Eager Scheduling can't 
allocate source for ExecutionGraph correctly

These are critical scheduler bugs, Stephan can probably say more about them 
than I can.

 - https://issues.apache.org/jira/browse/FLINK-7143: 
 Partition assignment for 
Kafka consumer is not stable
 - https://issues.apache.org/jira/browse/FLINK-7195: 
 FlinkKafkaConsumer should 
not respect fetched partitions to filter restored partition states
 - https://issues.apache.org/jira/browse/FLINK-6996: 
 FlinkKafkaProducer010 
doesn't guarantee at-least-once semantic

The first one means that you can have duplicate data because several consumers 
would be consuming from one partition, without noticing it. The second one 
causes partitions to be dropped from state if a broker is temporarily not 
reachable.

The first two issues would have been caught by my proposed testing procedures, 
the third and fourth might be caught but are very tricky to provoke. I’m 
currently experimenting with this testing procedure to Flink 1.3.1 to see if I 
can provoke it.

The Kafka bugs are super hard to provoke because they only occur if Kafka has 
some temporary problems or there are communication problems.

I forgot to mention that I have actually two goals with this: 1) thoroughly 
test Flink and 2) build expertise in the community, i.e. we’re forced to try 
cluster environments/distributions that we are not familiar with and we 
actually deploy a full job and play around with it.

Best,
Aljoscha


> On 19. Jul 2017, at 15:49, Shaoxuan Wang  wrote:
> 
> Hi Aljoscha,
> Glad to see that we have a more thorough testing procedure. Could you
> please share us what (the critical issues you mentioned) have been broken
> in 1.3.0 & 1.3.1, and how the new proposed "functional testing section and
> a combination of systems/configurations" can cover this. This will help us
> to improve our production verification as well.
> 
> Regards,
> Shaoxuan
> 
> 
> On Wed, Jul 19, 2017 at 9:11 PM, Aljoscha Krettek 
> wrote:
> 
>> Hi Everyone,
>> 
>> We are on the verge of starting the release process for Flink 1.3.2 and
>> there have been some critical issues in both Flink 1.3.0 and 1.3.1. For
>> Flink 1.3.2 I want to make very sure that we test as much as possible. For
>> this I’m proposing a slightly changed testing procedure [1]. This is
>> similar to the testing document we used for previous releases but has a new
>> functional testing section that tries to outline a testing procedure and a
>> combination of systems/configurations that we have to test. Please have a
>> look and comment on whether you think this is sufficient (or a bit too
>> much).
>> 
>> What do you think?
>> 
>> Best,
>> Aljoscha
>> 
>> [1] https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj
>> 4_CEmMTpCkY81s/edit?usp=sharing

Fwd: Re: AVRO Union type support in Flink

[Timo Walther]

We have similar checks in our KafkaAvroTableSource, but I could not find 
such a check in AvroTypeInfo. The field should have a generic type, so 
you can work with it. If you want to use it as key, you might have to 
use a mapper before and convert it into a valid key type.


Timo



 Weitergeleitete Nachricht 
Betreff:Re: AVRO Union type support in Flink
Datum:  Wed, 19 Jul 2017 10:26:24 -0400
Von:Vishnu Viswanath 
An: Timo Walther 



Hi Timo,

Thanks for checking that. I did not try yet. My current application uses 
Cascading and it has the limitation that Union cannot contain two 
concrete types - link 
, 
so was wondering if I can use Flink. Will give it a try.


Hi Martin,
The documentation is here 


I use it to create AVRO files from source files in S3 and write to Kafka.

Thanks,
Vishnu


On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther > wrote:


   Hi Vishnu,

   I took a look into the code. Actually, we should support it.
   However, those types might be mapped to Java Objects that will be
   serialized with our generic Kryo serializer. Have you tested it?

   Regards,
   Timo


   Am 19.07.17 um 06:30 schrieb Martin Eden:

Hey Vishnu,

For those of us on the list that are not very familiar with Flink
and Avro can you give a pointed to the docs you are referring to
and how you intend to use it? Just so we gain understanding as well.

Thanks,
Martin

On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath
> wrote:

Hi All,

Does Flink support AVRO union types - Documentation says it
supports nullable types: {"name": "type_double_test", "type":
["null", "double"]}

But my schema has something like : {"name": "union_field",
"type": ["string", "double"]}

Thanks
Vishnu

[jira] [Created] (FLINK-7230) Travis sometimes fails due to java version imcompatibilities

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7230:
---

 Summary: Travis sometimes fails due to java version 
imcompatibilities
 Key: FLINK-7230
 URL: https://issues.apache.org/jira/browse/FLINK-7230
 Project: Flink
  Issue Type: Bug
  Components: Tests, Travis
Affects Versions: 1.4.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Priority: Critical
 Fix For: 1.4.0


The travis builds currently fail sometimes because for some reason snapshot 
artifacts are downloaded when executing tests, after compilation. This causes 
issues for the java 7 profiles, as the snapshot artifacts are released with 
java 8.

The cause for this is currently unknown; I'm currently trying builds that 
forbid downloading snapshot artifacts or downloading artifacts during testing 
in general.

Since this issue doesn't always occur I will run these tests multiple times 
until tomorrow, and if the issue hasn't appeared again will open a PR.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[VOTE] Release Apache Flink-shaded 1.0 (RC1)

[Chesnay Schepler]

Dear Flink community,

Please vote on releasing the following candidate as Apache Flink-shaded 
version 1.0.


The commit to be voted in:
https://gitbox.apache.org/repos/asf/flink-shaded/commit/fd3033ba9ead310478963bf43e09cd50d1e36d71

Branch:
release-1.0-rc1

The release artifacts to be voted on can be found at: 
http://home.apache.org/~chesnay/flink-shaded-1.0-rc1/ 



The release artifacts are signed with the key with fingerprint 
19F2195E1B4816D765A2C324C2EED7B111D464BA:

http://www.apache.org/dist/flink/KEYS

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapacheflink-1130

-


The vote ends on Monday (5pm CEST), July 24th, 2017.

[ ] +1 Release this package as Apache Flink-shaded 1.0
[ ] -1 Do not release this package, because ...

-


The flink-shaded project contains a number of shaded dependencies for 
Apache Flink.


This release includes asm-all:5.0.4, guava:18.0, netty-all:4.0.27-FINAL 
and netty-router:1.10 . Note that netty-all and netty-router are bundled 
as a single dependency.


The purpose of these dependencies is to provide a single instance of a 
shaded dependency in the Apache Flink distribution, instead of each 
individual module shading the dependency.


For more information, see
https://issues.apache.org/jira/browse/FLINK-6529.

Re: [DISCUSS] Release testing procedures, Flink 1.3.2

[Shaoxuan Wang]

Hi Aljoscha,
Glad to see that we have a more thorough testing procedure. Could you
please share us what (the critical issues you mentioned) have been broken
in 1.3.0 & 1.3.1, and how the new proposed "functional testing section and
a combination of systems/configurations" can cover this. This will help us
to improve our production verification as well.

Regards,
Shaoxuan


On Wed, Jul 19, 2017 at 9:11 PM, Aljoscha Krettek 
wrote:

> Hi Everyone,
>
> We are on the verge of starting the release process for Flink 1.3.2 and
> there have been some critical issues in both Flink 1.3.0 and 1.3.1. For
> Flink 1.3.2 I want to make very sure that we test as much as possible. For
> this I’m proposing a slightly changed testing procedure [1]. This is
> similar to the testing document we used for previous releases but has a new
> functional testing section that tries to outline a testing procedure and a
> combination of systems/configurations that we have to test. Please have a
> look and comment on whether you think this is sufficient (or a bit too
> much).
>
> What do you think?
>
> Best,
> Aljoscha
>
> [1] https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj
> 4_CEmMTpCkY81s/edit?usp=sharing

[DISCUSS] Release testing procedures, Flink 1.3.2

[Aljoscha Krettek]

Hi Everyone,

We are on the verge of starting the release process for Flink 1.3.2 and there 
have been some critical issues in both Flink 1.3.0 and 1.3.1. For Flink 1.3.2 I 
want to make very sure that we test as much as possible. For this I’m proposing 
a slightly changed testing procedure [1]. This is similar to the testing 
document we used for previous releases but has a new functional testing section 
that tries to outline a testing procedure and a combination of 
systems/configurations that we have to test. Please have a look and comment on 
whether you think this is sufficient (or a bit too much).

What do you think?

Best,
Aljoscha

[1] 
https://docs.google.com/document/d/16fU1cpxoYf3o9cCDyakj7ZDnUoJTj4_CEmMTpCkY81s/edit?usp=sharing

[jira] [Created] (FLINK-7229) Flink doesn't deleted old checkpoint

[Jason Zhou (JIRA)]

Jason Zhou created FLINK-7229:
-

 Summary: Flink doesn't deleted old checkpoint
 Key: FLINK-7229
 URL: https://issues.apache.org/jira/browse/FLINK-7229
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.3.1
 Environment: Six Flink nodes running on Ubuntu 14.04.5 LTS (GNU/Linux 
3.13.0-121-generic x86_64)
Reporter: Jason Zhou


I have a 6-node Flink cluster where one contains jobmanager and the others have 
only taskmanagers. All taskmanagers have the following config :
```
state.backend: rocksdb
state.backend.rocksdb.checkpointdir: file:///opt/flink/data/local-checkpoints
state.backend.fs.checkpointdir: file:///opt/flink/data/local-checkpoints
state.checkpoints.dir: file:///opt/flink/data/glusterfs/external-checkpoints
state.checkpoints.num-retained: 3
```
And both checkpoints and external-checkpoints are enabled in my code.  I can 
see that for the external-checkpoint, Flink retains 3 checkpoints metadata. 
However, for the local-checkpoint, only one node(with jobmanager) retains 3 
checkpoints and the others don't delete checkpoints. 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7227) OR expression with more than 2 predicates is not pushed into a TableSource

[Usman Younas (JIRA)]

Usman Younas created FLINK-7227:
---

 Summary: OR expression with more than 2 predicates is not pushed 
into a TableSource
 Key: FLINK-7227
 URL: https://issues.apache.org/jira/browse/FLINK-7227
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.3.1
Reporter: Usman Younas


It seems that {{RexNodeToExpressionConverter}} cannot handle OR expressions 
with more than 2 predicates. Therefore the expression is not pushed into a 
{{FilterableTableSource}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7226) REST responses contain invalid content-encoding header

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7226:
---

 Summary: REST responses contain invalid content-encoding header
 Key: FLINK-7226
 URL: https://issues.apache.org/jira/browse/FLINK-7226
 Project: Flink
  Issue Type: Bug
  Components: Webfrontend
Affects Versions: 1.3.1, 1.1.4, 1.2.0, 1.4.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.4.0, 1.3.2


FLINK-5705 made changes to the {{RuntimeMonitorHandler}} to set the 
{{content-encoding}} header to {{UTF-8}}. This however isn't a valid value for 
this header, and should instead be included in the {{content-type}} header.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7225) Cutoff exception message in StateDescriptor

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-7225:
---

 Summary: Cutoff exception message in StateDescriptor
 Key: FLINK-7225
 URL: https://issues.apache.org/jira/browse/FLINK-7225
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.4.0
Reporter: Chesnay Schepler


When the type extraction fails in the StateDescriptor constructor an exception 
is thrown, but the message is cutoff and doesn't contain any advice to remedy 
the situation.

{code}
try {
this.typeInfo = TypeExtractor.createTypeInfo(type);
} catch (Exception e) {
throw new RuntimeException("Cannot create full type 
information based on the given class. If the type has generics, please", e);
}
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Exported metrics by Datadog/Statsd cannot be grouped due to varying metric name

[Chesnay Schepler]

If you're using the StatsDReporter there is currently no way to use 
tags. There is an open PR to

add support for tags though: https://github.com/apache/flink/pull/4188

For DataDog specifically we have a separate DatadogReporter that uses 
tags. For this reporter
the metric name can be configured using scope formats, as described in 
the metrics documentation 
.


On 14.07.2017 12:51, Mustafa AKIN wrote:

In my machine, metrics from task manager are currently exported to Datadog
as:

"localhost.taskmanager.9fbab10326796246e65034ca260516ce.Status.JVM.GarbageColl..."

However, using this metric name causes software to treat them as different
metrics, and I cannout group by/aggregate across many hosts. It would be
much better, if they could export with "taskmanager" metric name, and
hostname within the tags. I also exported to both statsd and graphite for
InluxDB and the situation is same.

I would be willing to send a pull request, if you would like.

Mustafa Akın
www.mustafaak.in

Re: Dropping Java 7 support

[Chesnay Schepler]

Are the specific things we want to change right away? (build profiles 
would be one thing)


Would be neat to collect them in an umbrella issue.

On 18.07.2017 16:49, Timo Walther wrote:

Hurray! Finally IntStreams, LongStreams, etc. in our stream processor ;-)

Timo

Am 18.07.17 um 16:31 schrieb Stephan Ewen:

Hi all!

Over the last days, there was a longer poll running concerning 
dropping the

support for Java 7.

The feedback from users was unanimous - in favor of dropping Java 7 and
going ahead with Java 8.

So let's do that!

Greetings,
Stephan

-- Forwarded message --
From: Stephan Ewen 
Date: Tue, Jul 18, 2017 at 4:29 PM
Subject: Re: [POLL] Who still uses Java 7 with Flink ?
To: user 


All right, thanks everyone.

I think the consensus here is clear :-)

On Thu, Jul 13, 2017 at 5:17 PM, nragon 

Re: Akka dissassociated

[Stephan Ewen]

Hi Greg!

Akka disassociation means that the network connection between the
JobManager and TaskManager broke.

This can be cause by
 - actual failures of JobManager / TaskManager (I assume is not the case
here)
 - A limit in the number of open file handles
 - Network flakeyness
 - I have seen cases where it looks like it was caused by network overload
where shuffles basically starve/suppress the akka network connections


Handling this in akka is not very nice - one does not get an easy way to
actually deal with the root issues, but the actor systems sort of hides
these things. This is yet one more reason why I am thinking to move away
from akka - it may simply not be the best match for what we are doing.


Greetings,
Stephan


On Fri, Jul 14, 2017 at 7:11 PM, Greg Hogan  wrote:

> Hi all,
>
> I’m having some issues with Akka running on a modest cluster where
> increasing the parallelism results in disassociation messages.
>
> I am running a batch job, Gelly’s TriangleListing (for simplicity) which
> is join-based. I have not seen this issue running AdamicAdar which is
> sort-based.
>
> I have increased both of the following timeouts and the job takes less
> than 100 seconds.
> akka.ask.timeout: 1000 s
> akka.lookup.timeout: 100 s
>
> I have not changed taskmanager.exit-on-fatal-akka-error from the default
> value of false but the JobManager is dropping all TaskManager connections.
>
> I can run the TriangleListing job with the same 127 TaskManagers with a
> smaller parallelism. Dropping from 2286 to around 1000 is often successful.
>
> CPU and memory should not be a bottleneck for the JobManager (18 cores and
> 18 GB).
>
> I would be grateful for solutions, suggestions, or pointers to debugging
> this issue.
>
> Thanks,
> Greg
>
>
> 2017-07-14 16:50:08,119 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (30/2286) (
> 5a2e8f0a00530bd2216d7d3ee10688f7) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:08,312 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (26/2286) (
> c6a91db2d6b6797768596d9f746d316f) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:09,831 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (131/2286) (
> 2c77b1e4b90b951d3be1e09bf4cf41d2) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:10,057 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (133/2286) (
> d0c4c4eda4f0c44fe594a1b94eb66c93) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:11,861 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (70/2286) (
> 69ce8d91fbbad943c277ee92d3c38aaa) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:15,029 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (38/2286) (
> a72c2dee009342bc4d90ec98427fa717) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:16,583 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (27/2286) (
> e79ec6229d4afdc6669c1c221a19ad8c) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:19,498 INFO  
> org.apache.flink.runtime.executiongraph.ExecutionGraph
>   - GroupReduce (Generate triplets) (44/2286) (
> 53e35ddbd0e02d256620e5310276bea6) switched from RUNNING to FINISHED.
> 2017-07-14 16:50:21,021 WARN  akka.remote.ReliableDeliverySupervisor
>   - Association with remote system
> [akka.tcp://flink@ip-10-0-28-115:40713] has failed, address is now gated
> for [5000] ms. Reason: [Disassociated]
> 2017-07-14 16:50:21,097 WARN  akka.remote.ReliableDeliverySupervisor
>   - Association with remote system
> [akka.tcp://flink@ip-10-0-21-141:45899] has failed, address is now gated
> for [5000] ms. Reason: [Disassociated]
> 2017-07-14 16:50:21,129 WARN  akka.remote.ReliableDeliverySupervisor
>   - Association with remote system
> [akka.tcp://flink@ip-10-0-27-236:37471] has failed, address is now gated
> for [5000] ms. Reason: [Disassociated]
> 2017-07-14 16:50:21,132 WARN  akka.remote.ReliableDeliverySupervisor
>   - Association with remote system
> [akka.tcp://flink@ip-10-0-18-79:45765] has failed, address is now gated
> for [5000] ms. Reason: [Disassociated]
> 2017-07-14 16:50:21,140 WARN  akka.remote.ReliableDeliverySupervisor
>   - Association with remote system
> [akka.tcp://flink@ip-10-0-29-112:41017] has failed, address is now gated
> for [5000] ms. Reason: [Disassociated]
> 2017-07-14 16:50:21,142 WARN  akka.remote.ReliableDeliverySupervisor
>   - Association with remote system
> [akka.tcp://flink@ip-10-0-25-70:39625] has failed, address is now gated
> for [5000] ms. Reason: [Disassociated]
> 2017-07-14 16:50:21,159 WARN  

Re: Latency Measurement

[Chesnay Schepler]

I originally meant startNewChain(), but disableChaining() should work too.

Can you rerun the job with the logging level set to DEBUG, and check for 
any message from org.apache.flink.runtime.metrics?


Also looping in Robert, maybe he has an idea.

On 17.07.2017 14:23, Paolo Cristofanelli wrote:

Hi Chesnay,

thanks for your answer. I have not found the method createNewChain(), 
I used instead disableChaining(), but with no effect:


 DataStream stream = env.addSource(

new FlinkKafkaConsumer08<>(

"MyTopic", new SimpleStringSchema(), properties) );


   stream.map( new ConsumerMap()).disableChaining();


   env.execute();



Best Regards,
Paolo

On 17 July 2017 at 13:10, Chesnay Schepler > wrote:


Hello,

As for 1), my suspicion is that this is caused by chaining. If the
map function is chained to the kafka source then the latency
markers are always immediately forwarded, regardless of what your
map function is doing.
If the map function is indeed chained to the source, could you try
again after disabling the chain by calling
`X.map(...).createNewChain()` and report back?

As for 2), I don't think this is possible right now.

Regards,
Chesnay


On 17.07.2017 12:42, Paolo Cristofanelli wrote:

Hi,

I would like to understand how to measure the latency of a record.
I have set up a simple project with a Kafka consumer that
reads from a topic and performs a simple map (with a thread
sleep inside).

In order to measure the latency of this mapper I have added
env.getConfig().setLatencyTrackingInterval(10);

After that, I was planning to access the latency through the
webUI interface but the related graph does not show any values.
I do not understand why. I was thinking that I in the graph I
should observe at least the sleep duration.

I also have another question:

I am using a count window, aggregating every 100 input records
and then I perform a map. I want to see the latency as the
difference between the time at which the output record is
emitted and the arrival time of the earliest input record.

For example, the first value arrives at x. After x +5 I all
the 100 values arrived and the system can aggregate them. Now
I perform the map operation and we emit the output record at
time x+15.
I would like to obtain 15 as latency.
Do you have any suggestion on how to proceed?

Thanks for your time,
Paolo Cristofanelli

Re: [DISCUSS] A more thorough Pull Request check list and template

[Stephan Ewen]

@Chesnay:

Put text into template => contributor will have to read it
Put link to text into template => most contributors will ignore the link

Yes, that is pretty much what my observation from the past is.



On Tue, Jul 18, 2017 at 11:03 PM, Chesnay Schepler 
wrote:

> I'm sorry but i can't follow your logic.
>
> Put text into template => contributor will definitely read it
> Put link to text into template => contributor will completely ignore the
> link
>
> The advantage of the link is we don't duplicate the contribution guide in
> the docs and in the template.
> Furthermore, you don't even need to remove something from the template,
> since it's just a single line.
>
>
> On 18.07.2017 19:25, Stephan Ewen wrote:
>
>> Concerning moving text to the contributors guide:
>>
>> I can only say it again: I believe the contribution guide is almost dead
>> text. Very few people read it.
>> Before the current template was introduced, new contributors rarely gave
>> the pull request a name with Jira number. That is a good indicator about
>> how many read this guide.
>> Putting the test in the template is a way that every one reads it.
>>
>>
>> I am also wondering what the concern is.
>> A new contributor should clearly read through a bit of text, to learn what
>> we look for in contributions.
>> A recurring contributor will not have to read it again, simply remove the
>> text from the pull request message and go on.
>>
>> Where is the disadvantage?
>>
>>
>> On Tue, Jul 18, 2017 at 5:35 PM, Nico Kruber 
>> wrote:
>>
>> I like the new template but also agree with the text being too long and
>>> would
>>> move the intro to the contributors guide with a link in the PR template.
>>>
>>> Regarding the questions to fill out - I'd like the headings to be short
>>> and
>>> have the affected components last so that documentation is not lost
>>> (although
>>> being more important than this checklist), e.g.:
>>>
>>> * Purpose of the change
>>> * Brief change log
>>> * Verifying the change
>>> * Documentation
>>> * Affected components
>>>
>>> The verification options in the original template look a bit too large
>>> but
>>> it
>>> stresses what tests should be added, especially for bigger changes. Can't
>>> think of a way to make it shorter though.
>>>
>>>
>>> Nico
>>>
>>> On Tuesday, 18 July 2017 11:20:41 CEST Chesnay Schepler wrote:
>>>
 I fully agree with Fabian.

 Multiple-choice questions provide little value to the reviewer, since
 the
 validity has to be verified in any case. While text answers have to be
 validated as well,
 they give some hint to the reviewer as to how it can be verified and
 which steps the
 contributor did to do so.

 I also agree that it is too long; IMO this is really intimidating to new
 contributors to be greeted with this.

 Ideally we only link to the contributors guide and ask 3 questions:

* What is the problem?
* How was it fixed?
* How can the fix be verified?

 On 18.07.2017 10:47, Fabian Hueske wrote:

> I like the sections about purpose, change log, and verification of the
> changes.
>
> However, I think the proposed template is too much text. This is
>
 probably
>>>
 the reason why the first attempt to establish a PR template failed.
> I would move most of the introduction and explanations incl. examples
>
 to
>>>
 the "Contribution Guidelines" and only pass a link.
> IMO, the template should be rather shorter than the current one and
>
 only
>>>
 have the link, the sections to fill out, and checkboxes.
>
> I'm also not sure how much the detailed questions will help.
> For example even if the question about changed dependencies is answered
> with "no", the reviewer still has to check that.
>
> I think the questions of the current template work differently.
> A question "Does the PR include tests?" suggests to the contributor
>
 that
>>>
 those should be included. Same for documentation.
>
> Cheers,
> Fabian
>
> 2017-07-18 10:05 GMT+02:00 Tzu-Li (Gordon) Tai :
>
>> +1, I like this a lot.
>> With the previous template, it doesn’t really resonate with what we
>> should
>> care about, and therefore most of the time I think contributors just
>> delete
>> that template and write down something on their own.
>>
>> I would also like to add: “Savepoint / checkpoint binary formats” to
>>
> the
>>>
 potential affect scope check list.
>> I think changes to those deserves an independent yes / no check of its
>> own.
>>
>> Cheers,
>> Gordon
>>
>> On 18 July 2017 at 3:49:42 PM, Ufuk Celebi (u...@apache.org) wrote:
>>
>> I really like this and vote to change our current template.
>>
>> The simple yes/no/... options are a really good 

[jira] [Created] (FLINK-7224) Incorrect Javadoc description in all Kafka consumer versions

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-7224:
--

 Summary: Incorrect Javadoc description in all Kafka consumer 
versions
 Key: FLINK-7224
 URL: https://issues.apache.org/jira/browse/FLINK-7224
 Project: Flink
  Issue Type: Improvement
  Components: Kafka Connector
Affects Versions: 1.3.1, 1.4.0
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai


Currently, all Kafka consumer version still have this in the Javadoc:

{code}
The implementation currently accesses partition metadata when the consumer
is constructed. That means that the client that submits the program needs to be 
able to
reach the Kafka brokers or ZooKeeper.
{code}

This is no longer true since Flink 1.3. partition metadata happens only in 
{{open()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)