I've submitted Wordcount 410 times to a testing cluster and a streaming job
290 times and I could not reproduce the issue with 1.1.3. Also, the
heapdump of one of the TaskManagers looked pretty normal.

Do you have any ideas how to reproduce the issue?

On Fri, Dec 2, 2016 at 3:21 PM, Robert Metzger <rmetz...@apache.org> wrote:

> Thank you for reporting the issue Konstantin.
> I've filed a JIRA for the jackson issue: https://issues.apache.o
> rg/jira/browse/FLINK-5233.
> As I said in the JIRA, I propose to upgrade to Jackson 2.7.8, as this
> version contains the fix for the issue, but its not a major jackson upgrade.
>
> Any chance you could try to if 2.7.8 fixes the issue as well?
>
>
> On Fri, Dec 2, 2016 at 11:12 AM, Fabian Hueske <fhue...@gmail.com> wrote:
>
>> Hi Konstantin,
>>
>> Regarding 2): I've opened FLINK-5227 to update the documentation [1].
>>
>> Regarding the Row type: The Row type was introduced for flink-table and
>> was later used by other modules. There is FLINK-5186 to move Row and all
>> the related TypeInfo (+serializer and comparator) to flink-core [2]. That
>> should solve your issue.
>>
>> Some of the connector modules which provide TableSource and TableSinks
>> have dependencies on flink-table as well. I'll check that these are
>> optional dependencies to avoid that we pull in Calcite through connectors
>> for jobs that do not not need it.
>>
>> Thanks,
>> Fabian
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-5227
>> [2] https://issues.apache.org/jira/browse/FLINK-5186
>>
>> 2016-11-30 17:51 GMT+01:00 Konstantin Knauf <konstantin.kn...@tngtech.com
>> >:
>>
>>> Hi Stefan,
>>>
>>> unfortunately, I can not share any heap dumps with you. I was able to
>>> resolve some of the issues my self today, the root causes were different
>>> for different jobs.
>>>
>>> 1) Jackson 2.7.2 (which comes with Flink) has a known class loading
>>> issue (see https://github.com/FasterXML/jackson-databind/issues/1363).
>>> Shipping a shaded version of Jackson 2.8.4 with our user code helped. I
>>> recommend upgrading Flink's Jackson version soon.
>>>
>>> 2) We have a dependency on the flink-table [1] , which ships with
>>> Calcite including the Calcite JDBC Driver, which can not been collected
>>> cause of the known problem with the java.sql.DriverManager. Putting the
>>> flink-table in Flink's lib dir instead of shipping it with the user code
>>> helps. You should update the documentation, because this will always
>>> happen when using flink-table, I think. So I wonder, why this has not
>>> come up before actually.
>>>
>>> 3) Unresolved: Some Threads in a custom source which are not proberly
>>> shut down and keep references to the UserCodeClassLoader. I did not have
>>> time to look into this issue so far.
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>> [1] Side note: We only need flink-table for the "Row" class used in the
>>> JdbcOutputFormat, so it might make sense to move this class somewhere
>>> else. Naturally, we also tried to exclude the "transitive" dependency on
>>> org.apache.calcite until we noticed that calcite is packaged with
>>> flink-table, so that you can not even exclude it. What is the reasons
>>> for this?
>>>
>>>
>>>
>>>
>>> On 30.11.2016 00:55, Stefan Richter wrote:
>>> > Hi,
>>> >
>>> > could you somehow provide us a heap dump from a TM that run for a
>>> while (ideally, shortly before an OOME)? This would greatly help us to
>>> figure out if there is a classloader leak that causes the problem.
>>> >
>>> > Best,
>>> > Stefan
>>> >
>>> >> Am 29.11.2016 um 18:39 schrieb Konstantin Knauf <
>>> konstantin.kn...@tngtech.com>:
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> since upgrading to Flink 1.1.3 we observe frequent OOME Permgen
>>> Taskmanager Failures. Monitoring the permgen size on one of the
>>> Taskamanagers you can see that each Job (New Job and Restarts) adds a few
>>> MB, which can not be collected. Eventually, the OOME happens. This happens
>>> with all our Jobs, Streaming and Batch, on Yarn 2.4 as well as Stand-Alone.
>>> >>
>>> >> On Flink 1.0.2 this was not a problem, but I will investigate it
>>> further.
>>> >>
>>> >> The assumption is that Flink is somehow using one of the classes,
>>> which comes with our jar and by that prevents the gc of the whole class
>>> loader. Our Jars do not include any flink dependencies though
>>> (compileOnly), but of course many others.
>>> >>
>>> >> Any ideas anyone?
>>> >>
>>> >> Cheers and thank you,
>>> >>
>>> >> Konstantin
>>> >>
>>> >> sent from my phone. Plz excuse brevity and tpyos.
>>> >> ---
>>> >> Konstantin Knauf *konstantin.kn...@tngtech.com * +49-174-3413182
>>> >> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>>> >> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>>> >
>>> >
>>>
>>> --
>>> Konstantin Knauf * konstantin.kn...@tngtech.com * +49-174-3413182
>>> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>>> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>>> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>>>
>>>
>>
>

Reply via email to