[
https://issues.apache.org/jira/browse/FLINK-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750256#comment-16750256
]
Piotr Nowojski edited comment on FLINK-11397 at 1/23/19 5:34 PM:
-----------------------------------------------------------------
[~Tison] thanks for reporting the issue. Can you write how did you investigate
it? Don't get me wrong, I'm a big opponent of using mockito for things like
that and it would be very valuable on it's own to replace this {{mockTask =
mock(StreamTask.class)}} with a proper mock or some dummy implementation, but I
couldn't fully reproduce your findings.
I wrote a simple micro benchmark for both {{mock(StreamTask.class)}} and {{new
OneInputStreamOperatorTestHarness()}} constructor and it seemed like first one
takes under 0.01ms to completed. Second one is significantly slower, somewhere
between 0.1 and 1ms, but I couldn't get the more precise number nor I couldn't
investigate/profile the code what's taking so much time because apparently
{{new OneInputStreamOperatorTestHarness()}} is leaking some resources and I was
getting:
{code:java}
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:88)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:69)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:60)
at
org.apache.flink.runtime.operators.testutils.MockEnvironment.<init>(MockEnvironment.java:144)
at
org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder.build(MockEnvironmentBuilder.java:126)
at
org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness.<init>(AbstractStreamOperatorTestHarness.java:164)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:92)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:83)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:75)
{code}
Regardless of that, 1ms is still pretty far away from "a few seconds".
But as I wrote above - even ignoring the speed up I would more then happy to
review/merge the code that gets rid of this mockito mock :)
was (Author: pnowojski):
[~Tison] thanks for reporting the issue. Can you tell how have you investigated
it? Don't get me wrong, I'm a big opponent of using mockito for things like
that and it would be very valuable on it's own to replace this {{mockTask =
mock(StreamTask.class)}} with a proper mock or some dummy implementation, but I
couldn't fully reproduce your findings.
I wrote a simple micro benchmark for both {{mock(StreamTask.class)}} and {{new
OneInputStreamOperatorTestHarness()}} constructor and it seemed like first one
takes under 0.01ms to completed. Second one is significantly slower, somewhere
between 0.1 and 1ms, but I couldn't get the more precise number nor I couldn't
investigate/profile the code what's taking so much time because apparently
{{new OneInputStreamOperatorTestHarness()}} is leaking some resources and I was
getting:
{code:java}
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:88)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:69)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.<init>(IOManagerAsync.java:60)
at
org.apache.flink.runtime.operators.testutils.MockEnvironment.<init>(MockEnvironment.java:144)
at
org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder.build(MockEnvironmentBuilder.java:126)
at
org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness.<init>(AbstractStreamOperatorTestHarness.java:164)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:92)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:83)
at
org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness.<init>(OneInputStreamOperatorTestHarness.java:75)
{code}
Regardless of that, 1ms is still pretty far away from "a few seconds".
But as I wrote above - even ignoring the speed up I would more then happy to
review/merge the code that gets rid of this mockito mock :)
> Speed up initialization of AbstractStreamOperatorTestHarness
> ------------------------------------------------------------
>
> Key: FLINK-11397
> URL: https://issues.apache.org/jira/browse/FLINK-11397
> Project: Flink
> Issue Type: Improvement
> Components: Tests
> Affects Versions: 1.8.0
> Reporter: TisonKun
> Priority: Major
>
> Currently Kafka connector tests are unbearably slow, which is tracked by
> FLINK-10603. With investigation, the construction and initialization of
> {{AbstractStreamOperatorTestHarness}} is quite slow.
> When walk down the code, it amazed me that {{mockTask =
> mock(StreamTask.class);}} cost a few of second to finish. If we can introduce
> a test class instead of mock framework, the situation might be loosen.
> cc [~Zentol] [~pnowojski]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
