[jira] [Updated] (BEAM-1146) Decrease spark runner startup overhead
[
https://issues.apache.org/jira/browse/BEAM-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aviem Zur updated BEAM-1146:
Description:
BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver &
executors) which utilizes reflection to find all subclasses of Source and Coder
While this is beneficial in its own right, the change added about one minute of
overhead in spark runner startup time (which cause the first job/stage to take
up to a minute).
The change is in class {{BeamSparkRunnerRegistrator}}.
was:
BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver &
executors) which utilizes reflection to find all subclasses of Source and Coder
While this is beneficial in it's own right, the change added about one minute
of overhead in spark runner startup time (which cause the first job/stage to
take up to a minute).
The change is in class {{BeamSparkRunnerRegistrator}}
The reason reflection (specifically reflections library) was used here is
because there is no current way of knowing all the source and coder classes at
runtime.
> Decrease spark runner startup overhead
> --
>
> Key: BEAM-1146
> URL: https://issues.apache.org/jira/browse/BEAM-1146
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
> Fix For: 0.5.0
>
>
> BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver &
> executors) which utilizes reflection to find all subclasses of Source and
> Coder
> While this is beneficial in its own right, the change added about one minute
> of overhead in spark runner startup time (which cause the first job/stage to
> take up to a minute).
> The change is in class {{BeamSparkRunnerRegistrator}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (BEAM-1146) Decrease spark runner startup overhead
[
https://issues.apache.org/jira/browse/BEAM-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aviem Zur updated BEAM-1146:
Description:
BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver &
executors) which utilizes reflection to find all subclasses of Source and Coder
While this is beneficial in it's own right, the change added about one minute
of overhead in spark runner startup time (which cause the first job/stage to
take up to a minute).
The change is in class {{BeamSparkRunnerRegistrator}}
The reason reflection (specifically reflections library) was used here is
because there is no current way of knowing all the source and coder classes at
runtime.
was:
BEAM-921 introduced a lazy singleton instantiated once in each machine (driver
& executors) which utilizes reflection to find all subclasses of Source and
Coder
While this is beneficial in it's own right, the change added about one minute
of overhead in spark runner startup time (which cause the first job/stage to
take up to a minute).
The change is in class {{BeamSparkRunnerRegistrator}}
The reason reflection (specifically reflections library) was used here is
because there is no current way of knowing all the source and coder classes at
runtime.
> Decrease spark runner startup overhead
> --
>
> Key: BEAM-1146
> URL: https://issues.apache.org/jira/browse/BEAM-1146
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Aviem Zur
>Assignee: Aviem Zur
> Fix For: 0.5.0
>
>
> BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver &
> executors) which utilizes reflection to find all subclasses of Source and
> Coder
> While this is beneficial in it's own right, the change added about one minute
> of overhead in spark runner startup time (which cause the first job/stage to
> take up to a minute).
> The change is in class {{BeamSparkRunnerRegistrator}}
> The reason reflection (specifically reflections library) was used here is
> because there is no current way of knowing all the source and coder classes
> at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
