[
https://issues.apache.org/jira/browse/TWILL-63?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Terence Yim updated TWILL-63:
-----------------------------
Description:
Currently when launching an application, two new jars are always created
locally, one for AM (appMaster.jar) and one for Container (container.jar) and
copied to HDFS before submitting the application. The jar files could
potentially be big and if it doesn't changed, it should require copying to HDFS
again.
The general approach is better jar files management and to cache and reuse jar
files created through
class dependency tracing. The changes are further broken down as follows:
# Refactor jars generation
#* One jar containing the TwillLauncher (launcher.jar), created through
dependency tracing.
#** This jar is the same for all applications.
#* One jar containing all twill classes (twill.jar), created through
dependency tracing.
#** This jar is the same for all applications.
#* One jar containing the application class, created through dependency
tracing.
#** This jar is generated based on the application being launched. It is
reusable when launching the same app multiple times.
#* One jar containing user resources setup through TwillPreparer.
#** This jar is not reused between apps.
#* One jar containing runtime config needed by Twill
#** logback.xml, jvm opts, environment, classpaths, ... etc
# Let YARN to expand jars when localizing to containers instead of expanding it
programatically
#* This save time in jar expansion when multiple containers are running on
the same host
# Introduce a new configuration "twill.location.cache.dir" to enable jar
caching and reuse
#* Currently only the launcher.jar, twill.jar and application jar will be
cached and reuse when possible
#* Cache cleanup logic is also in place to remove files in cache directory
that is no longer used by application
# The ApplicationBundler is improved to allow more flexible usage
was:Currently when launching an application, two new jars are always created
locally, one for AM (appMaster.jar) and one for Container (container.jar) and
copied to HDFS before submitting the application. The jar files could
potentially be big and if it doesn't changed, it should require copying to HDFS
again.
> Speed up application launch time
> --------------------------------
>
> Key: TWILL-63
> URL: https://issues.apache.org/jira/browse/TWILL-63
> Project: Apache Twill
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 0.2.0-incubating
> Reporter: Terence Yim
> Assignee: Terence Yim
> Fix For: 0.10.0
>
>
> Currently when launching an application, two new jars are always created
> locally, one for AM (appMaster.jar) and one for Container (container.jar) and
> copied to HDFS before submitting the application. The jar files could
> potentially be big and if it doesn't changed, it should require copying to
> HDFS again.
> The general approach is better jar files management and to cache and reuse
> jar files created through
> class dependency tracing. The changes are further broken down as follows:
> # Refactor jars generation
> #* One jar containing the TwillLauncher (launcher.jar), created through
> dependency tracing.
> #** This jar is the same for all applications.
> #* One jar containing all twill classes (twill.jar), created through
> dependency tracing.
> #** This jar is the same for all applications.
> #* One jar containing the application class, created through dependency
> tracing.
> #** This jar is generated based on the application being launched. It is
> reusable when launching the same app multiple times.
> #* One jar containing user resources setup through TwillPreparer.
> #** This jar is not reused between apps.
> #* One jar containing runtime config needed by Twill
> #** logback.xml, jvm opts, environment, classpaths, ... etc
> # Let YARN to expand jars when localizing to containers instead of expanding
> it programatically
> #* This save time in jar expansion when multiple containers are running on
> the same host
> # Introduce a new configuration "twill.location.cache.dir" to enable jar
> caching and reuse
> #* Currently only the launcher.jar, twill.jar and application jar will be
> cached and reuse when possible
> #* Cache cleanup logic is also in place to remove files in cache directory
> that is no longer used by application
> # The ApplicationBundler is improved to allow more flexible usage
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
