[
https://issues.apache.org/jira/browse/SLIDER-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gour Saha updated SLIDER-1090:
------------------------------
Description:
Today both SliderClient and SliderAM read package metainfo from HDFS. This
requires downloading the whole package from HDFS. If package size is big
(~200MB), this operation wastes tens of seconds in starting up the application.
To speed up the operation of reading metainfo, we can read from an summary
xml/json file if it exists in the same HDFS folder as the zip package. If the
summary file does not exist, it falls back to the old way of downloading the
zip package.
was:
Today both SliderClient and SliderAM read package metainfo from HDFS. This
requires downloading the whole package from HDFS. If package size is big
(~200MB), this operation wastes tens of seconds in starting up the application.
To speed up the operation of reading metainfo, we can read from an accompany
xml/json file if it exists in the same HDFS folder as the zip package. If the
accompany file does not exist, it falls back to the old way of downloading the
zip package.
> Speed up SliderClient and AM by reading metainfo from summary file
> ------------------------------------------------------------------
>
> Key: SLIDER-1090
> URL: https://issues.apache.org/jira/browse/SLIDER-1090
> Project: Slider
> Issue Type: Bug
> Affects Versions: Slider 0.81, Slider 0.90.2
> Reporter: shanyu zhao
> Assignee: shanyu zhao
> Attachments: SLIDER-1090.1.patch, SLIDER-1090.patch
>
>
> Today both SliderClient and SliderAM read package metainfo from HDFS. This
> requires downloading the whole package from HDFS. If package size is big
> (~200MB), this operation wastes tens of seconds in starting up the
> application.
> To speed up the operation of reading metainfo, we can read from an summary
> xml/json file if it exists in the same HDFS folder as the zip package. If the
> summary file does not exist, it falls back to the old way of downloading the
> zip package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
