The HDFS-6581 branch has been merged into trunk.
On Tue, Sep 30, 2014 at 2:30 PM, Arpit Agarwal
wrote:
> HDFS Devs,
>
> The vote passes with 4 +1s (Jitendra, Jing, Suresh and mine), two -0s and
> no binding vetoes. The branch will be merged to trunk by tomorrow.
>
> Thank you to everyone who vo
HDFS Devs,
The vote passes with 4 +1s (Jitendra, Jing, Suresh and mine), two -0s and
no binding vetoes. The branch will be merged to trunk by tomorrow.
Thank you to everyone who voted and provided feedback. The changes will be
merged to branch-2 when we have a resolution for the merge blocker rai
> I know Arpit had prototyped short-circuit writes a while
> ago, and SCW has the potential to take us well past 39% gains. However,
> apparently there are no future plans in this direction.
And to give credit where it is due, the prototype work was done by Vikram
Dixit and Jing Zhao. As you might
Andrew, I think you are reading hostility or lack of warmth where it wasn't
intended. I apologize if it came through in my responses to you. Please
consider that it could be due to the limitations of communicating over
email.
Your technical feedback is always welcome and I thank you for it again,
-0.
The goal of this feature is to improve write performance. I thought it was
thus reasonable for me to ask for write benchmarks. However, this request
was not received warmly. For a performance oriented feature, I expected
that benchmarks would have been run both to guide the design and evaluate
> I implemented a modified 2Q today on HDFS-7142... I'd appreciate a
> review.
Great, thank you. I'll review it later today.
On Wed, Sep 24, 2014 at 6:23 PM, Colin McCabe
wrote:
> On Wed, Sep 24, 2014 at 4:03 PM, Arpit Agarwal
> wrote:
> >> I would appreciate in the future if benchmarks were p
On Wed, Sep 24, 2014 at 4:03 PM, Arpit Agarwal wrote:
>> I would appreciate in the future if benchmarks were posted
>> a day or two before a merge vote of a performance improvement was
>> suggested.
>> I feel that it would have been better to float the idea of a
>> merge on the JIRA before actuall
> I would appreciate in the future if benchmarks were posted
> a day or two before a merge vote of a performance improvement was
> suggested.
> I feel that it would have been better to float the idea of a
> merge on the JIRA before actually calling it,
Colin, I noted my intention to start the merg
On Wed, Sep 24, 2014 at 2:19 PM, Colin McCabe
wrote:
> On Wed, Sep 24, 2014 at 11:12 AM, Suresh Srinivas
> wrote:
> > Features are done in a separate branch for two reasons: 1) During a
> feature
> > development the branch may be not functional 2) The high level approach
> and
> > design is not
On Wed, Sep 24, 2014 at 11:12 AM, Suresh Srinivas
wrote:
> Features are done in a separate branch for two reasons: 1) During a feature
> development the branch may be not functional 2) The high level approach and
> design is not very clear and development can continue while that is being
> sorted
I would consider following as blockers to merge to trunk:
a) Any bugs that makes the feature unusable or cause instability or cause
any regression.
b) Any bad design choices that add technical debt.
c) Any missing functionality, that makes the feature useless or difficult
to use for its intended u
I am +1 on the merge.
On Wed, Sep 24, 2014 at 11:12 AM, Suresh Srinivas
wrote:
> Features are done in a separate branch for two reasons: 1) During a
> feature development the branch may be not functional 2) The high level
> approach and design is not very clear and development can continue while
Features are done in a separate branch for two reasons: 1) During a feature
development the branch may be not functional 2) The high level approach and
design is not very clear and development can continue while that is being
sorted out. In case of this feature, clearly (2) is not an issue. We have
Colin, the feature should be evaluated on its technical merits and not on
the speed of development. The high level approach was decided during the
community hangout on Apr 30 - five months ago - which you attended. Nothing
we have developed is contrary to what was decided that day.
We have discuss
This seems like a really aggressive timeframe for a merge. We still
haven't implemented:
* Checksum skipping on read and write from lazy persisted replicas.
* Allowing mmaped reads from the lazy persisted data.
* Any eviction strategy other than LRU.
* Integration with cache pool limits (how do H
+1. This is a very useful feature and the performance number looks
convincing.
Thanks,
-Jing
On Tue, Sep 23, 2014 at 5:12 PM, Xiaoyu Yao wrote:
> +1 non-binding. I’ve reviewed most of the changes in the branch and the
> E2E perf numbers looks good.
>
> Thanks,
> Xiaoyu
>
> On Sep 23, 2014, at 4
+1 non-binding. I’ve reviewed most of the changes in the branch and the E2E
perf numbers looks good.
Thanks,
Xiaoyu
On Sep 23, 2014, at 4:55 PM, Arpit Agarwal wrote:
> I have posted write benchmark results to the Jira.
>
> On Tue, Sep 23, 2014 at 3:41 PM, Arpit Agarwal
> wrote:
>
>> Hi Andr
I have posted write benchmark results to the Jira.
On Tue, Sep 23, 2014 at 3:41 PM, Arpit Agarwal
wrote:
> Hi Andrew, I said "it is not going to be a substantial fraction of memory
> bandwidth". That is certainly not the same as saying it won't be good or
> there won't be any improvement.
>
> An
Hi Andrew, I said "it is not going to be a substantial fraction of memory
bandwidth". That is certainly not the same as saying it won't be good or
there won't be any improvement.
Any time you have transfers over RPC or the network stack you will not get
close to the memory bandwidth even for intra
Hi Arpit,
Here is the comment. It was certainly not my intention to misquote anyone.
https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14138223&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14138223
Quote:
It would be nice to see that would could g
Andrew, don't misquote me. Can you link the comment where I said
performance wasn't going to be good?
I will add some add some preliminary write results to the Jira later today.
> What's the plan to improve write performance?
I described this in response to your and Colin's comments on the Jira.
Hi Arpit,
On HDFS-6581, I asked for write benchmarks on Sep 19th, and you responded
that the performance wasn't going to be good. However, I thought the
primary goal of this JIRA was to improve write performance, and write
performance is listed as the first feature requirement in the design doc.
+1. I have reviewed most of the code in the branch, and I think its ready
to be merged to trunk.
On Mon, Sep 22, 2014 at 5:24 PM, Arpit Agarwal
wrote:
> HDFS Devs,
>
> We propose merging the HDFS-6581 development branch to trunk.
>
> The work adds support to write to HDFS blocks in memory. The
HDFS Devs,
We propose merging the HDFS-6581 development branch to trunk.
The work adds support to write to HDFS blocks in memory. The target use
case covers applications writing relatively small, intermediate data sets
with low latency. We introduce a new CreateFlag for the existing CreateFile
AP
24 matches
Mail list logo
