git line endings trouble since recent trunk

[Sandy Ryza]

Has anybody else been having trouble with line endings since pulling trunk
recently?

hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
shows up as modified even though I haven't touched it, and I can't check it
out or reset to a previous version to make that go away.  The only thing I
can do to neutralize it is to put it in a dummy commit, but I have to do
this every time I switch branches or rebase.

This appears to have began after the release notes commit
 (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
endings change.

-Sandy

Re: git line endings trouble since recent trunk

[Zhijie Shen]

Have the same problem here, have to edit the patch manually to exclude the
changes in releasenotes.html


On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza wrote:

> Has anybody else been having trouble with line endings since pulling trunk
> recently?
>
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
> shows up as modified even though I haven't touched it, and I can't check it
> out or reset to a previous version to make that go away.  The only thing I
> can do to neutralize it is to put it in a dummy commit, but I have to do
> this every time I switch branches or rebase.
>
> This appears to have began after the release notes commit
>  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
> endings change.
>
> -Sandy
>



-- 
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Hitesh Shah]

Hi Arun, 

From a YARN perspective, YARN-791 and YARN-727 are 2 jiras that may potentially 
change the apis. They can implemented in a backward compat fashion if committed 
after 2.1.0. However, this will require adding of differently-named apis ( 
different urls in case of the webservices ) and make the current version of the 
api deprecated and/or obsolete. YARN-818 which is currently patch available 
also changes behavior.  

Assuming that as soon as 2.1.0 is released, we are to follow a very strict 
backward-compat retaining approach to all user-facing layers  ( 
api/webservices/rpc/... ) in common/hdfs/yarn/mapreduce, does it make sense to 
try and pull them in and roll out a new RC after they are ready? Perhaps Vinod 
can chime in if he is aware of any other such jiras under YARN-386 which should 
be considered compat-related blockers for a 2.1.0 RC. 

thanks
-- Hitesh

On Jun 26, 2013, at 1:17 AM, Arun C Murthy wrote:

> Folks,
> 
> I've created a release candidate (rc0) for hadoop-2.1.0-beta that I would 
> like to get released.
> 
> This release represents a *huge* amount of work done by the community (639 
> fixes) which includes several major advances including:
> # HDFS Snapshots
> # Windows support
> # YARN API stabilization
> # MapReduce Binary Compatibility with hadoop-1.x
> # Substantial amount of integration testing with rest of projects in the 
> ecosystem
> 
> The RC is available at: 
> http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/
> The RC tag in svn is here: 
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0
> 
> The maven artifacts are available via repository.apache.org.
> 
> Please try the release and vote; the vote will run for the usual 7 days.
> 
> thanks,
> Arun
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Sandy Ryza]

For YARN-791, if we can come to consensus on the correct approach, I can
try to have a patch ASAP.

-Sandy


On Fri, Jun 28, 2013 at 12:03 PM, Hitesh Shah  wrote:

> Hi Arun,
>
> From a YARN perspective, YARN-791 and YARN-727 are 2 jiras that may
> potentially change the apis. They can implemented in a backward compat
> fashion if committed after 2.1.0. However, this will require adding of
> differently-named apis ( different urls in case of the webservices ) and
> make the current version of the api deprecated and/or obsolete. YARN-818
> which is currently patch available also changes behavior.
>
> Assuming that as soon as 2.1.0 is released, we are to follow a very strict
> backward-compat retaining approach to all user-facing layers  (
> api/webservices/rpc/... ) in common/hdfs/yarn/mapreduce, does it make sense
> to try and pull them in and roll out a new RC after they are ready? Perhaps
> Vinod can chime in if he is aware of any other such jiras under YARN-386
> which should be considered compat-related blockers for a 2.1.0 RC.
>
> thanks
> -- Hitesh
>
> On Jun 26, 2013, at 1:17 AM, Arun C Murthy wrote:
>
> > Folks,
> >
> > I've created a release candidate (rc0) for hadoop-2.1.0-beta that I
> would like to get released.
> >
> > This release represents a *huge* amount of work done by the community
> (639 fixes) which includes several major advances including:
> > # HDFS Snapshots
> > # Windows support
> > # YARN API stabilization
> > # MapReduce Binary Compatibility with hadoop-1.x
> > # Substantial amount of integration testing with rest of projects in the
> ecosystem
> >
> > The RC is available at:
> http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/
> > The RC tag in svn is here:
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0
> >
> > The maven artifacts are available via repository.apache.org.
> >
> > Please try the release and vote; the vote will run for the usual 7 days.
> >
> > thanks,
> > Arun
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
>

Re: git line endings trouble since recent trunk

[Omkar Joshi]

Sandy... did you fix this? any jira to track? me too facing same problem..

Thanks,
Omkar Joshi
*Hortonworks Inc.* 


On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen  wrote:

> Have the same problem here, have to edit the patch manually to exclude the
> changes in releasenotes.html
>
>
> On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza  >wrote:
>
> > Has anybody else been having trouble with line endings since pulling
> trunk
> > recently?
> >
> > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
> > shows up as modified even though I haven't touched it, and I can't check
> it
> > out or reset to a previous version to make that go away.  The only thing
> I
> > can do to neutralize it is to put it in a dummy commit, but I have to do
> > this every time I switch branches or rebase.
> >
> > This appears to have began after the release notes commit
> >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
> > endings change.
> >
> > -Sandy
> >
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>

[jira] [Created] (HADOOP-9675) releasenotes.html always shows up as modified because of line endings issues

[Sandy Ryza (JIRA)]

Sandy Ryza created HADOOP-9675:
--

 Summary: releasenotes.html always shows up as modified because of 
line endings issues
 Key: HADOOP-9675
 URL: https://issues.apache.org/jira/browse/HADOOP-9675
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza


hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
shows up as modified even though I haven't touched it, and I can't check it out 
or reset to a previous version to make that go away.  The only thing I can do 
to neutralize it is to put it in a dummy commit, but I have to do this every 
time I switch branches or rebase.

This appears to have began after the release notes commit  
(8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line endings 
change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Roman Shaposhnik]

I think it would very useful if somebody more familiar with HDFS
(Suresh?) could take a look at:
 https://issues.apache.org/jira/browse/HDFS-4940

Suresh was very helpful in unblocking the client side of things, but
even after I followed his recommendations on the heap size, etc.
The NN still OOMs it just takes longer.

A quick look at the provided heap dump could help us understand
whether this is something serious or not.

This is the last bit that blocks the Bigtop side of things as far as
2.1 is concerned.

Thanks,
Roman.

On Wed, Jun 26, 2013 at 1:17 AM, Arun C Murthy  wrote:
> Folks,
>
> I've created a release candidate (rc0) for hadoop-2.1.0-beta that I would 
> like to get released.
>
> This release represents a *huge* amount of work done by the community (639 
> fixes) which includes several major advances including:
> # HDFS Snapshots
> # Windows support
> # YARN API stabilization
> # MapReduce Binary Compatibility with hadoop-1.x
> # Substantial amount of integration testing with rest of projects in the 
> ecosystem
>
> The RC is available at: 
> http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/
> The RC tag in svn is here: 
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0
>
> The maven artifacts are available via repository.apache.org.
>
> Please try the release and vote; the vote will run for the usual 7 days.
>
> thanks,
> Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>

Re: git line endings trouble since recent trunk

[Sandy Ryza]

I haven't been able to find a solution.  Just filed
https://issues.apache.org/jira/browse/HADOOP-9675.


On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi wrote:

> Sandy... did you fix this? any jira to track? me too facing same problem..
>
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* 
>
>
> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
> wrote:
>
> > Have the same problem here, have to edit the patch manually to exclude
> the
> > changes in releasenotes.html
> >
> >
> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza  > >wrote:
> >
> > > Has anybody else been having trouble with line endings since pulling
> > trunk
> > > recently?
> > >
> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
> > > shows up as modified even though I haven't touched it, and I can't
> check
> > it
> > > out or reset to a previous version to make that go away.  The only
> thing
> > I
> > > can do to neutralize it is to put it in a dummy commit, but I have to
> do
> > > this every time I switch branches or rebase.
> > >
> > > This appears to have began after the release notes commit
> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
> > > endings change.
> > >
> > > -Sandy
> > >
> >
> >
> >
> > --
> > Zhijie Shen
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
>

Re: git line endings trouble since recent trunk

[Colin McCabe]

I think the fix for this is to set svn:eol-style to "native" on this
file.  It's set on many other files, just not on this one:

cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
./hadoop-project-dist/README.txt
native
cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
cmccabe@keter:~/hadoopST/trunk>

It might also be a good idea to run dos2unix on it.

I thought that in general we wanted to have 'LF' everywhere, so
perhaps we should add this to the patch.sh script to prevent this from
re-occurring.

Colin


On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza  wrote:
> I haven't been able to find a solution.  Just filed
> https://issues.apache.org/jira/browse/HADOOP-9675.
>
>
> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi wrote:
>
>> Sandy... did you fix this? any jira to track? me too facing same problem..
>>
>> Thanks,
>> Omkar Joshi
>> *Hortonworks Inc.* 
>>
>>
>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
>> wrote:
>>
>> > Have the same problem here, have to edit the patch manually to exclude
>> the
>> > changes in releasenotes.html
>> >
>> >
>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza > > >wrote:
>> >
>> > > Has anybody else been having trouble with line endings since pulling
>> > trunk
>> > > recently?
>> > >
>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
>> > > shows up as modified even though I haven't touched it, and I can't
>> check
>> > it
>> > > out or reset to a previous version to make that go away.  The only
>> thing
>> > I
>> > > can do to neutralize it is to put it in a dummy commit, but I have to
>> do
>> > > this every time I switch branches or rebase.
>> > >
>> > > This appears to have began after the release notes commit
>> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
>> > > endings change.
>> > >
>> > > -Sandy
>> > >
>> >
>> >
>> >
>> > --
>> > Zhijie Shen
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>>

Re: git line endings trouble since recent trunk

[Colin McCabe]

Clarification: svn:eol-style = native causes the files to contain
whatever the native platform used to check out the code uses.  I think
just setting this property on all the HTML files should resolve this
and future problems.

patch posted.
C.

On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe  wrote:
> I think the fix for this is to set svn:eol-style to "native" on this
> file.  It's set on many other files, just not on this one:
>
> cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> ./hadoop-project-dist/README.txt
> native
> cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
> cmccabe@keter:~/hadoopST/trunk>
>
> It might also be a good idea to run dos2unix on it.
>
> I thought that in general we wanted to have 'LF' everywhere, so
> perhaps we should add this to the patch.sh script to prevent this from
> re-occurring.
>
> Colin
>
>
> On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza  wrote:
>> I haven't been able to find a solution.  Just filed
>> https://issues.apache.org/jira/browse/HADOOP-9675.
>>
>>
>> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi wrote:
>>
>>> Sandy... did you fix this? any jira to track? me too facing same problem..
>>>
>>> Thanks,
>>> Omkar Joshi
>>> *Hortonworks Inc.* 
>>>
>>>
>>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
>>> wrote:
>>>
>>> > Have the same problem here, have to edit the patch manually to exclude
>>> the
>>> > changes in releasenotes.html
>>> >
>>> >
>>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza >> > >wrote:
>>> >
>>> > > Has anybody else been having trouble with line endings since pulling
>>> > trunk
>>> > > recently?
>>> > >
>>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
>>> > > shows up as modified even though I haven't touched it, and I can't
>>> check
>>> > it
>>> > > out or reset to a previous version to make that go away.  The only
>>> thing
>>> > I
>>> > > can do to neutralize it is to put it in a dummy commit, but I have to
>>> do
>>> > > this every time I switch branches or rebase.
>>> > >
>>> > > This appears to have began after the release notes commit
>>> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
>>> > > endings change.
>>> > >
>>> > > -Sandy
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Zhijie Shen
>>> > Hortonworks Inc.
>>> > http://hortonworks.com/
>>> >
>>>

[jira] [Created] (HADOOP-9676) make maximum RPC buffer size configurable

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HADOOP-9676:


 Summary: make maximum RPC buffer size configurable
 Key: HADOOP-9676
 URL: https://issues.apache.org/jira/browse/HADOOP-9676
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor


Currently the RPC server just allocates however much memory the client asks 
for, without validating.  It would be nice to make the maximum RPC buffer size 
configurable.  This would prevent a rogue client from bringing down the 
NameNode (or other Hadoop daemon) with a few requests for 2 GB buffers.  It 
would also make it easier to debug issues with super-large RPCs or malformed 
headers, since OOMs can be difficult for developers to reproduce.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Arun C Murthy]

Thanks Hitesh & Roman, I'll roll RC1 once these are fixed.

Arun

On Jun 28, 2013, at 12:25 PM, Roman Shaposhnik  wrote:

> I think it would very useful if somebody more familiar with HDFS
> (Suresh?) could take a look at:
> https://issues.apache.org/jira/browse/HDFS-4940
> 
> Suresh was very helpful in unblocking the client side of things, but
> even after I followed his recommendations on the heap size, etc.
> The NN still OOMs it just takes longer.
> 
> A quick look at the provided heap dump could help us understand
> whether this is something serious or not.
> 
> This is the last bit that blocks the Bigtop side of things as far as
> 2.1 is concerned.
> 
> Thanks,
> Roman.
> 
> On Wed, Jun 26, 2013 at 1:17 AM, Arun C Murthy  wrote:
>> Folks,
>> 
>> I've created a release candidate (rc0) for hadoop-2.1.0-beta that I would 
>> like to get released.
>> 
>> This release represents a *huge* amount of work done by the community (639 
>> fixes) which includes several major advances including:
>> # HDFS Snapshots
>> # Windows support
>> # YARN API stabilization
>> # MapReduce Binary Compatibility with hadoop-1.x
>> # Substantial amount of integration testing with rest of projects in the 
>> ecosystem
>> 
>> The RC is available at: 
>> http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/
>> The RC tag in svn is here: 
>> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0
>> 
>> The maven artifacts are available via repository.apache.org.
>> 
>> Please try the release and vote; the vote will run for the usual 7 days.
>> 
>> thanks,
>> Arun
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: git line endings trouble since recent trunk

[Doug Cutting]

http://www.apache.org/dev/version-control.html#https-svn-config

Doug
On Jun 28, 2013 1:03 PM, "Colin McCabe"  wrote:

> Clarification: svn:eol-style = native causes the files to contain
> whatever the native platform used to check out the code uses.  I think
> just setting this property on all the HTML files should resolve this
> and future problems.
>
> patch posted.
> C.
>
> On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe 
> wrote:
> > I think the fix for this is to set svn:eol-style to "native" on this
> > file.  It's set on many other files, just not on this one:
> >
> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> > ./hadoop-project-dist/README.txt
> > native
> > cmccabe@keter:~/hadoopST/trunk> svn propget svn:eol-style
> > ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
> > cmccabe@keter:~/hadoopST/trunk>
> >
> > It might also be a good idea to run dos2unix on it.
> >
> > I thought that in general we wanted to have 'LF' everywhere, so
> > perhaps we should add this to the patch.sh script to prevent this from
> > re-occurring.
> >
> > Colin
> >
> >
> > On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza 
> wrote:
> >> I haven't been able to find a solution.  Just filed
> >> https://issues.apache.org/jira/browse/HADOOP-9675.
> >>
> >>
> >> On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi  >wrote:
> >>
> >>> Sandy... did you fix this? any jira to track? me too facing same
> problem..
> >>>
> >>> Thanks,
> >>> Omkar Joshi
> >>> *Hortonworks Inc.* 
> >>>
> >>>
> >>> On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen 
> >>> wrote:
> >>>
> >>> > Have the same problem here, have to edit the patch manually to
> exclude
> >>> the
> >>> > changes in releasenotes.html
> >>> >
> >>> >
> >>> > On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza <
> sandy.r...@cloudera.com
> >>> > >wrote:
> >>> >
> >>> > > Has anybody else been having trouble with line endings since
> pulling
> >>> > trunk
> >>> > > recently?
> >>> > >
> >>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
> >>> > > shows up as modified even though I haven't touched it, and I can't
> >>> check
> >>> > it
> >>> > > out or reset to a previous version to make that go away.  The only
> >>> thing
> >>> > I
> >>> > > can do to neutralize it is to put it in a dummy commit, but I have
> to
> >>> do
> >>> > > this every time I switch branches or rebase.
> >>> > >
> >>> > > This appears to have began after the release notes commit
> >>> > >  (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a
> line
> >>> > > endings change.
> >>> > >
> >>> > > -Sandy
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Zhijie Shen
> >>> > Hortonworks Inc.
> >>> > http://hortonworks.com/
> >>> >
> >>>
>