Re: Thinking ahead to hadoop-2.6

2014-10-23 Thread Yongjun Zhang
Hi,

I posted a patch to hadoop-10895 couple of days back. Wonder if anyone
could help doing a review?

Thanks a lot.

--Yongjun


On Sun, Oct 19, 2014 at 11:35 AM, Yongjun Zhang  wrote:

> Thanks Arun for the info and reminder.
>
> Hi Alejandro,
>
> HADOOP-10895 is a blocker in the list Arun sent out and it's currently
> owned by you. I wonder whether you will have time to work on. If not, would
> you mind reassigning it to me and I will spend time on it?
>
> Thanks.
>
> --Yongjun
>
>
> On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy 
> wrote:
>
>> 2.6.0 is close now.
>>
>> Here are the remaining blockers, I'm hoping cut an RC in the next week or
>> so:
>> http://s.apache.org/hadoop-2.6.0-blockers
>>
>> thanks,
>> Arun
>>
>> On Sep 30, 2014, at 10:42 AM, Arun C Murthy  wrote:
>>
>> > Folks,
>> >
>> >  I've created branch-2.6 to stabilize the release.
>> >
>> >  Committers, please exercise caution henceforth on commits other than
>> the ones we've discussed on this thread already.
>> >
>> >  By default new features should now be targeted to the version "2.7"
>> henceforth - I've ensure all the projects have that version on jira.
>> >
>> > thanks,
>> > Arun
>> >
>> > On Sep 26, 2014, at 1:08 AM, Arun Murthy  wrote:
>> >
>> >> Sounds good. I'll branch this weekend and we can merge the jiras we
>> >> discussed in this thread as they they get wrapped next week.
>> >>
>> >> Thanks everyone.
>> >>
>> >> Arun
>> >>
>> >>
>> >>> On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org> wrote:
>> >>>
>> >>> We can branch off in a week or two so that work on branch-2 itself
>> can go
>> >>> ahead with other features that can't fit in 2.6. Independent of that,
>> we
>> >>> can then decide on the timeline of the release candidates once
>> branch-2.6
>> >>> is close to being done w.r.t the planned features.
>> >>>
>> >>> Branching it off can let us focus on specific features that we want
>> in for
>> >>> 2.6 and then eventually blockers for the release, nothing else. There
>> is a
>> >>> trivial pain of committing to one more branch, but it's worth it in
>> this
>> >>> case IMO.
>> >>>
>> >>> A lot of efforts are happening in parallel from the YARN side from
>> where I
>> >>> see. 2.6 is a little bulky if only on the YARN side and I'm afraid if
>> we
>> >>> don't branch off and selectively try to get stuff in, it is likely to
>> be in
>> >>> a perpetual delay.
>> >>>
>> >>> My 2 cents.
>> >>>
>> >>> +Vinod
>> >>>
>> >>> On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas <
>> sur...@hortonworks.com>
>> >>> wrote:
>> >>>
>>  Given some of the features are in final stages of stabilization,
>>  Arun, we should hold off creating 2.6 branch or building an RC by a
>> week?
>>  All the features in flux are important ones and worth delaying the
>> release
>>  by a week.
>> 
>>  On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang <
>> andrew.w...@cloudera.com>
>>  wrote:
>> 
>> > Hey Nicholas,
>> >
>> > My concern about Archival Storage isn't related to the code quality
>> or
>>  the
>> > size of the feature. I think that you and Jing did good work. My
>> concern
>>  is
>> > that once we ship, we're locked into that set of archival storage
>> APIs,
>>  and
>> > these APIs are not yet finalized. Simply being able to turn off the
>>  feature
>> > does not change the compatibility story.
>> >
>> > I'm willing to devote time to help review these JIRAs and kick the
>> tires
>>  on
>> > the APIs, but my point above was that I'm not sure it'd all be done
>> by
>>  the
>> > end of the week. Testing might also reveal additional changes that
>> need
>>  to
>> > be made, which also might not happen by end-of-week.
>> >
>> > I guess the question before us is if we're comfortable putting
>> something
>>  in
>> > branch-2.6 and then potentially adding API changes after. I'm okay
>> with
>> > that as long as we're all aware that this might happen.
>> >
>> > Arun, as RM is this cool with you? Again, I like this feature and
>> I'm
>>  fine
>> > with it's inclusion, just a heads up that we might need some extra
>> time
>>  to
>> > finalize things before an RC can be cut.
>> >
>> > Thanks,
>> > Andrew
>> >
>> > On Tue, Sep 23, 2014 at 7:30 PM, Tsz Wo (Nicholas), Sze <
>> > s29752-hadoop...@yahoo.com.invalid> wrote:
>> >
>> >> Hi,
>> >>
>> >> I am worry about KMS and transparent encryption since there are
>> quite
>> > many
>> >> bugs discovered after it got merged to branch-2.  It gives us an
>> > impression
>> >> that the feature is not yet well tested.  Indeed, transparent
>>  encryption
>> > is
>> >> a complicated feature which changes the core part of HDFS.  It is
>> not
>> > easy
>> >> to get everything right.
>> >>
>> >>
>> >> For HDFS-6584: Archival Storage, it is a relatively simple and low
>

[jira] [Created] (HADOOP-11224) Improve error messages for all permission related failures

2014-10-23 Thread Harsh J (JIRA)
Harsh J created HADOOP-11224:


 Summary: Improve error messages for all permission related failures
 Key: HADOOP-11224
 URL: https://issues.apache.org/jira/browse/HADOOP-11224
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0
Reporter: Harsh J
Priority: Trivial


If a bad file create request fails, you get a juicy error self-describing the 
reason almost:

{code}Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
 Permission denied: user=root, access=WRITE, 
inode="/":hdfs:supergroup:drwxr-xr-x{code}

However, if a setPermission fails, one only gets a vague:

{code}Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
 Permission denied{code}

It would be nicer if all forms of permission failures logged the accessed inode 
and current ownership and permissions in the same way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()

2014-10-23 Thread Gopal V (JIRA)
Gopal V created HADOOP-11223:


 Summary: Offer a read-only conf alternative to new Configuration()
 Key: HADOOP-11223
 URL: https://issues.apache.org/jira/browse/HADOOP-11223
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Gopal V


new Configuration() is called from several static blocks across Hadoop.

This is incredibly inefficient, since each one of those involves primarily XML 
parsing at a point where the JIT won't be triggered & interpreter mode is 
essentially forced on the JVM.

The alternate solution would be to offer a {{Configuration::getDefault()}} 
alternative which disallows any modifications.

At the very least, such a method would need to be called from 

# org.apache.hadoop.io.nativeio.NativeIO::()
# org.apache.hadoop.security.SecurityUtil::()
# org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HADOOP-11222) Fix findBugs error in SpanReceiverHost

2014-10-23 Thread Arun Suresh (JIRA)
Arun Suresh created HADOOP-11222:


 Summary: Fix findBugs error in SpanReceiverHost
 Key: HADOOP-11222
 URL: https://issues.apache.org/jira/browse/HADOOP-11222
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Arun Suresh
Priority: Minor


Trivial patch to fix findBugs warning



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HADOOP-11221) JAVA specification for hashcode does not enforce it to be non-negative, but IdentityHashStore assumes System.identityHashCode() is non-negative

2014-10-23 Thread Jinghui Wang (JIRA)
Jinghui Wang created HADOOP-11221:
-

 Summary: JAVA specification for hashcode does not enforce it to be 
non-negative, but IdentityHashStore assumes System.identityHashCode() is 
non-negative
 Key: HADOOP-11221
 URL: https://issues.apache.org/jira/browse/HADOOP-11221
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.4.1
Reporter: Jinghui Wang
Assignee: Jinghui Wang


The following code snippet shows that IdentityHashStore assumes the hashCode is 
always non-negative.

{code:title=Bar.java|borderStyle=solid}
   private void putInternal(Object k, Object v) {
 int hash = System.identityHashCode(k);
 final int numEntries = buffer.length / 2;
 int index = hash % numEntries;
 ...
   }
   
private int getElementIndex(K k) {
 ...
 final int numEntries = buffer.length / 2;
 int hash = System.identityHashCode(k);
 int index = hash % numEntries;
 int firstIndex = index;
 ...
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)