[jira] [Created] (HADOOP-10275) Serialization should remove its type parameter

2014-01-24 Thread Hiroshi Ikeda (JIRA) 
Hiroshi Ikeda created HADOOP-10275:
--

 Summary: Serialization should remove its type parameter
 Key: HADOOP-10275
 URL: https://issues.apache.org/jira/browse/HADOOP-10275
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Hiroshi Ikeda
Priority: Minor


org.apache.hadoop.io.serializer.Serialization is defined as:
{code}
public interface SerializationT {
...
SerializerT getSerializer(ClassT c);
DeserializerT getDeserializer(ClassT c);
}
{code}
but the type parameter T is semantically invalid, and type mismatchings in 
the code are suppressed by explicit cast and annotations.

This interface should be defined as follows:
{code}
public interface Serialization {
...
T SerializerT getSerializer(ClassT c);
T DeserializerT getDeserializer(ClassT c);
}
{code}




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10276) CLONE - RawLocalFs#getFileLinkStatus does not fill in the link owner and mode

2014-01-24 Thread Jason Lowe (JIRA) 
Jason Lowe created HADOOP-10276:
---

 Summary: CLONE - RawLocalFs#getFileLinkStatus does not fill in the 
link owner and mode
 Key: HADOOP-10276
 URL: https://issues.apache.org/jira/browse/HADOOP-10276
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
 Fix For: 2.4.0


{{RawLocalFs#getFileLinkStatus}} does not actually get the owner and mode of 
the symlink, but instead uses the owner and mode of the symlink target.  If the 
target can't be found, it fills in bogus values (the empty string and 
FsPermission.getDefault) for these.

Symlinks have an owner distinct from the owner of the target they point to, and 
getFileLinkStatus ought to expose this.

In some operating systems, symlinks can have a permission other than 0777.  We 
ought to expose this in RawLocalFilesystem and other places, although we don't 
necessarily have to support this behavior in HDFS.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Logistics for releasing 2.4

2014-01-24 Thread Jason Lowe 
I just committed the addendum patch for HADOOP-9652 which should resolve 
the performance issue.


Speaking of critical issues to fix for 2.3, I do wonder what to do with 
all the Blocker/Criticals targeted for 2.4 (the old branch-2 HEAD 
release) which apparently is now going to be 2.3 (the new branch-2 HEAD 
release).  We should either get these resolved before the 2.3 release or 
decide which ones aren't really blockers or that critical for 2.3 and 
target them for a future release (e.g.: 2.4). According to my 
calculations, there's currently 24 JIRAs that need to be scrubbed 
accordingly.


https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2C%20hdfs%2C%20mapreduce%2C%20yarn%29%20and%20resolution%20%3D%20Unresolved%20and%20priority%20in%20%28Blocker%2C%20Critical%29%20and%20%22Target%20Version%2Fs%22%20in%20%28%222.3.0%22%2C%20%222.4.0%22%29

Jason

On 01/23/2014 06:53 PM, Sangjin Lee wrote:

I also wanted to bring your attention to HADOOP-9652. As this would address
a major performance issue with the local filesystem, it would be good to
get this in for 2.4 (or is it 2.3?).

Regards,
Sangjin


On Thu, Jan 23, 2014 at 2:23 AM, Steve Loughran ste...@hortonworks.comwrote:


the JIRA I referenced is ready to go -it just needs review


On 22 January 2014 20:04, Andrew Wang andrew.w...@cloudera.com wrote:


Thanks for the comments everyone.

Vinod, if you think YARN-149 isn't ready yet, we can leave it out.
Alternatively, we could release note it as beta with said known issues,
and let people kick the tires. It looks like a bunch of the core
functionality is already in place.

Unless anyone else objects, I plan to cut a 2.4 branch later this week.
Steve, Stack, if you think those two JIRAs can wrap up in that time frame
we can include it. Since this more experiment will hopefully go well, we
can include them in a 2.5 next month.

Best,
Andrew


On Wed, Jan 22, 2014 at 11:28 AM, Stack st...@duboce.net wrote:


Thanks Andrew for bringing this up.

+1 on more frequent releases and an effort at (roughly time-based

release.

We are working to get 'HDFS-5776 Support 'hedged' reads in DFSClient'

to

land in time for 2.4 (but don't hold up the release for us!)

St.Ack


On Tue, Jan 21, 2014 at 3:51 PM, Vinod Kumar Vavilapalli 
vino...@hortonworks.com wrote:


If the timeline is to cut one next week, I don't think we can ship
YARN-149 as part of that and call it stable. There are a bunch of

major

things that are still missing there: YARN-1202, YARN-1410, YARN-1525

and

YARN-1611/YARN-1459.

We need to start labeling individual features alpha/beta/stable now

that

we have a stable 2.2 base.

Thanks
+Vinod

On Jan 21, 2014, at 1:26 PM, Andrew Wang andrew.w...@cloudera.com

wrote:

So, as per my earlier email, I think a 2.4 with just HDFS-4949,

HDFS-2832,

and YARN-149 would be an attractive and stable release, and is

something

we

could actually cut this week and vote on before the month is out.

The

other

stuff we can ship in Feb or March when it's gotten a chance to bake

for a

bit, and culturally speaking, the fact that it's in 2.5 rather than

2.4

shouldn't be a big deal.

Thanks,
Andrew


On Tue, Jan 21, 2014 at 11:37 AM, Suresh Srinivas 

sur...@hortonworks.comwrote:

There is not much progress on symlinks issue. I think we should

move

forward with 2.4 release with symlinks disabled.

Status of 2.4 features from HDFS so far:
- HDFS-2832 Heterogeneous storage support has been merged
- HDFS-5535 rolling upgrades work is in progress
- HDFS-4685 ACL related work is close to completion
- HDFS-4949 As Andrew has proposed, this will be soon merged into

2.4

Regards,
Suresh

On Tue, Jan 21, 2014 at 11:12 AM, Arun C Murthy 

a...@hortonworks.com

wrote:


Andrew,

I'm almost ready to push out rc0 for 2.3 (been testing it

overnight),

I'm

pretty sure I'll get that out tonight.

However, AHS (YARN-321) is very close (merge vote going on) … so

that

will definitely make it in very soon.

So, my plan is essentially the same i.e. release 2.4 end of the

month

(after a bit more testing of RM HA in secure mode). Thanks for

the

offer,

I'll ping you if I need any help.

OTOH, can someone from HDFS chime in on status of symlinks?

Arun

On Jan 20, 2014, at 4:19 PM, Andrew Wang 

andrew.w...@cloudera.com

wrote:

Hi all,

I'm pretty excited to see a 2.4 this month if possible. Since I

think

people were favorable to the idea of time-based releases, how do

we

feel

about just cutting branch-2 and spinning up the release process

for

our

January goal?

Looking at the roadmap (https://wiki.apache.org/hadoop/Roadmap

),

on

the

HDFS side, I plan to post a branch-2 patch for HDFS-4949 this

week,

and

HDFS-2832 is already in. On the YARN side, it appears that RM HA

is

in,

but

the other three features (AHS, unmanaged containers, and dynamic

resource

configuration) remain unresolved.

I think a 2.4 with HDFS-4949, HDFS-2832, and YARN-149 is

already a

pretty

nice release. If it'd

Re: Logistics for releasing 2.4

2014-01-24 Thread Kihwal Lee 
Thanks for the list, Jason.  I scanned through them and except HDFS-5356, all 
jiras were currently targeted for 2.4.0. I changed the target version of 
HDFS-5356 to 2.4.0, so now all are targeted for 2.4.0.

Assuming 2.3.0 label is clean and doesn't need any mass/batch update, I think 
we can go over these jiras and re-target for 2.3.0, if one needs to be fixed in 
2.3.0. 

Kihwal



On Friday, January 24, 2014 10:07 AM, Jason Lowe jl...@yahoo-inc.com wrote:
 
I just committed the addendum patch for HADOOP-9652 which should resolve 
the performance issue.

Speaking of critical issues to fix for 2.3, I do wonder what to do with 
all the Blocker/Criticals targeted for 2.4 (the old branch-2 HEAD 
release) which apparently is now going to be 2.3 (the new branch-2 HEAD 
release).  We should either get these resolved before the 2.3 release or 
decide which ones aren't really blockers or that critical for 2.3 and 
target them for a future release (e.g.: 2.4). According to my 
calculations, there's currently 24 JIRAs that need to be scrubbed 
accordingly.

https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2C%20hdfs%2C%20mapreduce%2C%20yarn%29%20and%20resolution%20%3D%20Unresolved%20and%20priority%20in%20%28Blocker%2C%20Critical%29%20and%20%22Target%20Version%2Fs%22%20in%20%28%222.3.0%22%2C%20%222.4.0%22%29

Jason


On 01/23/2014 06:53 PM, Sangjin Lee wrote:
 I also wanted to bring your attention to HADOOP-9652. As this would address
 a major performance issue with the local filesystem, it would be good to
 get this in for 2.4 (or is it 2.3?).

 Regards,
 Sangjin


 On Thu, Jan 23, 2014 at 2:23 AM, Steve Loughran ste...@hortonworks.comwrote:

 the JIRA I referenced is ready to go -it just needs review


 On 22 January 2014 20:04, Andrew Wang andrew.w...@cloudera.com wrote:

 Thanks for the comments everyone.

 Vinod, if you think YARN-149 isn't ready yet, we can leave it out.
 Alternatively, we could release note it as beta with said known issues,
 and let people kick the tires. It looks like a bunch of the core
 functionality is already in place.

 Unless anyone else objects, I plan to cut a 2.4 branch later this week.
 Steve, Stack, if you think those two JIRAs can wrap up in that time frame
 we can include it. Since this more experiment will hopefully go well, we
 can include them in a 2.5 next month.

 Best,
 Andrew


 On Wed, Jan 22, 2014 at 11:28 AM, Stack st...@duboce.net wrote:

 Thanks Andrew for bringing this up.

 +1 on more frequent releases and an effort at (roughly time-based
 release.
 We are working to get 'HDFS-5776 Support 'hedged' reads in DFSClient'
 to
 land in time for 2.4 (but don't hold up the release for us!)

 St.Ack


 On Tue, Jan 21, 2014 at 3:51 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:

 If the timeline is to cut one next week, I don't think we can ship
 YARN-149 as part of that and call it stable. There are a bunch of
 major
 things that are still missing there: YARN-1202, YARN-1410, YARN-1525
 and
 YARN-1611/YARN-1459.

 We need to start labeling individual features alpha/beta/stable now
 that
 we have a stable 2.2 base.

 Thanks
 +Vinod

 On Jan 21, 2014, at 1:26 PM, Andrew Wang andrew.w...@cloudera.com
 wrote:
 So, as per my earlier email, I think a 2.4 with just HDFS-4949,
 HDFS-2832,
 and YARN-149 would be an attractive and stable release, and is
 something
 we
 could actually cut this week and vote on before the month is out.
 The
 other
 stuff we can ship in Feb or March when it's gotten a chance to bake
 for a
 bit, and culturally speaking, the fact that it's in 2.5 rather than
 2.4
 shouldn't be a big deal.

 Thanks,
 Andrew


 On Tue, Jan 21, 2014 at 11:37 AM, Suresh Srinivas 
 sur...@hortonworks.comwrote:
 There is not much progress on symlinks issue. I think we should
 move
 forward with 2.4 release with symlinks disabled.

 Status of 2.4 features from HDFS so far:
 - HDFS-2832 Heterogeneous storage support has been merged
 - HDFS-5535 rolling upgrades work is in progress
 - HDFS-4685 ACL related work is close to completion
 - HDFS-4949 As Andrew has proposed, this will be soon merged into
 2.4
 Regards,
 Suresh

 On Tue, Jan 21, 2014 at 11:12 AM, Arun C Murthy 
 a...@hortonworks.com
 wrote:

 Andrew,

 I'm almost ready to push out rc0 for 2.3 (been testing it
 overnight),
 I'm
 pretty sure I'll get that out tonight.

 However, AHS (YARN-321) is very close (merge vote going on) … so
 that
 will definitely make it in very soon.

 So, my plan is essentially the same i.e. release 2.4 end of the
 month
 (after a bit more testing of RM HA in secure mode). Thanks for
 the
 offer,
 I'll ping you if I need any help.

 OTOH, can someone from HDFS chime in on status of symlinks?

 Arun

 On Jan 20, 2014, at 4:19 PM, Andrew Wang 
 andrew.w...@cloudera.com
 wrote:
 Hi all,

 I'm pretty excited to see a 2.4 this month if possible. Since I
 think
 people were favorable to the idea of time-based releases, how do
 we
 feel
 about just cutting

[jira] [Created] (HADOOP-10277) setfacl -x fails to parse ACL spec if trying to remove the mask entry.

2014-01-24 Thread Chris Nauroth (JIRA) 
Chris Nauroth created HADOOP-10277:
--

 Summary: setfacl -x fails to parse ACL spec if trying to remove 
the mask entry.
 Key: HADOOP-10277
 URL: https://issues.apache.org/jira/browse/HADOOP-10277
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth


You should be able to use setfacl -x to remove the mask entry (which then 
triggers recalculation of an automatically inferred mask if the file has an 
extended ACL).  Right now, this causes a failure to parse the ACL spec due to a 
bug in {{AclEntry#parseAclSpec}}.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10278) Refactor to make CallQueue pluggable

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10278:
-

 Summary: Refactor to make CallQueue pluggable
 Key: HADOOP-10278
 URL: https://issues.apache.org/jira/browse/HADOOP-10278
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc
Reporter: Chris Li


* Refactor CallQueue into an interface, base, and default implementation that 
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize 
coupling



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10279) Create multiplexer, a requirement for the fair queue

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10279:
-

 Summary: Create multiplexer, a requirement for the fair queue
 Key: HADOOP-10279
 URL: https://issues.apache.org/jira/browse/HADOOP-10279
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10280) Make Schedulables return a configurable identity of user or group

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10280:
-

 Summary: Make Schedulables return a configurable identity of user 
or group
 Key: HADOOP-10280
 URL: https://issues.apache.org/jira/browse/HADOOP-10280
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10281) Create a scheduler, which assigns schedulables a priority level

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10281:
-

 Summary: Create a scheduler, which assigns schedulables a priority 
level
 Key: HADOOP-10281
 URL: https://issues.apache.org/jira/browse/HADOOP-10281
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10283) Add metrics to the FairCallQueue

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10283:
-

 Summary: Add metrics to the FairCallQueue
 Key: HADOOP-10283
 URL: https://issues.apache.org/jira/browse/HADOOP-10283
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10282) Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10282:
-

 Summary: Create a FairCallQueue: a multi-level call queue which 
schedules incoming calls and multiplexes outgoing calls
 Key: HADOOP-10282
 URL: https://issues.apache.org/jira/browse/HADOOP-10282
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10284) Add metrics to the HistoryRpcScheduler

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10284:
-

 Summary: Add metrics to the HistoryRpcScheduler
 Key: HADOOP-10284
 URL: https://issues.apache.org/jira/browse/HADOOP-10284
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10285) Allow CallQueue impls to be swapped at runtime

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10285:
-

 Summary: Allow CallQueue impls to be swapped at runtime
 Key: HADOOP-10285
 URL: https://issues.apache.org/jira/browse/HADOOP-10285
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10286) Allow RPCCallBenchmark to benchmark calls by different users

2014-01-24 Thread Chris Li (JIRA) 
Chris Li created HADOOP-10286:
-

 Summary: Allow RPCCallBenchmark to benchmark calls by different 
users
 Key: HADOOP-10286
 URL: https://issues.apache.org/jira/browse/HADOOP-10286
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Chris Li






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HADOOP-10287) FSOutputSummer should support any checksum size

2014-01-24 Thread Laurent Goujon (JIRA) 
Laurent Goujon created HADOOP-10287:
---

 Summary: FSOutputSummer should support any checksum size
 Key: HADOOP-10287
 URL: https://issues.apache.org/jira/browse/HADOOP-10287
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.2.0, 3.0.0
Reporter: Laurent Goujon


HADOOP-9114 only fixes if checksum size is 0, but doesn't handle the generic 
case.

FSOutputSummer should work with any checksum size (between 0 and 8 since 
Checksum.getValue() returns a long)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)