[jira] [Resolved] (HADOOP-11097) kms docs say proxyusers, not proxyuser for config params

2014-09-16 Thread Alejandro Abdelnur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-11097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur resolved HADOOP-11097.
-
   Resolution: Fixed
Fix Version/s: 2.6.0
 Hadoop Flags: Reviewed

Thanks Charles. Committed to trunk and branch-2.

> kms docs say proxyusers, not proxyuser for config params
> 
>
> Key: HADOOP-11097
> URL: https://issues.apache.org/jira/browse/HADOOP-11097
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.0.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Trivial
> Fix For: 2.6.0
>
> Attachments: HADOOP-11097.001.patch
>
>
> The KMS docs have the proxy parameters named proxyusers, not proxyuser.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11018) KMS should load multiple kerberos principals

2014-09-16 Thread Alejandro Abdelnur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-11018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur resolved HADOOP-11018.
-
Resolution: Duplicate

> KMS should load multiple kerberos principals
> 
>
> Key: HADOOP-11018
> URL: https://issues.apache.org/jira/browse/HADOOP-11018
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 2.6.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>
> This would allow multiple KMS instance to serve behind a VIP and provide 
> direct access to a particular instance as well (ie for monitoring purposes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11099) KMS return HTTP UNAUTHORIZED 401 on ACL failure

2014-09-16 Thread Alejandro Abdelnur (JIRA) 
Alejandro Abdelnur created HADOOP-11099:
---

 Summary: KMS return HTTP UNAUTHORIZED 401 on ACL failure
 Key: HADOOP-11099
 URL: https://issues.apache.org/jira/browse/HADOOP-11099
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.6.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur


The usual error, HTTP UNAUTHORIZED means is for authentication, not for 
authorization.

KMS should return HTTP FORBIDDEN in case of ACL failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: hbase use error

2014-09-16 Thread mike Zarrin 
Please unsubscribe me

 

From: Ted Yu [mailto:yuzhih...@gmail.com] 
Sent: Tuesday, September 16, 2014 7:34 PM
To: common-u...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org
Subject: Re: hbase use error

 

Which hadoop release are you using ?

Can you pastebin more of the server logs ?

 

bq. load file larger than 20M

 

Do you store such file(s) directly on hdfs and put its path in hbase ?

See HBASE-11339 HBase MOB

 

On Tue, Sep 16, 2014 at 7:29 PM, QiXiangming  wrote:

hello ,everyone

i use hbase to store small pic or files , and meet an exception raised 
from hdfs, as following :

 

 slave2:50010:DataXceiver error processing WRITE_BLOCK operation  src: 
/192.168.20.246:33162 dest: /192.168.20.247:50010
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:446)
at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:702)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:711)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229)
at java.lang.Thread.run(Thread.java:745)

 

when hbase stores pics or file  under 200k, it works well, 

but if you load file larger than 20M , hbase definitely down!

 

what's wrong with it ? 

can anyone help use? 

 

 

URGENT!!!

 

 

Qi Xiangming

Re: hbase use error

2014-09-16 Thread Ted Yu 
Which hadoop release are you using ?
Can you pastebin more of the server logs ?

bq. load file larger than 20M

Do you store such file(s) directly on hdfs and put its path in hbase ?
See HBASE-11339 HBase MOB

On Tue, Sep 16, 2014 at 7:29 PM, QiXiangming 
wrote:

> hello ,everyone
> i use hbase to store small pic or files , and meet an exception
> raised from hdfs, as following :
>
>  slave2:50010:DataXceiver error processing WRITE_BLOCK operation  src: /
> 192.168.20.246:33162 dest: /192.168.20.247:50010
> java.io.IOException: Premature EOF from inputStream
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
>
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
>
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
>
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
>
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:446)
>
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:702)
>
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:711)
>
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
>
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
>
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229)
> at java.lang.Thread.run(Thread.java:745)
>
> when hbase stores pics or file  under 200k, it works well,
> but if you load file larger than 20M , hbase definitely down!
>
> what's wrong with it ?
> can anyone help use?
>
>
> URGENT!!!
>
>
> Qi Xiangming
>

[jira] [Created] (HADOOP-11098) [jdk8] MaxDirectMemorySize default changed between JDK7 and 8

2014-09-16 Thread Travis Thompson (JIRA) 
Travis Thompson created HADOOP-11098:


 Summary: [jdk8] MaxDirectMemorySize default changed between JDK7 
and 8
 Key: HADOOP-11098
 URL: https://issues.apache.org/jira/browse/HADOOP-11098
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Travis Thompson


I noticed this because the NameNode UI shows "Max Non Heap Memory" which after 
some digging I found correlates to MaxDirectMemorySize.

JDK7
{noformat}
Heap Memory used 16.75 GB of 23 GB Heap Memory. Max Heap Memory is 23.7 GB.
Non Heap Memory used 57.32 MB of 67.38 MB Commited Non Heap Memory. Max Non 
Heap Memory is 130 MB. 
{noformat}

JDK8
{noformat}
Heap Memory used 3.02 GB of 7.65 GB Heap Memory. Max Heap Memory is 23.7 GB.
Non Heap Memory used 103.12 MB of 104.41 MB Commited Non Heap Memory. Max Non 
Heap Memory is -1 B. 
{noformat}

More information in first comment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Is it a bug in CombineFileSplit?

2014-09-16 Thread Benyi Wang 
I use Spark's SerializableWritable to wrap CombineFileSplit so I can pass
around the splits. But I ran into Serialization issues. In researching why
my code fails, I found that this might be a bug in CombineFileSplit:

CombineFileSplit doesn't serialize locations in write(DataOutput out) and
deserialize locations in readFields(DataInput in).

When I create a split in CombineFileInputFormat, locations is an array of
String[0], but after deserialization (default contructor, then readFields),
the locations will be null.

This will lead NPE.

[jira] [Created] (HADOOP-11097) kms docs say proxyusers, not proxyuser for config params

2014-09-16 Thread Charles Lamb (JIRA) 
Charles Lamb created HADOOP-11097:
-

 Summary: kms docs say proxyusers, not proxyuser for config params
 Key: HADOOP-11097
 URL: https://issues.apache.org/jira/browse/HADOOP-11097
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 3.0.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial


The KMS docs have the proxy parameters named proxyusers, not proxyuser.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11075) hadoop-kms is not being published to maven

2014-09-16 Thread Alejandro Abdelnur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur resolved HADOOP-11075.
-
Resolution: Duplicate

Integrating this into HDFS-7006, will give credit to 
[~anthony.young-gar...@cloudera.com] for it on commit.

> hadoop-kms is not being published to maven
> --
>
> Key: HADOOP-11075
> URL: https://issues.apache.org/jira/browse/HADOOP-11075
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Anthony Young-Garner
>Assignee: Anthony Young-Garner
>Priority: Minor
> Attachments: 0001-Exposed-hadoop-kms-classes-on-hadoop-KMS-POM.patch, 
> HADOOP-11075.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISUCSS] Reasonable Hadoop ACL Defaults

2014-09-16 Thread Allen Wittenauer 

Removing security@ , adding hdfs-dev@ .

On Sep 16, 2014, at 1:19 AM, Zhijie Shen  wrote:

> Hi folks,
> 
> There're a bunch of ACLs configuration defaults, which are set to "*":
> 
> 1. yarn.admin.acl in yarn-default.xml
> 2. 
> yarn.scheduler.capacity.root.default.[acl_submit_applications|acl_administer_queue]
> in capacity-scheduler.xml
> 3. security.*.protocol.acl in hadoop-policy.xml
> 
> When ACL (or server authorization) is enabled, the resources that are
> supposed to be protected are still accessible. However, anybody can
> still access them because the default configurations are "*",
> accepting anybody. These defaults seem not to make much sense, but
> only confuse users. Instead, the reasonable behavior should be that
> when ACL is enabled, a user is going to be denied by default unless we
> explicitly add him/her into the admin ACLs or the authorized
> user/group list.
> 
> I have a patch to invert "*" to " "  to block all users by default.
> Please let me how what you think about it, and how we should progress.


a) It would be an incompatible change and would need to go to trunk.
b) Users enabling ACLs should be expected to go through and check the 
settings to see what exactly they are enabling/disabling.

[jira] [Created] (HADOOP-11095) How about Null check when closing inputstream object in JavaKeyStoreProvider#() ?

2014-09-16 Thread skrho (JIRA) 
skrho created HADOOP-11095:
--

 Summary: How about Null check when closing inputstream object in 
JavaKeyStoreProvider#() ?
 Key: HADOOP-11095
 URL: https://issues.apache.org/jira/browse/HADOOP-11095
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: skrho
Priority: Minor


In the finally block:
  InputStream is = pwdFile.openStream();
  try {
password = IOUtils.toCharArray(is);
  } finally {
is.close();
  }
  
How about Null check when closing inputstream object?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Common-0.23-Build #1074

2014-09-16 Thread Apache Jenkins Server 
See 

--
[...truncated 8263 lines...]
Running org.apache.hadoop.io.TestBloomMapFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.879 sec
Running org.apache.hadoop.io.TestObjectWritableProtos
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.321 sec
Running org.apache.hadoop.io.TestTextNonUTF8
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec
Running org.apache.hadoop.io.nativeio.TestNativeIO
Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 0.159 sec
Running org.apache.hadoop.io.TestSortedMapWritable
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.196 sec
Running org.apache.hadoop.io.TestMapFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.639 sec
Running org.apache.hadoop.io.TestUTF8
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.446 sec
Running org.apache.hadoop.io.TestBoundedByteArrayOutputStream
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.044 sec
Running org.apache.hadoop.io.retry.TestRetryProxy
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.227 sec
Running org.apache.hadoop.io.retry.TestFailoverProxy
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.192 sec
Running org.apache.hadoop.io.TestSetFile
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.957 sec
Running org.apache.hadoop.io.serializer.TestWritableSerialization
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.316 sec
Running org.apache.hadoop.io.serializer.TestSerializationFactory
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.281 sec
Running org.apache.hadoop.io.serializer.avro.TestAvroSerialization
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.539 sec
Running org.apache.hadoop.util.TestGenericOptionsParser
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.694 sec
Running org.apache.hadoop.util.TestReflectionUtils
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.516 sec
Running org.apache.hadoop.util.TestJarFinder
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.782 sec
Running org.apache.hadoop.util.TestPureJavaCrc32
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.287 sec
Running org.apache.hadoop.util.TestHostsFileReader
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.183 sec
Running org.apache.hadoop.util.TestShutdownHookManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.144 sec
Running org.apache.hadoop.util.TestDiskChecker
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.521 sec
Running org.apache.hadoop.util.TestStringUtils
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.139 sec
Running org.apache.hadoop.util.TestGenericsUtil
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec
Running org.apache.hadoop.util.TestAsyncDiskService
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.125 sec
Running org.apache.hadoop.util.TestProtoUtil
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 sec
Running org.apache.hadoop.util.TestDataChecksum
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 sec
Running org.apache.hadoop.util.TestRunJar
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.134 sec
Running org.apache.hadoop.util.TestOptions
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec
Running org.apache.hadoop.util.TestShell
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.197 sec
Running org.apache.hadoop.util.TestIndexedSort
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.61 sec
Running org.apache.hadoop.util.TestStringInterner
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.114 sec
Running org.apache.hadoop.record.TestRecordVersioning
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.158 sec
Running org.apache.hadoop.record.TestBuffer
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec
Running org.apache.hadoop.record.TestRecordIO
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.178 sec
Running org.apache.hadoop.security.TestGroupFallback
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 sec
Running org.apache.hadoop.security.TestGroupsCaching
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.274 sec
Running org.apache.hadoop.security.TestProxyUserFromEnv
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.355 sec
Running org.apache.hadoop.security.TestUserGroupInformation
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.7 sec
Running org.apache.hadoop.security.TestJNIGroupsMapping
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.14 sec
Running

[DISUCSS] Reasonable Hadoop ACL Defaults

2014-09-16 Thread Zhijie Shen 
Hi folks,

There're a bunch of ACLs configuration defaults, which are set to "*":

1. yarn.admin.acl in yarn-default.xml
2. 
yarn.scheduler.capacity.root.default.[acl_submit_applications|acl_administer_queue]
in capacity-scheduler.xml
3. security.*.protocol.acl in hadoop-policy.xml

When ACL (or server authorization) is enabled, the resources that are
supposed to be protected are still accessible. However, anybody can
still access them because the default configurations are "*",
accepting anybody. These defaults seem not to make much sense, but
only confuse users. Instead, the reasonable behavior should be that
when ACL is enabled, a user is going to be denied by default unless we
explicitly add him/her into the admin ACLs or the authorized
user/group list.

I have a patch to invert "*" to " "  to block all users by default.
Please let me how what you think about it, and how we should progress.

Thanks,
Zhijie

-- 
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.