[jira] [Created] (HADOOP-10779) Generalize DFS_PERMISSIONS_SUPERUSERGROUP_KEY for any HCFS
Martin Bukatovic created HADOOP-10779:
-
Summary: Generalize DFS_PERMISSIONS_SUPERUSERGROUP_KEY for any HCFS
Key: HADOOP-10779
URL: https://issues.apache.org/jira/browse/HADOOP-10779
Project: Hadoop Common
Issue Type: Wish
Components: fs
Reporter: Martin Bukatovic
Priority: Minor
HDFS has configuration option {{dfs.permissions.superusergroup}} stored in
{{hdfs-site.xml}} configuration file:
{noformat}
property
namedfs.permissions.superusergroup/name
valuesupergroup/value
descriptionThe name of the group of super-users./description
/property
{noformat}
Since we have an option to use alternative Hadoop filesystems (HCFS), there is
a question how to specify a supergroup in such case.
Eg. would introducing HCFS option in say {{core-site.xml}} for this as shown
below make sense?
{noformat}
property
namehcfs.permissions.superusergroup/name
value${dfs.permissions.superusergroup}/value
descriptionThe name of the group of super-users./description
/property
{noformat}
Or would you solve it in different way? I would like to at least declare
a recommended approach for alternative Hadoop filesystems to follow.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Doubt in File concat implementaion
Hi All,
Usually we can't ensure lastblock alwaysfull...please let me know purpose of
following check..
long blockSize = trgInode.getPreferredBlockSize();
// check the end block to be full
final BlockInfo last = trgInode.getLastBlock();
if(blockSize != last.getNumBytes()) {
throw new HadoopIllegalArgumentException(The last block in + target
+ is not full; last block size = + last.getNumBytes()
+ but file block size = + blockSize);
}
If it is issue, I'll file jira.
Following is the trace..
exception in thread main
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.HadoopIllegalArgumentException):
The last block in /file_one.txt is not full; last block size = 14 but file
block size = 134217728
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concatInternal(FSNamesystem.java:1887)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concatInt(FSNamesystem.java:1833)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concat(FSNamesystem.java:1795)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.concat(NameNodeRpcServer.java:704)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.concat(ClientNamenodeProtocolServerSideTranslatorPB.java:512)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
Thanks Regards
Brahma Reddy Battula
[jira] [Resolved] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9597. Resolution: Done Fix Version/s: 2.5.0 FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Sub-task Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Assignee: Steve Loughran Priority: Minor Fix For: 2.5.0 The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10533) S3 input stream NPEs in MapReduce jon
[ https://issues.apache.org/jira/browse/HADOOP-10533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-10533. - Resolution: Fixed Fix Version/s: 2.5.0 S3 input stream NPEs in MapReduce jon - Key: HADOOP-10533 URL: https://issues.apache.org/jira/browse/HADOOP-10533 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 1.0.0, 1.0.3, 3.0.0, 2.4.0 Environment: Hadoop with default configurations Reporter: Benjamin Kim Assignee: Steve Loughran Priority: Minor Fix For: 2.5.0 I'm running a wordcount MR as follows hadoop jar WordCount.jar wordcount.WordCountDriver s3n://bucket/wordcount/input s3n://bucket/wordcount/output s3n://bucket/wordcount/input is a s3 object that contains other input files. However I get following NPE error 12/10/02 18:56:23 INFO mapred.JobClient: map 0% reduce 0% 12/10/02 18:56:54 INFO mapred.JobClient: map 50% reduce 0% 12/10/02 18:56:56 INFO mapred.JobClient: Task Id : attempt_201210021853_0001_m_01_0, Status : FAILED java.lang.NullPointerException at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:106) at java.io.BufferedInputStream.close(BufferedInputStream.java:451) at java.io.FilterInputStream.close(FilterInputStream.java:155) at org.apache.hadoop.util.LineReader.close(LineReader.java:83) at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) MR runs fine if i specify more specific input path such as s3n://bucket/wordcount/input/file.txt MR fails if I pass s3 folder as a parameter In summary, This works hadoop jar ./hadoop-examples-1.0.3.jar wordcount /user/hadoop/wordcount/input/ s3n://bucket/wordcount/output/ This doesn't work hadoop jar ./hadoop-examples-1.0.3.jar wordcount s3n://bucket/wordcount/input/ s3n://bucket/wordcount/output/ (both input path are directories) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9711) Write contract tests for S3Native; fix places where it breaks
[ https://issues.apache.org/jira/browse/HADOOP-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9711. Resolution: Fixed Fix Version/s: 2.5.0 Assignee: Steve Loughran Write contract tests for S3Native; fix places where it breaks - Key: HADOOP-9711 URL: https://issues.apache.org/jira/browse/HADOOP-9711 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 1.2.0, 3.0.0, 2.1.0-beta Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Fix For: 2.5.0 Attachments: HADOOP-9711-004.patch implement the abstract contract tests for S3, identify where it is failing to meet expectations and, where possible, fix. Blobstores tend to treat 0 byte files as directories, so tests overwriting files with dirs and vice versa may fail and have to be skipped -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10419) BufferedFSInputStream NPEs on getPos() on a closed stream
[
https://issues.apache.org/jira/browse/HADOOP-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-10419.
-
Resolution: Fixed
Fix Version/s: 2.5.0
BufferedFSInputStream NPEs on getPos() on a closed stream
-
Key: HADOOP-10419
URL: https://issues.apache.org/jira/browse/HADOOP-10419
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Fix For: 2.5.0
Original Estimate: 0.25h
Remaining Estimate: 0.25h
if you call getPos on a {{ChecksumFileSystem}} after a {{close()}} you get an
NPE.
While throwing an exception in this states is legitimate (HDFS does, RawLocal
does not), it should be an {{IOException}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-9371) Define Semantics of FileSystem more rigorously
[
https://issues.apache.org/jira/browse/HADOOP-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-9371.
Resolution: Fixed
Fix Version/s: 2.5.0
Define Semantics of FileSystem more rigorously
--
Key: HADOOP-9371
URL: https://issues.apache.org/jira/browse/HADOOP-9371
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 1.2.0, 3.0.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Fix For: 2.5.0
Attachments: HADOOP-9361.2.patch, HADOOP-9361.patch,
HADOOP-9371-003.patch, HadoopFilesystemContract.pdf
Original Estimate: 48h
Remaining Estimate: 48h
The semantics of {{FileSystem}} and {{FileContext}} are not completely
defined in terms of
# core expectations of a filesystem
# consistency requirements.
# concurrency requirements.
# minimum scale limits
Furthermore, methods are not defined strictly enough in terms of their
outcomes and failure modes.
The requirements and method semantics should be defined more strictly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-9495) Define behaviour of Seekable.seek(), write tests, fix all hadoop implementations for compliance
[
https://issues.apache.org/jira/browse/HADOOP-9495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-9495.
Resolution: Fixed
Fix Version/s: 2.5.0
Define behaviour of Seekable.seek(), write tests, fix all hadoop
implementations for compliance
---
Key: HADOOP-9495
URL: https://issues.apache.org/jira/browse/HADOOP-9495
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 1.2.0, 3.0.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Fix For: 2.5.0
Attachments: HADOOP-9495.patch, HADOOP-9545-002.patch
{{Seekable.seek()}} seems a good starting point for specifying, testing and
implementing FS API compliance: one method, relatively non-ambiguous
semantics, easily assessed used in the Hadoop codebase. Specify and test it
first
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10780) namenode throws java.lang.OutOfMemoryError upon DatanodeProtocol.versionRequest from datanode
Dmitry Sivachenko created HADOOP-10780: -- Summary: namenode throws java.lang.OutOfMemoryError upon DatanodeProtocol.versionRequest from datanode Key: HADOOP-10780 URL: https://issues.apache.org/jira/browse/HADOOP-10780 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Environment: FreeBSD-10/stable openjdk version 1.7.0_60 OpenJDK Runtime Environment (build 1.7.0_60-b19) OpenJDK 64-Bit Server VM (build 24.60-b09, mixed mode) Reporter: Dmitry Sivachenko I am trying hadoop-2.4.1 on FreeBSD-10/stable. namenode starts up, but after first datanode contacts it, it throws an exception. All limits seem to be high enough: % limits -a Resource limits (current): cputime infinity secs filesize infinity kB datasize 33554432 kB stacksize 524288 kB coredumpsize infinity kB memoryuseinfinity kB memorylocked infinity kB maxprocesses 122778 openfiles 14 sbsize infinity bytes vmemoryuse infinity kB pseudo-terminals infinity swapuse infinity kB 14944 1 S0:06.59 /usr/local/openjdk7/bin/java -Dproc_namenode -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop -Dhadoop.log.file=hadoop-hdfs-namenode-nezabudka3-00.log -Dhadoop.home.dir=/usr/local -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx32768m -Xms32768m -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m -Djava.library.path=/usr/local/lib -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode From the namenode's log: 2014-07-03 23:28:15,070 WARN [IPC Server handler 5 on 8020] ipc.Server (Server.java:run(2032)) - IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.server.protocol.Datano deProtocol.versionRequest from 5.255.231.209:57749 Call#842 Retry#0 java.lang.OutOfMemoryError at org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Native Method) at org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(JniBasedUnixGroupsMapping.java:80) at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50) at org.apache.hadoop.security.Groups.getGroups(Groups.java:139) at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1417) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.init(FSPermissionChecker.java:81) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3331) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkSuperuserPrivilege(FSNamesystem.java:5491) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.versionRequest(NameNodeRpcServer.java:1082) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.versionRequest(DatanodeProtocolServerSideTranslatorPB.java:234) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28069) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) I did not have such an issue with hadoop-1.2.1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
[ https://issues.apache.org/jira/browse/HADOOP-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HADOOP-10693. --- Resolution: Fixed committed to fs-encryption branch Implementation of AES-CTR CryptoCodec using JNI to OpenSSL -- Key: HADOOP-10693 URL: https://issues.apache.org/jira/browse/HADOOP-10693 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Yi Liu Assignee: Yi Liu Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) Attachments: HADOOP-10693.1.patch, HADOOP-10693.2.patch, HADOOP-10693.3.patch, HADOOP-10693.4.patch, HADOOP-10693.5.patch, HADOOP-10693.6.patch, HADOOP-10693.7.patch, HADOOP-10693.8.patch, HADOOP-10693.patch In HADOOP-10603, we have an implementation of AES-CTR CryptoCodec using Java JCE provider. To get high performance, the configured JCE provider should utilize native code and AES-NI, but in JDK6,7 the Java embedded provider doesn't support it. Considering not all hadoop user will use the provider like Diceros or able to get signed certificate from oracle to develop a custom provider, so this JIRA will have an implementation of AES-CTR CryptoCodec using JNI to OpenSSL directly. -- This message was sent by Atlassian JIRA (v6.2#6252)
