[jira] [Created] (HADOOP-10781) Unportable getgrouplist() usage breaks FreeBSD
Dmitry Sivachenko created HADOOP-10781: -- Summary: Unportable getgrouplist() usage breaks FreeBSD Key: HADOOP-10781 URL: https://issues.apache.org/jira/browse/HADOOP-10781 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Reporter: Dmitry Sivachenko getgrouplist() has different return values on Linux and FreeBSD: Linux: either the number of groups (positive) or -1 on error FreeBSD: 0 on success or -1 on error The return value of getgrouplist() is analyzed in Linux-specific way in hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/hadoop_user_info.c, in function hadoop_user_info_getgroups() which breaks FreeBSD. In this function you have 3 choices for the return value ret = getgrouplist(uinfo->pwd.pw_name, uinfo->pwd.pw_gid, uinfo->gids, &ngroups); 1) ret > 0 : OK for Linux, it will be zero on FreeBSD. I propose to change this to ret >= 0 2) First condition is false and ret != -1: impossible according to manpage 3) ret == 1 -- OK for both Linux and FreeBSD So I propose to change "ret > 0" to "ret >= 0" and (optionally) return 2nd case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10693) Implementation of AES-CTR CryptoCodec using JNI to OpenSSL
[ https://issues.apache.org/jira/browse/HADOOP-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HADOOP-10693. --- Resolution: Fixed committed to fs-encryption branch > Implementation of AES-CTR CryptoCodec using JNI to OpenSSL > -- > > Key: HADOOP-10693 > URL: https://issues.apache.org/jira/browse/HADOOP-10693 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HADOOP-10693.1.patch, HADOOP-10693.2.patch, > HADOOP-10693.3.patch, HADOOP-10693.4.patch, HADOOP-10693.5.patch, > HADOOP-10693.6.patch, HADOOP-10693.7.patch, HADOOP-10693.8.patch, > HADOOP-10693.patch > > > In HADOOP-10603, we have an implementation of AES-CTR CryptoCodec using Java > JCE provider. > To get high performance, the configured JCE provider should utilize native > code and AES-NI, but in JDK6,7 the Java embedded provider doesn't support it. > > Considering not all hadoop user will use the provider like Diceros or able to > get signed certificate from oracle to develop a custom provider, so this JIRA > will have an implementation of AES-CTR CryptoCodec using JNI to OpenSSL > directly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10780) namenode throws java.lang.OutOfMemoryError upon DatanodeProtocol.versionRequest from datanode
Dmitry Sivachenko created HADOOP-10780: -- Summary: namenode throws java.lang.OutOfMemoryError upon DatanodeProtocol.versionRequest from datanode Key: HADOOP-10780 URL: https://issues.apache.org/jira/browse/HADOOP-10780 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.1 Environment: FreeBSD-10/stable openjdk version "1.7.0_60" OpenJDK Runtime Environment (build 1.7.0_60-b19) OpenJDK 64-Bit Server VM (build 24.60-b09, mixed mode) Reporter: Dmitry Sivachenko I am trying hadoop-2.4.1 on FreeBSD-10/stable. namenode starts up, but after first datanode contacts it, it throws an exception. All limits seem to be high enough: % limits -a Resource limits (current): cputime infinity secs filesize infinity kB datasize 33554432 kB stacksize 524288 kB coredumpsize infinity kB memoryuseinfinity kB memorylocked infinity kB maxprocesses 122778 openfiles 14 sbsize infinity bytes vmemoryuse infinity kB pseudo-terminals infinity swapuse infinity kB 14944 1 S0:06.59 /usr/local/openjdk7/bin/java -Dproc_namenode -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop -Dhadoop.log.file=hadoop-hdfs-namenode-nezabudka3-00.log -Dhadoop.home.dir=/usr/local -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx32768m -Xms32768m -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m -Djava.library.path=/usr/local/lib -Xmx32768m -Xms32768m -Djava.library.path=/usr/local/lib -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode >From the namenode's log: 2014-07-03 23:28:15,070 WARN [IPC Server handler 5 on 8020] ipc.Server (Server.java:run(2032)) - IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.server.protocol.Datano deProtocol.versionRequest from 5.255.231.209:57749 Call#842 Retry#0 java.lang.OutOfMemoryError at org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroupsForUser(Native Method) at org.apache.hadoop.security.JniBasedUnixGroupsMapping.getGroups(JniBasedUnixGroupsMapping.java:80) at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50) at org.apache.hadoop.security.Groups.getGroups(Groups.java:139) at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1417) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:81) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3331) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkSuperuserPrivilege(FSNamesystem.java:5491) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.versionRequest(NameNodeRpcServer.java:1082) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.versionRequest(DatanodeProtocolServerSideTranslatorPB.java:234) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28069) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) I did not have such an issue with hadoop-1.2.1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9495) Define behaviour of Seekable.seek(), write tests, fix all hadoop implementations for compliance
[
https://issues.apache.org/jira/browse/HADOOP-9495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-9495.
Resolution: Fixed
Fix Version/s: 2.5.0
> Define behaviour of Seekable.seek(), write tests, fix all hadoop
> implementations for compliance
> ---
>
> Key: HADOOP-9495
> URL: https://issues.apache.org/jira/browse/HADOOP-9495
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 1.2.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.5.0
>
> Attachments: HADOOP-9495.patch, HADOOP-9545-002.patch
>
>
> {{Seekable.seek()}} seems a good starting point for specifying, testing and
> implementing FS API compliance: one method, relatively non-ambiguous
> semantics, easily assessed used in the Hadoop codebase. Specify and test it
> first
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-9371) Define Semantics of FileSystem more rigorously
[
https://issues.apache.org/jira/browse/HADOOP-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-9371.
Resolution: Fixed
Fix Version/s: 2.5.0
> Define Semantics of FileSystem more rigorously
> --
>
> Key: HADOOP-9371
> URL: https://issues.apache.org/jira/browse/HADOOP-9371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 1.2.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: 2.5.0
>
> Attachments: HADOOP-9361.2.patch, HADOOP-9361.patch,
> HADOOP-9371-003.patch, HadoopFilesystemContract.pdf
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The semantics of {{FileSystem}} and {{FileContext}} are not completely
> defined in terms of
> # core expectations of a filesystem
> # consistency requirements.
> # concurrency requirements.
> # minimum scale limits
> Furthermore, methods are not defined strictly enough in terms of their
> outcomes and failure modes.
> The requirements and method semantics should be defined more strictly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-10419) BufferedFSInputStream NPEs on getPos() on a closed stream
[
https://issues.apache.org/jira/browse/HADOOP-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-10419.
-
Resolution: Fixed
Fix Version/s: 2.5.0
> BufferedFSInputStream NPEs on getPos() on a closed stream
> -
>
> Key: HADOOP-10419
> URL: https://issues.apache.org/jira/browse/HADOOP-10419
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.5.0
>
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> if you call getPos on a {{ChecksumFileSystem}} after a {{close()}} you get an
> NPE.
> While throwing an exception in this states is legitimate (HDFS does, RawLocal
> does not), it should be an {{IOException}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-9711) Write contract tests for S3Native; fix places where it breaks
[ https://issues.apache.org/jira/browse/HADOOP-9711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9711. Resolution: Fixed Fix Version/s: 2.5.0 Assignee: Steve Loughran > Write contract tests for S3Native; fix places where it breaks > - > > Key: HADOOP-9711 > URL: https://issues.apache.org/jira/browse/HADOOP-9711 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 1.2.0, 3.0.0, 2.1.0-beta >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.5.0 > > Attachments: HADOOP-9711-004.patch > > > implement the abstract contract tests for S3, identify where it is failing to > meet expectations and, where possible, fix. Blobstores tend to treat 0 byte > files as directories, so tests overwriting files with dirs and vice versa may > fail and have to be skipped -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10533) S3 input stream NPEs in MapReduce jon
[ https://issues.apache.org/jira/browse/HADOOP-10533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-10533. - Resolution: Fixed Fix Version/s: 2.5.0 > S3 input stream NPEs in MapReduce jon > - > > Key: HADOOP-10533 > URL: https://issues.apache.org/jira/browse/HADOOP-10533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 1.0.0, 1.0.3, 3.0.0, 2.4.0 > Environment: Hadoop with default configurations >Reporter: Benjamin Kim >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.5.0 > > > I'm running a wordcount MR as follows > hadoop jar WordCount.jar wordcount.WordCountDriver > s3n://bucket/wordcount/input s3n://bucket/wordcount/output > > s3n://bucket/wordcount/input is a s3 object that contains other input files. > However I get following NPE error > 12/10/02 18:56:23 INFO mapred.JobClient: map 0% reduce 0% > 12/10/02 18:56:54 INFO mapred.JobClient: map 50% reduce 0% > 12/10/02 18:56:56 INFO mapred.JobClient: Task Id : > attempt_201210021853_0001_m_01_0, Status : FAILED > java.lang.NullPointerException > at > org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:106) > at java.io.BufferedInputStream.close(BufferedInputStream.java:451) > at java.io.FilterInputStream.close(FilterInputStream.java:155) > at org.apache.hadoop.util.LineReader.close(LineReader.java:83) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > MR runs fine if i specify more specific input path such as > s3n://bucket/wordcount/input/file.txt > MR fails if I pass s3 folder as a parameter > In summary, > This works > hadoop jar ./hadoop-examples-1.0.3.jar wordcount > /user/hadoop/wordcount/input/ s3n://bucket/wordcount/output/ > This doesn't work > hadoop jar ./hadoop-examples-1.0.3.jar wordcount > s3n://bucket/wordcount/input/ s3n://bucket/wordcount/output/ > (both input path are directories) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9597. Resolution: Done Fix Version/s: 2.5.0 > FileSystem open() API is not clear if FileNotFoundException is throw when the > path does not exist > - > > Key: HADOOP-9597 > URL: https://issues.apache.org/jira/browse/HADOOP-9597 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs >Affects Versions: 2.0.4-alpha >Reporter: Jerry He >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.5.0 > > > The current FileSystem open() method throws a generic IOException in its API > specification. > Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more > specific FileNotFoundException if the path does not exist. Some throws > IOException only (FTPFileSystem, HftpFileSystem ...). > If we have a new FileSystem implementation, what should we follow exactly for > open()? > What should the application expect in this case. > -- This message was sent by Atlassian JIRA (v6.2#6252)
Doubt in File concat implementaion
Hi All,
Usually we can't ensure lastblock alwaysfull...please let me know purpose of
following check..
long blockSize = trgInode.getPreferredBlockSize();
// check the end block to be full
final BlockInfo last = trgInode.getLastBlock();
if(blockSize != last.getNumBytes()) {
throw new HadoopIllegalArgumentException("The last block in " + target
+ " is not full; last block size = " + last.getNumBytes()
+ " but file block size = " + blockSize);
}
If it is issue, I'll file jira.
Following is the trace..
exception in thread "main"
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.HadoopIllegalArgumentException):
The last block in /file_one.txt is not full; last block size = 14 but file
block size = 134217728
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concatInternal(FSNamesystem.java:1887)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concatInt(FSNamesystem.java:1833)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concat(FSNamesystem.java:1795)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.concat(NameNodeRpcServer.java:704)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.concat(ClientNamenodeProtocolServerSideTranslatorPB.java:512)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
Thanks & Regards
Brahma Reddy Battula
Re: Making changes to "Distributions and Commercial Support" page
On 2 July 2014 17:44, Katherine Xavier wrote: > KatherineXavier try now -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-10779) Generalize DFS_PERMISSIONS_SUPERUSERGROUP_KEY for any HCFS
Martin Bukatovic created HADOOP-10779:
-
Summary: Generalize DFS_PERMISSIONS_SUPERUSERGROUP_KEY for any HCFS
Key: HADOOP-10779
URL: https://issues.apache.org/jira/browse/HADOOP-10779
Project: Hadoop Common
Issue Type: Wish
Components: fs
Reporter: Martin Bukatovic
Priority: Minor
HDFS has configuration option {{dfs.permissions.superusergroup}} stored in
{{hdfs-site.xml}} configuration file:
{noformat}
dfs.permissions.superusergroup
supergroup
The name of the group of super-users.
{noformat}
Since we have an option to use alternative Hadoop filesystems (HCFS), there is
a question how to specify a supergroup in such case.
Eg. would introducing HCFS option in say {{core-site.xml}} for this as shown
below make sense?
{noformat}
hcfs.permissions.superusergroup
${dfs.permissions.superusergroup}
The name of the group of super-users.
{noformat}
Or would you solve it in different way? I would like to at least declare
a recommended approach for alternative Hadoop filesystems to follow.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
