[jira] [Commented] (HADOOP-10087) UserGroupInformation.getGroupNames() fails to return primary group first when JniBasedUnixGroupsMappingWithFallback is used
[
https://issues.apache.org/jira/browse/HADOOP-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845061#comment-13845061
]
Hadoop QA commented on HADOOP-10087:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618167/HADOOP-10087.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3353//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3353//console
This message is automatically generated.
> UserGroupInformation.getGroupNames() fails to return primary group first when
> JniBasedUnixGroupsMappingWithFallback is used
> ---
>
> Key: HADOOP-10087
> URL: https://issues.apache.org/jira/browse/HADOOP-10087
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.0-beta, 2.2.0
> Environment: SUSE Linux Enterprise Server 11 (x86_64)
>Reporter: Yu Gao
>Assignee: Colin Patrick McCabe
> Labels: security
> Attachments: HADOOP-10087.001.patch, HADOOP-10087.002.patch
>
>
> When JniBasedUnixGroupsMappingWithFallback is used as the group mapping
> resolution provider, UserGroupInformation.getGroupNames() fails to return the
> primary group first in the list as documented.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HADOOP-10161) In the hadoop metrics framework,The default value of dmax is 0, which means gmond process will never delete this metirc although it is disappeared.
[ https://issues.apache.org/jira/browse/HADOOP-10161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang He updated HADOOP-10161: - Attachment: hadoop-metrics.properties > In the hadoop metrics framework,The default value of dmax is 0, which means > gmond process will never delete this metirc although it is disappeared. > --- > > Key: HADOOP-10161 > URL: https://issues.apache.org/jira/browse/HADOOP-10161 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Yang He >Priority: Minor > Attachments: HADOOP-10161_0_20131211.patch, > hadoop-metrics.properties, hadoop-metrics2.properties > > > The property of dmax is metric dead time , when one metrics is disappeared > ,so no more value of the metric will be emtic to the gmond,after 'dmax' times > ,then gmond will destroy the metric in memery . the default value is 0,which > means the gmond will never destroy the metric althought the metric is > disappeared ,the gmetad daemon also does not delete the rrdtool file. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10161) In the hadoop metrics framework,The default value of dmax is 0, which means gmond process will never delete this metirc although it is disappeared.
[ https://issues.apache.org/jira/browse/HADOOP-10161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang He updated HADOOP-10161: - Attachment: hadoop-metrics2.properties > In the hadoop metrics framework,The default value of dmax is 0, which means > gmond process will never delete this metirc although it is disappeared. > --- > > Key: HADOOP-10161 > URL: https://issues.apache.org/jira/browse/HADOOP-10161 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Yang He >Priority: Minor > Attachments: HADOOP-10161_0_20131211.patch, hadoop-metrics2.properties > > > The property of dmax is metric dead time , when one metrics is disappeared > ,so no more value of the metric will be emtic to the gmond,after 'dmax' times > ,then gmond will destroy the metric in memery . the default value is 0,which > means the gmond will never destroy the metric althought the metric is > disappeared ,the gmetad daemon also does not delete the rrdtool file. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10161) In the hadoop metrics framework,The default value of dmax is 0, which means gmond process will never delete this metirc although it is disappeared.
[ https://issues.apache.org/jira/browse/HADOOP-10161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang He updated HADOOP-10161: - Attachment: HADOOP-10161_0_20131211.patch > In the hadoop metrics framework,The default value of dmax is 0, which means > gmond process will never delete this metirc although it is disappeared. > --- > > Key: HADOOP-10161 > URL: https://issues.apache.org/jira/browse/HADOOP-10161 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Yang He >Priority: Minor > Attachments: HADOOP-10161_0_20131211.patch > > > The property of dmax is metric dead time , when one metrics is disappeared > ,so no more value of the metric will be emtic to the gmond,after 'dmax' times > ,then gmond will destroy the metric in memery . the default value is 0,which > means the gmond will never destroy the metric althought the metric is > disappeared ,the gmetad daemon also does not delete the rrdtool file. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10161) In the hadoop metrics framework,The default value of dmax is 0, which means gmond process will never delete this metirc although it is disappeared.
[ https://issues.apache.org/jira/browse/HADOOP-10161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang He updated HADOOP-10161: - Summary: In the hadoop metrics framework,The default value of dmax is 0, which means gmond process will never delete this metirc although it is disappeared. (was: In the hadoop metrics framework, the property of dmax in the ganglia configuration is the metrics dead time when the gmond destroy the metric in memery,The default value of it is 0, which means gmond process will never delete this metirc.) > In the hadoop metrics framework,The default value of dmax is 0, which means > gmond process will never delete this metirc although it is disappeared. > --- > > Key: HADOOP-10161 > URL: https://issues.apache.org/jira/browse/HADOOP-10161 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Yang He >Priority: Minor > > The property of dmax is metric dead time , when one metrics is disappeared > ,so no more value of the metric will be emtic to the gmond,after 'dmax' times > ,then gmond will destroy the metric in memery . the default value is 0,which > means the gmond will never destroy the metric althought the metric is > disappeared ,the gmetad daemon also does not delete the rrdtool file. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HADOOP-10161) In the hadoop metrics framework, the property of dmax in the ganglia configuration is the metrics dead time when the gmond destroy the metric in memery,The default va
Yang He created HADOOP-10161: Summary: In the hadoop metrics framework, the property of dmax in the ganglia configuration is the metrics dead time when the gmond destroy the metric in memery,The default value of it is 0, which means gmond process will never delete this metirc. Key: HADOOP-10161 URL: https://issues.apache.org/jira/browse/HADOOP-10161 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Yang He Priority: Minor The property of dmax is metric dead time , when one metrics is disappeared ,so no more value of the metric will be emtic to the gmond,after 'dmax' times ,then gmond will destroy the metric in memery . the default value is 0,which means the gmond will never destroy the metric althought the metric is disappeared ,the gmetad daemon also does not delete the rrdtool file. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10087) UserGroupInformation.getGroupNames() fails to return primary group first when JniBasedUnixGroupsMappingWithFallback is used
[ https://issues.apache.org/jira/browse/HADOOP-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845021#comment-13845021 ] Colin Patrick McCabe commented on HADOOP-10087: --- you're right. hadoop_user_info_free takes care of this. I removed the error handling from hadoop_user_info_getgroups. > UserGroupInformation.getGroupNames() fails to return primary group first when > JniBasedUnixGroupsMappingWithFallback is used > --- > > Key: HADOOP-10087 > URL: https://issues.apache.org/jira/browse/HADOOP-10087 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.1.0-beta, 2.2.0 > Environment: SUSE Linux Enterprise Server 11 (x86_64) >Reporter: Yu Gao >Assignee: Colin Patrick McCabe > Labels: security > Attachments: HADOOP-10087.001.patch, HADOOP-10087.002.patch > > > When JniBasedUnixGroupsMappingWithFallback is used as the group mapping > resolution provider, UserGroupInformation.getGroupNames() fails to return the > primary group first in the list as documented. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10087) UserGroupInformation.getGroupNames() fails to return primary group first when JniBasedUnixGroupsMappingWithFallback is used
[ https://issues.apache.org/jira/browse/HADOOP-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-10087: -- Attachment: HADOOP-10087.002.patch > UserGroupInformation.getGroupNames() fails to return primary group first when > JniBasedUnixGroupsMappingWithFallback is used > --- > > Key: HADOOP-10087 > URL: https://issues.apache.org/jira/browse/HADOOP-10087 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.1.0-beta, 2.2.0 > Environment: SUSE Linux Enterprise Server 11 (x86_64) >Reporter: Yu Gao >Assignee: Colin Patrick McCabe > Labels: security > Attachments: HADOOP-10087.001.patch, HADOOP-10087.002.patch > > > When JniBasedUnixGroupsMappingWithFallback is used as the group mapping > resolution provider, UserGroupInformation.getGroupNames() fails to return the > primary group first in the list as documented. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10142) Avoid groups lookup for unprivileged users such as "dr.who"
[
https://issues.apache.org/jira/browse/HADOOP-10142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844946#comment-13844946
]
Xi Fang commented on HADOOP-10142:
--
Hi [~cnauroth], thanks for pointing this out. I made a new patch
(HADOOP-10142-branch-1.2.patch) and tried to make the format as identical as
possible.
Thanks!
> Avoid groups lookup for unprivileged users such as "dr.who"
> ---
>
> Key: HADOOP-10142
> URL: https://issues.apache.org/jira/browse/HADOOP-10142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Vinay
>Assignee: Vinay
> Fix For: 2.3.0
>
> Attachments: HADOOP-10142-branch-1.2.patch,
> HADOOP-10142-branch-1.patch, HADOOP-10142.patch, HADOOP-10142.patch,
> HADOOP-10142.patch, HADOOP-10142.patch
>
>
> Reduce the logs generated by ShellBasedUnixGroupsMapping.
> For ex: Using WebHdfs from windows generates following log for each request
> {noformat}2013-12-03 11:34:56,589 WARN
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying
> to get groups for user dr.who
> org.apache.hadoop.util.Shell$ExitCodeException: id: dr.who: No such user
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
> at org.apache.hadoop.util.Shell.run(Shell.java:417)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:725)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:708)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:95)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1376)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:63)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3228)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4063)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4052)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:748)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getDirectoryListing(NamenodeWebHdfsMethods.java:715)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getListingStream(NamenodeWebHdfsMethods.java:727)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.access$400(NamenodeWebHdfsMethods.java:114)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:623)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:618)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getRoot(NamenodeWebHdfsMethods.java:586)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> c
[jira] [Updated] (HADOOP-10142) Avoid groups lookup for unprivileged users such as "dr.who"
[
https://issues.apache.org/jira/browse/HADOOP-10142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Fang updated HADOOP-10142:
-
Attachment: HADOOP-10142-branch-1.2.patch
> Avoid groups lookup for unprivileged users such as "dr.who"
> ---
>
> Key: HADOOP-10142
> URL: https://issues.apache.org/jira/browse/HADOOP-10142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Vinay
>Assignee: Vinay
> Fix For: 2.3.0
>
> Attachments: HADOOP-10142-branch-1.2.patch,
> HADOOP-10142-branch-1.patch, HADOOP-10142.patch, HADOOP-10142.patch,
> HADOOP-10142.patch, HADOOP-10142.patch
>
>
> Reduce the logs generated by ShellBasedUnixGroupsMapping.
> For ex: Using WebHdfs from windows generates following log for each request
> {noformat}2013-12-03 11:34:56,589 WARN
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying
> to get groups for user dr.who
> org.apache.hadoop.util.Shell$ExitCodeException: id: dr.who: No such user
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
> at org.apache.hadoop.util.Shell.run(Shell.java:417)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:725)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:708)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:95)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1376)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:63)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3228)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4063)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4052)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:748)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getDirectoryListing(NamenodeWebHdfsMethods.java:715)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getListingStream(NamenodeWebHdfsMethods.java:727)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.access$400(NamenodeWebHdfsMethods.java:114)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:623)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:618)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getRoot(NamenodeWebHdfsMethods.java:586)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHan
[jira] [Commented] (HADOOP-10087) UserGroupInformation.getGroupNames() fails to return primary group first when JniBasedUnixGroupsMappingWithFallback is used
[
https://issues.apache.org/jira/browse/HADOOP-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844894#comment-13844894
]
Andrew Wang commented on HADOOP-10087:
--
Patch looks good to me, thanks for fixing this Colin. Just one comment:
* Can we skip the {{goto error}} handling in hadoop_user_info_getgroups? I
think that free is already handled in JniBasedUnixGroupMapping and the main by
calling hadoop_user_info_free.
+1 once addressed.
> UserGroupInformation.getGroupNames() fails to return primary group first when
> JniBasedUnixGroupsMappingWithFallback is used
> ---
>
> Key: HADOOP-10087
> URL: https://issues.apache.org/jira/browse/HADOOP-10087
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.0-beta, 2.2.0
> Environment: SUSE Linux Enterprise Server 11 (x86_64)
>Reporter: Yu Gao
>Assignee: Colin Patrick McCabe
> Labels: security
> Attachments: HADOOP-10087.001.patch
>
>
> When JniBasedUnixGroupsMappingWithFallback is used as the group mapping
> resolution provider, UserGroupInformation.getGroupNames() fails to return the
> primary group first in the list as documented.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HADOOP-9639) truly shared cache for jars (jobjar/libjar)
[
https://issues.apache.org/jira/browse/HADOOP-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844802#comment-13844802
]
Sangjin Lee commented on HADOOP-9639:
-
{quote}
the upload mechanism assumes that rename() is atomic. This should be spelled
out, to avoid people trying to use blobstores as their cache infrastructure
{quote}
It’s a great point. I’ll spell out this requirement in the design doc so there
is no confusion.
{quote}
obviously: add a specific exception to indicate some kind of race condition
{quote}
I’m a little unsure as to which specific race you’re speaking of, or whether
you’re talking about a generic exception that can indicate any type of race
condition. Could you kindly clarify?
{quote}
The shared cache enabled flags are obviously things that admins would have to
right to set and make final in yarn-site.xml files, clients to handle this
without problems.
{quote}
That’s a good point. Whether the cluster has the shared cache enabled should be
a final config, and clients should not be able to override it, regardless of
whether they choose to use it or not. I’ll add that clarification.
Regarding your comment on the security (or the use case of mixing cached/shared
and uncached/private resources), I do think that that use case is supported. It
is true that using the shared cache for some resource means that resource is
public (available for others to see and use by identifying checksum). A close
analogy would be a jar that’s published to a maven repo via maven deploy.
However, it does not prevent a client from using the shared cache for some
resources (e.g. libjars) and the normal app-private or user-private distributed
cache for others (e.g. job-specific and sensitive configuration files), all
within the same job. The shared cache would merely enable you to take advantage
of public sharing of certain resources you’re comfortable sharing. I’ll add
that note to the design doc.
Does that answer your question?
{quote}
I (personally) think we should all just embrace the presence of 1+ ZK quorum on
the cluster...
{quote}
The main reason that we went without ZK was exactly what you mentioned; we did
not want to introduce the ZK requirement with a side feature such as this.
Having said that, I agree that using ZK for coordination would be more natural
than leveraging atomicity from the filesystem semantics. I also suspect that we
could arrive at a ZK-based solution with fairly straightforward changes to the
core idea. In the interest of time, however, I would propose proceeding with
the current design. We could certainly consider adding/replacing the
implementation with a ZK-based one in the next revision.
{quote}
HADOOP-9361 is attempting to formally define the semantics of a
Hadoop-compatible filesystem. If you could use that as the foundation
assumptions & perhaps even notation for defining your own behavior, the
analysis on P7 could be proved more rigorously
{quote}
I took a quick look at the formal definition you’re working on, and it looks
quite interesting. I will look at your definitions and notations to see if we
can describe a more formal proof and such at some point.
{quote}
The semantics of `happens-before comes from [Lamport78] Time, Clocks and the
Ordering of Events in a Distributed System, so should be used as the citation
as it is more appropriate than memory models of Java or out-of-order CPUs.
{quote}
Agreed on using that as a reference/citation. Will add that to the document.
{quote}
Script-wise, I've been evolving a [generic YARN service launcher, which is
nearly ready to submit as YARN-679: if the cleaner service were implemented as
a YARN service it could be invoked as a run-one command line, or deployed in a
YARN container service which provided cron-like services
{quote}
That looks interesting too. I’ll look at that when it gets merged, and
reconcile it with what we come up with here.
> truly shared cache for jars (jobjar/libjar)
> ---
>
> Key: HADOOP-9639
> URL: https://issues.apache.org/jira/browse/HADOOP-9639
> Project: Hadoop Common
> Issue Type: New Feature
> Components: filecache
>Affects Versions: 2.0.4-alpha
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: shared_cache_design.pdf, shared_cache_design_v2.pdf,
> shared_cache_design_v3.pdf, shared_cache_design_v4.pdf
>
>
> Currently there is the distributed cache that enables you to cache jars and
> files so that attempts from the same job can reuse them. However, sharing is
> limited with the distributed cache because it is normally on a per-job basis.
> On a large cluster, sometimes copying of jobjars and libjars becomes so
> prevalent that it consumes a large portion of the network bandwidth, not to
> speak of defeating the pur
[jira] [Created] (HADOOP-10160) Unregister NFS and Mount service when NFS gateway is shutting down
Brandon Li created HADOOP-10160: --- Summary: Unregister NFS and Mount service when NFS gateway is shutting down Key: HADOOP-10160 URL: https://issues.apache.org/jira/browse/HADOOP-10160 Project: Hadoop Common Issue Type: Bug Components: nfs Reporter: Brandon Li The services should be unregistered if the gateway is asked to shutdown gracefully. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-9640) RPC Congestion Control with FairCallQueue
[ https://issues.apache.org/jira/browse/HADOOP-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844743#comment-13844743 ] Chris Li commented on HADOOP-9640: -- bq. Add a new configuration in common called "hadoop.application.context" to HDFS. Other services that want to do the same thing can either use this same configuration and find another way to configure it. This information should be marshalled from the client to the server. The congestion control can be built based on that. Just to be clear, would an example be, 1. Cluster operator specifies ipc.8020.application.context = hadoop.yarn 2. Namenode sees this, knows to load the class that generates job IDs from the Connection/Call? Or were you thinking of physically adding the id into the RPC call itself, which would make the rpc call size larger, but is a cleaner solution (albeit one that the client could spoof). bq. Lets also make identities used for accounting configurable. They can be either based on "context", "user", "token", or "default". That way people who do not like the default configuration can make changes. Sounds like a good idea. > RPC Congestion Control with FairCallQueue > - > > Key: HADOOP-9640 > URL: https://issues.apache.org/jira/browse/HADOOP-9640 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0 >Reporter: Xiaobo Peng > Labels: hdfs, qos, rpc > Attachments: MinorityMajorityPerformance.pdf, > NN-denial-of-service-updated-plan.pdf, faircallqueue.patch, > faircallqueue2.patch, faircallqueue3.patch, faircallqueue4.patch, > faircallqueue5.patch, rpc-congestion-control-draft-plan.pdf > > > Several production Hadoop cluster incidents occurred where the Namenode was > overloaded and failed to respond. > We can improve quality of service for users during namenode peak loads by > replacing the FIFO call queue with a [Fair Call > Queue|https://issues.apache.org/jira/secure/attachment/12616864/NN-denial-of-service-updated-plan.pdf]. > (this plan supersedes rpc-congestion-control-draft-plan). > Excerpted from the communication of one incident, “The map task of a user was > creating huge number of small files in the user directory. Due to the heavy > load on NN, the JT also was unable to communicate with NN...The cluster > became responsive only once the job was killed.” > Excerpted from the communication of another incident, “Namenode was > overloaded by GetBlockLocation requests (Correction: should be getFileInfo > requests. the job had a bug that called getFileInfo for a nonexistent file in > an endless loop). All other requests to namenode were also affected by this > and hence all jobs slowed down. Cluster almost came to a grinding > halt…Eventually killed jobtracker to kill all jobs that are running.” > Excerpted from HDFS-945, “We've seen defective applications cause havoc on > the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories > (60k files) etc.” -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Resolved] (HADOOP-6239) Command-line for append
[
https://issues.apache.org/jira/browse/HADOOP-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee resolved HADOOP-6239.
Resolution: Duplicate
> Command-line for append
> ---
>
> Key: HADOOP-6239
> URL: https://issues.apache.org/jira/browse/HADOOP-6239
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
>
> Once append is implemented in hdfs, it would be nice if users can append
> files from a command-line.
> Either have a separate 'append' command or add '-append' option for 'put' and
> 'cp'
> Like,
> {noformat}
> % cat mylocalfile | hadoop dfs -put -append - /user/knoguchi/myfile('-'
> for stdin)
> % hadoop dfs -cp -append myhdfsfile1 myhdfsfile2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Created] (HADOOP-10159) Maven artifacts have incorrect javadoc jar
Sean Busbey created HADOOP-10159: Summary: Maven artifacts have incorrect javadoc jar Key: HADOOP-10159 URL: https://issues.apache.org/jira/browse/HADOOP-10159 Project: Hadoop Common Issue Type: Bug Components: build, documentation Affects Versions: 2.2.0, 2.0.6-alpha, 2.1.1-beta, 0.23.10, 0.23.9, 0.23.8, 2.0.4-alpha, 2.1.0-beta, 0.23.7, 0.23.6, 0.23.5, 2.0.3-alpha, 0.23.4, 2.0.2-alpha, 2.0.1-alpha, 2.0.0-alpha, 0.23.3 Reporter: Sean Busbey Several of the generated Maven artifacts have a (labeled as) javadoc jar that contain things besides javadocs. I checked the following, likely other artifacts are also impacted: * hadoop-client (all the 2.x contain shaded deps and no docs) * hadoop-common (all the 2.x and 0.23.3+ have both javadocs and shaded deps) * hadoop-hdfs (same as hadoop-common, plus classes) * hadoop-streaming (all 2.x and 0.23.3+ have compiled classes and javadocs) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10020) disable symlinks temporarily
[ https://issues.apache.org/jira/browse/HADOOP-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844712#comment-13844712 ] Chris Nauroth commented on HADOOP-10020: +1 for the branch-2 backport. Thanks for the patch, Colin! > disable symlinks temporarily > > > Key: HADOOP-10020 > URL: https://issues.apache.org/jira/browse/HADOOP-10020 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.2.0 >Reporter: Colin Patrick McCabe >Assignee: Sanjay Radia >Priority: Blocker > Fix For: 2.2.0 > > Attachments: > 0001-HADOOP-10020-addendum.-Fix-TestOfflineEditsViewer.patch, > HADOOP-10020-b2.001.patch, Hadoop-10020-2.patch, Hadoop-10020-3.patch, > Hadoop-10020-4-forBranch2.1beta.patch, Hadoop-10020-4.patch, > Hadoop-10020.patch > > > disable symlinks temporarily until we can make them production-ready in > Hadoop 2.3 -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10020) disable symlinks temporarily
[ https://issues.apache.org/jira/browse/HADOOP-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844701#comment-13844701 ] Colin Patrick McCabe commented on HADOOP-10020: --- Thanks for the careful review. bq. Question: are you aware of any downstream projects that are dependent on creating local symlinks using a RawLocalFileSystem? I'm not personally aware of any, but I also didn't do an exhaustive search. I don't think there are any, because HADOOP-9417 (support for symlink resolution in LocalFileSystem / RawLocalFileSystem) has never made it to a stable release. It wasn't in branch-2.3. > disable symlinks temporarily > > > Key: HADOOP-10020 > URL: https://issues.apache.org/jira/browse/HADOOP-10020 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 2.2.0 >Reporter: Colin Patrick McCabe >Assignee: Sanjay Radia >Priority: Blocker > Fix For: 2.2.0 > > Attachments: > 0001-HADOOP-10020-addendum.-Fix-TestOfflineEditsViewer.patch, > HADOOP-10020-b2.001.patch, Hadoop-10020-2.patch, Hadoop-10020-3.patch, > Hadoop-10020-4-forBranch2.1beta.patch, Hadoop-10020-4.patch, > Hadoop-10020.patch > > > disable symlinks temporarily until we can make them production-ready in > Hadoop 2.3 -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10020) disable symlinks temporarily
[
https://issues.apache.org/jira/browse/HADOOP-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844635#comment-13844635
]
Chris Nauroth commented on HADOOP-10020:
Hi Colin. I agree that we need to disable HDFS symlinks in branch-2.
The branch-2 patch looks good. I assume you've done a test run already. (We
don't have the benefit of Jenkins on this.)
I see that you've disabled symlinks in {{RawLocalFileSystem}} too. The
original patch hadn't done this. Question: are you aware of any downstream
projects that are dependent on creating local symlinks using a
{{RawLocalFileSystem}}? I'm not personally aware of any, but I also didn't do
an exhaustive search.
> disable symlinks temporarily
>
>
> Key: HADOOP-10020
> URL: https://issues.apache.org/jira/browse/HADOOP-10020
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sanjay Radia
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments:
> 0001-HADOOP-10020-addendum.-Fix-TestOfflineEditsViewer.patch,
> HADOOP-10020-b2.001.patch, Hadoop-10020-2.patch, Hadoop-10020-3.patch,
> Hadoop-10020-4-forBranch2.1beta.patch, Hadoop-10020-4.patch,
> Hadoop-10020.patch
>
>
> disable symlinks temporarily until we can make them production-ready in
> Hadoop 2.3
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HADOOP-10142) Avoid groups lookup for unprivileged users such as "dr.who"
[
https://issues.apache.org/jira/browse/HADOOP-10142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844589#comment-13844589
]
Chris Nauroth commented on HADOOP-10142:
Hi, [~xifang]. The backport looks good, but I just noticed a few minor
differences from the trunk patch in terms of whitespace and line wrapping.
Could you please make the whitespace and line wrapping identical to the trunk
patch? Sorry to nitpick on this, but if we keep the code as similar as
possible between the two branches, then it will make any future backports
easier. Thank you!
> Avoid groups lookup for unprivileged users such as "dr.who"
> ---
>
> Key: HADOOP-10142
> URL: https://issues.apache.org/jira/browse/HADOOP-10142
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Vinay
>Assignee: Vinay
> Fix For: 2.3.0
>
> Attachments: HADOOP-10142-branch-1.patch, HADOOP-10142.patch,
> HADOOP-10142.patch, HADOOP-10142.patch, HADOOP-10142.patch
>
>
> Reduce the logs generated by ShellBasedUnixGroupsMapping.
> For ex: Using WebHdfs from windows generates following log for each request
> {noformat}2013-12-03 11:34:56,589 WARN
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying
> to get groups for user dr.who
> org.apache.hadoop.util.Shell$ExitCodeException: id: dr.who: No such user
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
> at org.apache.hadoop.util.Shell.run(Shell.java:417)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:725)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:708)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:95)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1376)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:63)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3228)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4063)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4052)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:748)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getDirectoryListing(NamenodeWebHdfsMethods.java:715)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getListingStream(NamenodeWebHdfsMethods.java:727)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.access$400(NamenodeWebHdfsMethods.java:114)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:623)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:618)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:618)
> at
> org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getRoot(NamenodeWebHdfsMethods.java:586)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.mode
[jira] [Updated] (HADOOP-10020) disable symlinks temporarily
[
https://issues.apache.org/jira/browse/HADOOP-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-10020:
--
Attachment: HADOOP-10020-b2.001.patch
Here is a patch for branch-2, as discussed on the mailing list. I added a call
to {{FileSystem#enableSymlinks}} inside {{MiniDFSCluster}} to increase the
number of tests that don't need to manually add such an enable call.
> disable symlinks temporarily
>
>
> Key: HADOOP-10020
> URL: https://issues.apache.org/jira/browse/HADOOP-10020
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Sanjay Radia
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments:
> 0001-HADOOP-10020-addendum.-Fix-TestOfflineEditsViewer.patch,
> HADOOP-10020-b2.001.patch, Hadoop-10020-2.patch, Hadoop-10020-3.patch,
> Hadoop-10020-4-forBranch2.1beta.patch, Hadoop-10020-4.patch,
> Hadoop-10020.patch
>
>
> disable symlinks temporarily until we can make them production-ready in
> Hadoop 2.3
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HADOOP-9640) RPC Congestion Control with FairCallQueue
[ https://issues.apache.org/jira/browse/HADOOP-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844506#comment-13844506 ] Suresh Srinivas commented on HADOOP-9640: - I had in person meeting with [~chrili] on this. This is excellent work! bq. Parsing the MapReduce job name out of the DFSClient name is kind of an ugly hack. The client name also isn't that reliable since it's formed from the client's Configuration I had suggested this to [~chrili]. I realize that the configuration passed from MapReduce is actually a task ID. So the client name based on that will not be useful, unless we parse it to get the job ID. I agree that this is not the way the final solution should work. I propose adding some kind of configuration that can be passed to establish context in which access to services is happening. Currently this is done by mapreduce framework. It sets the configuration "" which gets used in forming DFSClient name. We could do the following to satisfy the various user requirements: # Add a new configuration in common called "hadoop.application.context" to HDFS. Other services that want to do the same thing can either use this same configuration and find another way to configure it. This information should be marshalled from the client to the server. The congestion control can be built based on that. # Lets also make identities used for accounting configurable. They can be either based on "context", "user", "token", or "default". That way people who do not like the default configuration can make changes. > RPC Congestion Control with FairCallQueue > - > > Key: HADOOP-9640 > URL: https://issues.apache.org/jira/browse/HADOOP-9640 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0 >Reporter: Xiaobo Peng > Labels: hdfs, qos, rpc > Attachments: MinorityMajorityPerformance.pdf, > NN-denial-of-service-updated-plan.pdf, faircallqueue.patch, > faircallqueue2.patch, faircallqueue3.patch, faircallqueue4.patch, > faircallqueue5.patch, rpc-congestion-control-draft-plan.pdf > > > Several production Hadoop cluster incidents occurred where the Namenode was > overloaded and failed to respond. > We can improve quality of service for users during namenode peak loads by > replacing the FIFO call queue with a [Fair Call > Queue|https://issues.apache.org/jira/secure/attachment/12616864/NN-denial-of-service-updated-plan.pdf]. > (this plan supersedes rpc-congestion-control-draft-plan). > Excerpted from the communication of one incident, “The map task of a user was > creating huge number of small files in the user directory. Due to the heavy > load on NN, the JT also was unable to communicate with NN...The cluster > became responsive only once the job was killed.” > Excerpted from the communication of another incident, “Namenode was > overloaded by GetBlockLocation requests (Correction: should be getFileInfo > requests. the job had a bug that called getFileInfo for a nonexistent file in > an endless loop). All other requests to namenode were also affected by this > and hence all jobs slowed down. Cluster almost came to a grinding > halt…Eventually killed jobtracker to kill all jobs that are running.” > Excerpted from HDFS-945, “We've seen defective applications cause havoc on > the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories > (60k files) etc.” -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-9867) org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
[ https://issues.apache.org/jira/browse/HADOOP-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844428#comment-13844428 ] Jason Lowe commented on HADOOP-9867: Thanks for updating the patch, Vinay. Comments: * I don't think LineReader is the best place to put split-specific code. Its sole purpose is to read lines from an input stream regardless of split boundaries. There are users of this class that are not necessarily processing splits. That's why I created SplitLineReader in MapReduce, and I believe this logic is better placed there. * I don't think we want to change Math.max(maxBytesToConsume(pos), maxLineLength)) to Math.min(maxBytesToConsume(pos), maxLineLength)). We need to be able to read a record past the end of the split when the record crosses the split boundary, but I think this change could allow a truncated record to be returned for an uncompressed input stream. e.g.: fillBuffer happens to return data only up to the end of the split, record is incomplete (no delimiter found), but maxBytesToConsume keeps us from filling the buffer with more data and a truncated record is returned. I think a more straightforward approach would be to have SplitLineReader be aware of the end of the split and track it in fillBuffer() much like CompressedLineSplitReader does. The fillBuffer callback already indicates whether we're mid-delimiter or not, so we can simply check if fillBuffer is being called after the split has ended but we're mid-delimiter. In that case we need an additional record. > org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record > delimiters well > -- > > Key: HADOOP-9867 > URL: https://issues.apache.org/jira/browse/HADOOP-9867 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 0.20.2, 0.23.9, 2.2.0 > Environment: CDH3U2 Redhat linux 5.7 >Reporter: Kris Geusebroek >Assignee: Vinay >Priority: Critical > Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch > > > Having defined a recorddelimiter of multiple bytes in a new InputFileFormat > sometimes has the effect of skipping records from the input. > This happens when the input splits are split off just after a > recordseparator. Starting point for the next split would be non zero and > skipFirstLine would be true. A seek into the file is done to start - 1 and > the text until the first recorddelimiter is ignored (due to the presumption > that this record is already handled by the previous maptask). Since the re > ord delimiter is multibyte the seek only got the last byte of the delimiter > into scope and its not recognized as a full delimiter. So the text is skipped > until the next delimiter (ignoring a full record!!) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844371#comment-13844371 ] Larry McCay commented on HADOOP-10150: -- Hi Yi - I am a bit confused by this latest comment. Can you please clarify "hadoop-crypto component was removed from latest patch as a result of Diceros emerging. "? Are you saying that initially you had a cipher provider implementation but have decided not to provide one since there is one available in yet another non-apache project? I don't believe that these sorts of external references are really appropriate. Neither Rhino or Diceros are a TLP or incubation project in Apache. Since it appears to be an intel specific implementation, it seems appropriate to remove it from the patch though. Do you plan to provide an all java implementation for this work? > Hadoop cryptographic file system > > > Key: HADOOP-10150 > URL: https://issues.apache.org/jira/browse/HADOOP-10150 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file > system.pdf > > > There is an increasing need for securing data when Hadoop customers use > various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so > on. > HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based > on HADOOP “FilterFileSystem” decorating DFS or other file systems, and > transparent to upper layer applications. It’s configurable, scalable and fast. > High level requirements: > 1.Transparent to and no modification required for upper layer > applications. > 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if > the wrapped file system supports them. > 3.Very high performance for encryption and decryption, they will not > become bottleneck. > 4.Can decorate HDFS and all other file systems in Hadoop, and will not > modify existing structure of file system, such as namenode and datanode > structure if the wrapped file system is HDFS. > 5.Admin can configure encryption policies, such as which directory will > be encrypted. > 6.A robust key management framework. > 7.Support Pread and append operations if the wrapped file system supports > them. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10151) Implement a Buffer-Based Chiper InputStream and OutputStream
[ https://issues.apache.org/jira/browse/HADOOP-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844350#comment-13844350 ] Yi Liu commented on HADOOP-10151: - This patch includes cipher InputStream and OutputStream interfaces, and buffer-based implementation DecryptorStream/EncryptorStream, they use Encryptor and Decryptor interfaces to do encryption/decryption. > Implement a Buffer-Based Chiper InputStream and OutputStream > > > Key: HADOOP-10151 > URL: https://issues.apache.org/jira/browse/HADOOP-10151 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10151.patch > > > Cipher InputStream and OuputStream are buffer-based, and the buffer is used > to cache the encrypted data or result. Cipher InputStream is used to read > encrypted data, and the result is plain text . Cipher OutputStream is used to > write plain data and result is encrypted data. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10150) Hadoop cryptographic file system
[ https://issues.apache.org/jira/browse/HADOOP-10150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844347#comment-13844347 ] Yi Liu commented on HADOOP-10150: - Create a sub task: HADOOP-10156: This JIRA defines Encryptor and Decryptor which are buffer-based interfaces for encryption and decryption. Standard javax.security.Cipher interface was employed to provide AES/CTR encryption/decryption implemention. In this way, one can replace javax.security.Cipher implementation by plug other JCE provider such as Diceros. Diceros was opensource project under Rhino project, implement a set of Cipher interface which provide high performance encyption/decryption compared to default JCE provider. The initial performance test result shows 20x speedup in CTR mode compared to default JCE provider in JDK 1.7_u45. Moreover, Encryptor/Decryptor interfaces implements a internal buffer to further improve the performance over javax.security.Cipher. hadoop-crypto component was removed from latest patch as a result of Diceros emerging. One can use "cfs.cipher.provider" to specify the JCE provider, for example, Diceros project link: https://github.com/intel-hadoop/diceros > Hadoop cryptographic file system > > > Key: HADOOP-10150 > URL: https://issues.apache.org/jira/browse/HADOOP-10150 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: CryptographicFileSystem.patch, HADOOP cryptographic file > system.pdf > > > There is an increasing need for securing data when Hadoop customers use > various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so > on. > HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based > on HADOOP “FilterFileSystem” decorating DFS or other file systems, and > transparent to upper layer applications. It’s configurable, scalable and fast. > High level requirements: > 1.Transparent to and no modification required for upper layer > applications. > 2.“Seek”, “PositionedReadable” are supported for input stream of CFS if > the wrapped file system supports them. > 3.Very high performance for encryption and decryption, they will not > become bottleneck. > 4.Can decorate HDFS and all other file systems in Hadoop, and will not > modify existing structure of file system, such as namenode and datanode > structure if the wrapped file system is HDFS. > 5.Admin can configure encryption policies, such as which directory will > be encrypted. > 6.A robust key management framework. > 7.Support Pread and append operations if the wrapped file system supports > them. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
[ https://issues.apache.org/jira/browse/HADOOP-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844349#comment-13844349 ] Yi Liu commented on HADOOP-10156: - This patch defines Encryptor and Decryptor which are buffer-based interfaces for encryption and decryption. Standard javax.security.Cipher interface was employed to provide AES/CTR encryption/decryption implemention. In this way, one can replace javax.security.Cipher implementation by plug other JCE provider such as Diceros. Diceros was opensource project under Rhino project, implement a set of Cipher interface which provide high performance encyption/decryption compared to default JCE provider. The initial performance test result shows 20x speedup in CTR mode compared to default JCE provider in JDK 1.7_u45. Moreover, Encryptor/Decryptor interfaces implements a internal buffer to further improve the performance over javax.security.Cipher. One can use "cfs.cipher.provider" to specify the JCE provider, for example, Diceros project link: https://github.com/intel-hadoop/diceros > Define Buffer-based Encryptor/Decryptor interfaces and provide implementation > for AES CTR. > -- > > Key: HADOOP-10156 > URL: https://issues.apache.org/jira/browse/HADOOP-10156 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10156.patch > > > Define encryptor and decryptor interfaces, and they are buffer-based to > improve performance. We use direct buffer to avoid bytes copy between JAVA > and native if there is JNI call. In this JIRA, AES CTR mode encryption and > decryption are implemented. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10106) Incorrect thread name in RPC log messages
[
https://issues.apache.org/jira/browse/HADOOP-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844346#comment-13844346
]
Hadoop QA commented on HADOOP-10106:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618045/hadoop_10106_trunk_2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3352//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3352//console
This message is automatically generated.
> Incorrect thread name in RPC log messages
> -
>
> Key: HADOOP-10106
> URL: https://issues.apache.org/jira/browse/HADOOP-10106
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ming Ma
>Priority: Minor
> Attachments: hadoop_10106_trunk.patch, hadoop_10106_trunk_2.patch
>
>
> INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8020:
> readAndProcess from client 10.115.201.46 threw exception
> org.apache.hadoop.ipc.RpcServerException: Unknown out of band call
> #-2147483647
> This is thrown by a reader thread, so the message should be like
> INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020:
> readAndProcess from client 10.115.201.46 threw exception
> org.apache.hadoop.ipc.RpcServerException: Unknown out of band call
> #-2147483647
> Another example is Responder.processResponse, which can also be called by
> handler thread. When that happend, the thread name should be the handler
> thread, not the responder thread.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HADOOP-10158) SPNEGO should work with multiple interfaces/SPNs.
[ https://issues.apache.org/jira/browse/HADOOP-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-10158: Assignee: Daryn Sharp > SPNEGO should work with multiple interfaces/SPNs. > - > > Key: HADOOP-10158 > URL: https://issues.apache.org/jira/browse/HADOOP-10158 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Kihwal Lee >Assignee: Daryn Sharp > > This is the list of internal servlets added by namenode. > | Name | Auth | Need to be accessible by end users | > | StartupProgressServlet | none | no | > | GetDelegationTokenServlet | internal SPNEGO | yes | > | RenewDelegationTokenServlet | internal SPNEGO | yes | > | CancelDelegationTokenServlet | internal SPNEGO | yes | > | FsckServlet | internal SPNEGO | yes | > | GetImageServlet | internal SPNEGO | no | > | ListPathsServlet | token in query | yes | > | FileDataServlet | token in query | yes | > | FileChecksumServlets | token in query | yes | > | ContentSummaryServlet | token in query | yes | > GetDelegationTokenServlet, RenewDelegationTokenServlet, > CancelDelegationTokenServlet and FsckServlet are accessed by end users, but > hard-coded to use the internal SPNEGO filter. > If a name node HTTP server binds to multiple external IP addresses, the > internal SPNEGO service principal name may not work with an address to which > end users are connecting. The current SPNEGO implementation in Hadoop is > limited to use a single service principal per filter. > If the underlying hadoop kerberos authentication handler cannot easily be > modified, we can at least create a separate auth filter for the end-user > facing servlets so that their service principals can be independently > configured. If not defined, it should fall back to the current behavior. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10158) SPNEGO should work with multiple interfaces/SPNs.
[ https://issues.apache.org/jira/browse/HADOOP-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-10158: Summary: SPNEGO should work with multiple interfaces/SPNs. (was: Some namenode servlets should not be internal.) > SPNEGO should work with multiple interfaces/SPNs. > - > > Key: HADOOP-10158 > URL: https://issues.apache.org/jira/browse/HADOOP-10158 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Kihwal Lee > > This is the list of internal servlets added by namenode. > | Name | Auth | Need to be accessible by end users | > | StartupProgressServlet | none | no | > | GetDelegationTokenServlet | internal SPNEGO | yes | > | RenewDelegationTokenServlet | internal SPNEGO | yes | > | CancelDelegationTokenServlet | internal SPNEGO | yes | > | FsckServlet | internal SPNEGO | yes | > | GetImageServlet | internal SPNEGO | no | > | ListPathsServlet | token in query | yes | > | FileDataServlet | token in query | yes | > | FileChecksumServlets | token in query | yes | > | ContentSummaryServlet | token in query | yes | > GetDelegationTokenServlet, RenewDelegationTokenServlet, > CancelDelegationTokenServlet and FsckServlet are accessed by end users, but > hard-coded to use the internal SPNEGO filter. > If a name node HTTP server binds to multiple external IP addresses, the > internal SPNEGO service principal name may not work with an address to which > end users are connecting. The current SPNEGO implementation in Hadoop is > limited to use a single service principal per filter. > If the underlying hadoop kerberos authentication handler cannot easily be > modified, we can at least create a separate auth filter for the end-user > facing servlets so that their service principals can be independently > configured. If not defined, it should fall back to the current behavior. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Moved] (HADOOP-10158) Some namenode servlets should not be internal.
[ https://issues.apache.org/jira/browse/HADOOP-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee moved HDFS-5628 to HADOOP-10158: --- Target Version/s: 2.4.0 (was: 3.0.0, 2.4.0) Affects Version/s: (was: 2.2.0) 2.2.0 Key: HADOOP-10158 (was: HDFS-5628) Project: Hadoop Common (was: Hadoop HDFS) > Some namenode servlets should not be internal. > -- > > Key: HADOOP-10158 > URL: https://issues.apache.org/jira/browse/HADOOP-10158 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Kihwal Lee > > This is the list of internal servlets added by namenode. > | Name | Auth | Need to be accessible by end users | > | StartupProgressServlet | none | no | > | GetDelegationTokenServlet | internal SPNEGO | yes | > | RenewDelegationTokenServlet | internal SPNEGO | yes | > | CancelDelegationTokenServlet | internal SPNEGO | yes | > | FsckServlet | internal SPNEGO | yes | > | GetImageServlet | internal SPNEGO | no | > | ListPathsServlet | token in query | yes | > | FileDataServlet | token in query | yes | > | FileChecksumServlets | token in query | yes | > | ContentSummaryServlet | token in query | yes | > GetDelegationTokenServlet, RenewDelegationTokenServlet, > CancelDelegationTokenServlet and FsckServlet are accessed by end users, but > hard-coded to use the internal SPNEGO filter. > If a name node HTTP server binds to multiple external IP addresses, the > internal SPNEGO service principal name may not work with an address to which > end users are connecting. The current SPNEGO implementation in Hadoop is > limited to use a single service principal per filter. > If the underlying hadoop kerberos authentication handler cannot easily be > modified, we can at least create a separate auth filter for the end-user > facing servlets so that their service principals can be independently > configured. If not defined, it should fall back to the current behavior. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10106) Incorrect thread name in RPC log messages
[ https://issues.apache.org/jira/browse/HADOOP-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HADOOP-10106: - Attachment: hadoop_10106_trunk_2.patch ok. I have opened a new jira https://issues.apache.org/jira/i#browse/HADOOP-10157 to move doRead method. Here is the new patch that just includes the thead name fix. > Incorrect thread name in RPC log messages > - > > Key: HADOOP-10106 > URL: https://issues.apache.org/jira/browse/HADOOP-10106 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ming Ma >Priority: Minor > Attachments: hadoop_10106_trunk.patch, hadoop_10106_trunk_2.patch > > > INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8020: > readAndProcess from client 10.115.201.46 threw exception > org.apache.hadoop.ipc.RpcServerException: Unknown out of band call > #-2147483647 > This is thrown by a reader thread, so the message should be like > INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: > readAndProcess from client 10.115.201.46 threw exception > org.apache.hadoop.ipc.RpcServerException: Unknown out of band call > #-2147483647 > Another example is Responder.processResponse, which can also be called by > handler thread. When that happend, the thread name should be the handler > thread, not the responder thread. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HADOOP-10157) move doRead method from IPC Listener class to IPC Reader class
Ming Ma created HADOOP-10157: Summary: move doRead method from IPC Listener class to IPC Reader class Key: HADOOP-10157 URL: https://issues.apache.org/jira/browse/HADOOP-10157 Project: Hadoop Common Issue Type: Bug Components: ipc Reporter: Ming Ma Priority: Minor Current doRead method belongs to Listener class. Semantically it is better to move doRead method from Listener class to Reader class. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10151) Implement a Buffer-Based Chiper InputStream and OutputStream
[ https://issues.apache.org/jira/browse/HADOOP-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10151: Summary: Implement a Buffer-Based Chiper InputStream and OutputStream (was: Implement a Buffer-Based Chiper InputStream and OutPutStream) > Implement a Buffer-Based Chiper InputStream and OutputStream > > > Key: HADOOP-10151 > URL: https://issues.apache.org/jira/browse/HADOOP-10151 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10151.patch > > > Cipher InputStream and OuputStream are buffer-based, and the buffer is used > to cache the encrypted data or result. Cipher InputStream is used to read > encrypted data, and the result is plain text . Cipher OutputStream is used to > write plain data and result is encrypted data. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
[ https://issues.apache.org/jira/browse/HADOOP-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10156: Description: Define encryptor and decryptor interfaces, and they are buffer-based to improve performance. We use direct buffer to avoid bytes copy between JAVA and native if there is JNI call. In this JIRA, AES CTR mode encryption and decryption are implemented. > Define Buffer-based Encryptor/Decryptor interfaces and provide implementation > for AES CTR. > -- > > Key: HADOOP-10156 > URL: https://issues.apache.org/jira/browse/HADOOP-10156 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10156.patch > > > Define encryptor and decryptor interfaces, and they are buffer-based to > improve performance. We use direct buffer to avoid bytes copy between JAVA > and native if there is JNI call. In this JIRA, AES CTR mode encryption and > decryption are implemented. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10107) Server.getNumOpenConnections may throw NPE
[ https://issues.apache.org/jira/browse/HADOOP-10107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844208#comment-13844208 ] Hudson commented on HADOOP-10107: - FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #816 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/816/]) HADOOP-10148. backport hadoop-10107 to branch-0.23 (Chen He via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1549648) * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java > Server.getNumOpenConnections may throw NPE > -- > > Key: HADOOP-10107 > URL: https://issues.apache.org/jira/browse/HADOOP-10107 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Kihwal Lee > Fix For: 2.4.0 > > Attachments: HADOOP-10107.patch > > > Found this in [build > #5440|https://builds.apache.org/job/PreCommit-HDFS-Build/5440/testReport/junit/org.apache.hadoop.hdfs.server.blockmanagement/TestUnderReplicatedBlocks/testSetrepIncWithUnderReplicatedBlocks/] > Caused by: java.lang.NullPointerException > at org.apache.hadoop.ipc.Server.getNumOpenConnections(Server.java:2434) > at > org.apache.hadoop.ipc.metrics.RpcMetrics.numOpenConnections(RpcMetrics.java:74) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10148) backport hadoop-10107 to branch-0.23
[ https://issues.apache.org/jira/browse/HADOOP-10148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844210#comment-13844210 ] Hudson commented on HADOOP-10148: - FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #816 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/816/]) HADOOP-10148. backport hadoop-10107 to branch-0.23 (Chen He via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1549648) * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java > backport hadoop-10107 to branch-0.23 > > > Key: HADOOP-10148 > URL: https://issues.apache.org/jira/browse/HADOOP-10148 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc >Affects Versions: 0.23.10 >Reporter: Chen He >Assignee: Chen He >Priority: Minor > Fix For: 0.23.11 > > Attachments: HADOOP-10148.patch > > > Found this in [build > #5440|https://builds.apache.org/job/PreCommit-HDFS-Build/5440/testReport/junit/org.apache.hadoop.hdfs.server.blockmanagement/TestUnderReplicatedBlocks/testSetrepIncWithUnderReplicatedBlocks/] > Caused by: java.lang.NullPointerException > at org.apache.hadoop.ipc.Server.getNumOpenConnections(Server.java:2434) > at > org.apache.hadoop.ipc.metrics.RpcMetrics.numOpenConnections(RpcMetrics.java:74) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
[
https://issues.apache.org/jira/browse/HADOOP-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844150#comment-13844150
]
Hadoop QA commented on HADOOP-10156:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618011/HADOOP-10156.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3351//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3351//console
This message is automatically generated.
> Define Buffer-based Encryptor/Decryptor interfaces and provide implementation
> for AES CTR.
> --
>
> Key: HADOOP-10156
> URL: https://issues.apache.org/jira/browse/HADOOP-10156
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 3.0.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Labels: rhino
> Fix For: 3.0.0
>
> Attachments: HADOOP-10156.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Work started] (HADOOP-10151) Implement a Buffer-Based Chiper InputStream and OutPutStream
[ https://issues.apache.org/jira/browse/HADOOP-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-10151 started by Yi Liu. > Implement a Buffer-Based Chiper InputStream and OutPutStream > > > Key: HADOOP-10151 > URL: https://issues.apache.org/jira/browse/HADOOP-10151 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10151.patch > > > Cipher InputStream and OuputStream are buffer-based, and the buffer is used > to cache the encrypted data or result. Cipher InputStream is used to read > encrypted data, and the result is plain text . Cipher OutputStream is used to > write plain data and result is encrypted data. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10151) Implement a Buffer-Based Chiper InputStream and OutPutStream
[ https://issues.apache.org/jira/browse/HADOOP-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10151: Attachment: HADOOP-10151.patch > Implement a Buffer-Based Chiper InputStream and OutPutStream > > > Key: HADOOP-10151 > URL: https://issues.apache.org/jira/browse/HADOOP-10151 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10151.patch > > > Cipher InputStream and OuputStream are buffer-based, and the buffer is used > to cache the encrypted data or result. Cipher InputStream is used to read > encrypted data, and the result is plain text . Cipher OutputStream is used to > write plain data and result is encrypted data. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
[ https://issues.apache.org/jira/browse/HADOOP-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10156: Labels: rhino (was: ) Status: Patch Available (was: Open) > Define Buffer-based Encryptor/Decryptor interfaces and provide implementation > for AES CTR. > -- > > Key: HADOOP-10156 > URL: https://issues.apache.org/jira/browse/HADOOP-10156 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Labels: rhino > Fix For: 3.0.0 > > Attachments: HADOOP-10156.patch > > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
[ https://issues.apache.org/jira/browse/HADOOP-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HADOOP-10156: Attachment: HADOOP-10156.patch > Define Buffer-based Encryptor/Decryptor interfaces and provide implementation > for AES CTR. > -- > > Key: HADOOP-10156 > URL: https://issues.apache.org/jira/browse/HADOOP-10156 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: 3.0.0 > > Attachments: HADOOP-10156.patch > > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HADOOP-10156) Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR.
Yi Liu created HADOOP-10156: --- Summary: Define Buffer-based Encryptor/Decryptor interfaces and provide implementation for AES CTR. Key: HADOOP-10156 URL: https://issues.apache.org/jira/browse/HADOOP-10156 Project: Hadoop Common Issue Type: Sub-task Components: security Affects Versions: 3.0.0 Reporter: Yi Liu Assignee: Yi Liu Fix For: 3.0.0 -- This message was sent by Atlassian JIRA (v6.1.4#6159)
