[jira] [Resolved] (HADOOP-16590) IBM Java has deprecated OS login module classes and OS principal classes.
[
https://issues.apache.org/jira/browse/HADOOP-16590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-16590.
Fix Version/s: 3.3.0
Resolution: Fixed
[~nmarion] Thanks for the patch. I merged pull request 1484 to trunk.
> IBM Java has deprecated OS login module classes and OS principal classes.
> -
>
> Key: HADOOP-16590
> URL: https://issues.apache.org/jira/browse/HADOOP-16590
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Nicholas Marion
>Priority: Major
> Fix For: 3.3.0
>
>
> When building applications that rely on hadoop-commons and using IBM Java,
> errors such as `{{Exception in thread "main" java.io.IOException: failure to
> login}}` and `{{Unable to find JAAS
> classes:com.ibm.security.auth.LinuxPrincipal}}` can be seen.
> IBM Java has deprecated the following OS Login Module classes:
> {code:java}
> com.ibm.security.auth.module.Win64LoginModule
> com.ibm.security.auth.module.NTLoginModule
> com.ibm.security.auth.module.AIX64LoginModule
> com.ibm.security.auth.module.AIXLoginModule
> com.ibm.security.auth.module.LinuxLoginModule
> {code}
> and replaced with
> {code:java}
> com.ibm.security.auth.module.JAASLoginModule{code}
> IBM Java has deprecated the following OS Principal classes:
>
> {code:java}
> com.ibm.security.auth.UsernamePrincipal
> com.ibm.security.auth.NTUserPrincipal
> com.ibm.security.auth.AIXPrincipal
> com.ibm.security.auth.LinuxPrincipal
> {code}
> and replaced with
> {code:java}
> com.ibm.security.auth.UsernamePrincipal{code}
> Older issue HADOOP-15765 has same issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16614) Missing leveldbjni package of aarch64 platform
[ https://issues.apache.org/jira/browse/HADOOP-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-16614. Fix Version/s: 3.3.0 Resolution: Fixed Thank you [~seanlau] for the patch. +1 merged to trunk. > Missing leveldbjni package of aarch64 platform > -- > > Key: HADOOP-16614 > URL: https://issues.apache.org/jira/browse/HADOOP-16614 > Project: Hadoop Common > Issue Type: Bug >Reporter: liusheng >Priority: Major > Fix For: 3.3.0 > > > Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of > *org.fusesource.leveldbjni* group, but it cannot support ARM platform. > see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] > Because the leveldbjni community is inactivity and the code > ([https://github.com/fusesource/leveldbjni]) didn't updated a long time.I > will build the leveldbjni package of aarch64 platform, and upload it with > other platform packages of *org.fusesource.leveldbjni* to a new > *org.openlabtesting.leveldbjni* maven repo. In hadoop code, I will add a new > profile aarch64 for for automatically select the > *org.openlabtesting.leveldbjni* artifact group and using the aarch64 package > of leveldbjni when running on ARM server, this approach has no effect on > current code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16463) Migrate away from jsr305 jar
Eric Yang created HADOOP-16463: -- Summary: Migrate away from jsr305 jar Key: HADOOP-16463 URL: https://issues.apache.org/jira/browse/HADOOP-16463 Project: Hadoop Common Issue Type: Bug Reporter: Eric Yang JSR305 defines several annotations that is prefixed with javax packages. "javax.annotation.Nullable" is also used by findbugs to suppress code style warnings. "javax" is a reserved package name according to Oracle license agreement. Application can not use and ship these dependencies along with a JRE without violating the Oracle licence agreement. From JDK 9 and newer, [SecurityException|http://blog.anthavio.net/2013/11/how-many-javaxannotation-jars-is-out.html] would be thrown for attempt to run signed code JSR250 + JSR305. Many developers have look for solution to address [JSR305 annotation issue|https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use], but there is no good solution at this time. One possible solution is to use findbugsExcludeFile.xml to define the actual suppression and this will allow Hadoop to ship without jsr305 dependency. See other references: [Guava jsr305 issue|https://github.com/google/guava/issues/2960] [HBase jsr305 issue|https://issues.apache.org/jira/browse/HBASE-16321] This looks like a issue that needs to be addressed if we want to work in newer version of Java environment. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16457) Hadoop does not work without Kerberos for simple security
Eric Yang created HADOOP-16457:
--
Summary: Hadoop does not work without Kerberos for simple security
Key: HADOOP-16457
URL: https://issues.apache.org/jira/browse/HADOOP-16457
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Eric Yang
Assignee: Prabhu Joseph
When http filter initializers is setup to use StaticUserWebFilter, AuthFilter
is still setup. This prevents datanode to talk to namenode.
Error message in namenode logs:
{code}
2019-07-24 15:47:38,038 INFO org.apache.hadoop.hdfs.DFSUtil: Filter
initializers set :
org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
2019-07-24 16:06:26,212 WARN
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization failed for hdfs (auth:SIMPLE) for protocol=interface
org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol: this service is only
accessible by dn/eyang-5.openstacklo...@example.com
{code}
Errors in datanode log:
{code}
2019-07-24 16:07:01,253 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Problem connecting to server: eyang-1.openstacklocal/172.26.111.17:9000
{code}
The logic in HADOOP-16354 always added AuthFilter regardless which http filter
initializer is chosen. This is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16095) Support impersonation for AuthenticationFilter
[
https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-16095.
Resolution: Fixed
All related tasks have been closed, mark this as resolved.
Thank you, [~Prabhu Joseph] for the patches.
Thank you, [~lmccay], [~sunilg], and [~jojochuang] for input and reviews.
> Support impersonation for AuthenticationFilter
> --
>
> Key: HADOOP-16095
> URL: https://issues.apache.org/jira/browse/HADOOP-16095
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16095.004.patch
>
>
> External services or YARN service may need to call into WebHDFS or YARN REST
> API on behave of the user using web protocols. It would be good to support
> impersonation mechanism in AuthenticationFilter or similar extensions. The
> general design is similar to UserGroupInformation.doAs in RPC layer.
> The calling service credential is verified as a proxy user coming from a
> trusted host verifying Hadoop proxy user ACL on the server side. If proxy
> user ACL allows proxy user to become doAs user. HttpRequest object will
> report REMOTE_USER as doAs user. This feature enables web application logic
> to be written with minimal changes to call Hadoop API with
> UserGroupInformation.doAs() wrapper.
> h2. HTTP Request
> A few possible options:
> 1. Using query parameter to pass doAs user:
> {code:java}
> POST /service?doAs=foobar
> Authorization: [proxy user Kerberos token]
> {code}
> 2. Use HTTP Header to pass doAs user:
> {code:java}
> POST /service
> Authorization: [proxy user Kerberos token]
> x-hadoop-doas: foobar
> {code}
> h2. HTTP Response
> 403 - Forbidden (Including impersonation is not allowed)
> h2. Proxy User ACL requirement
> Proxy user kerberos token maps to a service principal, such as
> yarn/host1.example.com. The host part of the credential and HTTP request
> origin are both validated with *hadoop.proxyuser.yarn.hosts* ACL. doAs user
> group membership or identity is checked with either
> *hadoop.proxyuser.yarn.groups* or *hadoop.proxyuser.yarn.users*. This governs
> the caller is coming from authorized host and belong to authorized group.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16356) Distcp with webhdfs is not working with ProxyUserAuthenticationFilter or AuthenticationFilter
[
https://issues.apache.org/jira/browse/HADOOP-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-16356.
Resolution: Duplicate
> Distcp with webhdfs is not working with ProxyUserAuthenticationFilter or
> AuthenticationFilter
> -
>
> Key: HADOOP-16356
> URL: https://issues.apache.org/jira/browse/HADOOP-16356
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Prabhu Joseph
>Priority: Major
>
> When distcp is running with webhdfs://, there is no delegation token issued
> to mapreduce task because mapreduce task does not have kerberos tgt ticket.
> This stack trace was thrown when mapreduce task contacts webhdfs:
> {code}
> Error: org.apache.hadoop.security.AccessControlException: Authentication
> required
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:492)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:136)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:760)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:835)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:663)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:701)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:697)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1095)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1106)
> at org.apache.hadoop.tools.mapred.CopyMapper.setup(CopyMapper.java:124)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
> {code}
> There are two proposals:
> 1. Have a API to issue delegation token to pass along to webhdfs to maintain
> backward compatibility.
> 2. Have mapreduce task login to kerberos then perform webhdfs fetching.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16095) Support impersonation for AuthenticationFilter
[
https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HADOOP-16095:
Found an issue with distcp backward compatibility, opened HADOOP-16356 to track
required changes.
> Support impersonation for AuthenticationFilter
> --
>
> Key: HADOOP-16095
> URL: https://issues.apache.org/jira/browse/HADOOP-16095
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16095.004.patch
>
>
> External services or YARN service may need to call into WebHDFS or YARN REST
> API on behave of the user using web protocols. It would be good to support
> impersonation mechanism in AuthenticationFilter or similar extensions. The
> general design is similar to UserGroupInformation.doAs in RPC layer.
> The calling service credential is verified as a proxy user coming from a
> trusted host verifying Hadoop proxy user ACL on the server side. If proxy
> user ACL allows proxy user to become doAs user. HttpRequest object will
> report REMOTE_USER as doAs user. This feature enables web application logic
> to be written with minimal changes to call Hadoop API with
> UserGroupInformation.doAs() wrapper.
> h2. HTTP Request
> A few possible options:
> 1. Using query parameter to pass doAs user:
> {code:java}
> POST /service?doAs=foobar
> Authorization: [proxy user Kerberos token]
> {code}
> 2. Use HTTP Header to pass doAs user:
> {code:java}
> POST /service
> Authorization: [proxy user Kerberos token]
> x-hadoop-doas: foobar
> {code}
> h2. HTTP Response
> 403 - Forbidden (Including impersonation is not allowed)
> h2. Proxy User ACL requirement
> Proxy user kerberos token maps to a service principal, such as
> yarn/host1.example.com. The host part of the credential and HTTP request
> origin are both validated with *hadoop.proxyuser.yarn.hosts* ACL. doAs user
> group membership or identity is checked with either
> *hadoop.proxyuser.yarn.groups* or *hadoop.proxyuser.yarn.users*. This governs
> the caller is coming from authorized host and belong to authorized group.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16356) Distcp with webhdfs is not working with ProxyUserAuthenticationFilter or AuthenticationFilter
Eric Yang created HADOOP-16356:
--
Summary: Distcp with webhdfs is not working with
ProxyUserAuthenticationFilter or AuthenticationFilter
Key: HADOOP-16356
URL: https://issues.apache.org/jira/browse/HADOOP-16356
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Eric Yang
When distcp is running with webhdfs://, there is no delegation token issued to
mapreduce task because mapreduce task does not have kerberos tgt ticket.
This stack trace was thrown when mapreduce task contacts webhdfs:
{code}
Error: org.apache.hadoop.security.AccessControlException: Authentication
required
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:492)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:136)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.connect(WebHdfsFileSystem.java:760)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:835)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:663)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:701)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:697)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:1095)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:1106)
at org.apache.hadoop.tools.mapred.CopyMapper.setup(CopyMapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
{code}
There are two proposals:
1. Have a API to issue delegation token to pass along to webhdfs to maintain
backward compatibility.
2. Have mapreduce task login to kerberos then perform webhdfs fetching.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16095) Support impersonation for AuthenticationFilter
[
https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-16095.
Resolution: Fixed
Fix Version/s: 3.3.0
The current implementation is based on option 1. All sub-tasks have been
close. Mark this issue as resolved.
> Support impersonation for AuthenticationFilter
> --
>
> Key: HADOOP-16095
> URL: https://issues.apache.org/jira/browse/HADOOP-16095
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16095.004.patch
>
>
> External services or YARN service may need to call into WebHDFS or YARN REST
> API on behave of the user using web protocols. It would be good to support
> impersonation mechanism in AuthenticationFilter or similar extensions. The
> general design is similar to UserGroupInformation.doAs in RPC layer.
> The calling service credential is verified as a proxy user coming from a
> trusted host verifying Hadoop proxy user ACL on the server side. If proxy
> user ACL allows proxy user to become doAs user. HttpRequest object will
> report REMOTE_USER as doAs user. This feature enables web application logic
> to be written with minimal changes to call Hadoop API with
> UserGroupInformation.doAs() wrapper.
> h2. HTTP Request
> A few possible options:
> 1. Using query parameter to pass doAs user:
> {code:java}
> POST /service?doAs=foobar
> Authorization: [proxy user Kerberos token]
> {code}
> 2. Use HTTP Header to pass doAs user:
> {code:java}
> POST /service
> Authorization: [proxy user Kerberos token]
> x-hadoop-doas: foobar
> {code}
> h2. HTTP Response
> 403 - Forbidden (Including impersonation is not allowed)
> h2. Proxy User ACL requirement
> Proxy user kerberos token maps to a service principal, such as
> yarn/host1.example.com. The host part of the credential and HTTP request
> origin are both validated with *hadoop.proxyuser.yarn.hosts* ACL. doAs user
> group membership or identity is checked with either
> *hadoop.proxyuser.yarn.groups* or *hadoop.proxyuser.yarn.users*. This governs
> the caller is coming from authorized host and belong to authorized group.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16325) Add ability to run pytthon test and build docker in docker in start-build-env.sh
Eric Yang created HADOOP-16325: -- Summary: Add ability to run pytthon test and build docker in docker in start-build-env.sh Key: HADOOP-16325 URL: https://issues.apache.org/jira/browse/HADOOP-16325 Project: Hadoop Common Issue Type: Improvement Reporter: Eric Yang Ozone uses docker-compose, pytest and blockade to simulate network failure. It would be great to have ability to run these integration test tools in the developer docker environment. Ozone and YARN have optional profiles to build docker images using -Pdocker. It would be a good addition to have ability to build docker image inside the developer docker environment as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter
Eric Yang created HADOOP-16314: -- Summary: Make sure all end point URL is covered by the same AuthenticationFilter Key: HADOOP-16314 URL: https://issues.apache.org/jira/browse/HADOOP-16314 Project: Hadoop Common Issue Type: Improvement Reporter: Eric Yang In the enclosed spreadsheet, it shows the list of web applications deployed by Hadoop, and filters applied to each entry point. Hadoop web protocol impersonation has been inconsistent. Most of entry point do not support ?doAs parameter. This creates problem for secure gateway like Knox to proxy Hadoop web interface on behave of the end user. When the receiving end does not check for ?doAs flag, web interface would be accessed using proxy user credential. This can lead to all kind of security holes using path traversal to exploit Hadoop. In HADOOP-16287, ProxyUserAuthenticationFilter is proposed as solution to solve the web impersonation problem. This task is to track changes required in Hadoop code base to apply authentication filter globally for each of the web service port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16236) Update C code to run with c99 standard
Eric Yang created HADOOP-16236: -- Summary: Update C code to run with c99 standard Key: HADOOP-16236 URL: https://issues.apache.org/jira/browse/HADOOP-16236 Project: Hadoop Common Issue Type: Improvement Reporter: Eric Yang Several files in Hadoop common uses gnu99 standard for writing code. This includes: - inline assembly in bulk_crc32_x86.c - assignment between function pointer and ‘void *’ in SnappyCompressor.c. Apache httpd server and important Apache projects are using [c89 standard|https://github.com/apache/httpd/blob/16211a8cdd52251cb7ae251e693b9053fb545e20/configure.in#L636] for coding. It would be nice if Hadoop can compile using c99 standard to avoid grey area with GPL license. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16106) hadoop-aws project javadoc does not compile
[
https://issues.apache.org/jira/browse/HADOOP-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-16106.
Resolution: Duplicate
This is a duplicate of HADOOP-16098.
> hadoop-aws project javadoc does not compile
> ---
>
> Key: HADOOP-16106
> URL: https://issues.apache.org/jira/browse/HADOOP-16106
> Project: Hadoop Common
> Issue Type: Bug
> Components: hadoop-aws
>Reporter: Eric Yang
>Assignee: Steve Loughran
>Priority: Trivial
>
> Apache Hadoop Amazon Web Services support maven javadoc doesn't build
> properly because two non-html friendly characters in javadoc comments.
> {code}
> [ERROR]
> /home/eyang/test/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InternalConstants.java:31:
> error: bad HTML entity
> [ERROR] * Please don't refer to these outside of this module & its tests.
> [ERROR] ^
> [ERROR]
> /home/eyang/test/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AReadOpContext.java:115:
> error: bad use of '>'
> [ERROR]* @return a value >= 0
> [ERROR] ^
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16106) hadoop-aws project javadoc does not compile
Eric Yang created HADOOP-16106:
--
Summary: hadoop-aws project javadoc does not compile
Key: HADOOP-16106
URL: https://issues.apache.org/jira/browse/HADOOP-16106
Project: Hadoop Common
Issue Type: Bug
Components: hadoop-aws
Reporter: Eric Yang
Assignee: Steve Loughran
Apache Hadoop Amazon Web Services support maven javadoc doesn't build properly
because two non-html friendly characters in javadoc comments.
{code}
[ERROR]
/home/eyang/test/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InternalConstants.java:31:
error: bad HTML entity
[ERROR] * Please don't refer to these outside of this module & its tests.
[ERROR] ^
[ERROR]
/home/eyang/test/hadoop/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AReadOpContext.java:115:
error: bad use of '>'
[ERROR]* @return a value >= 0
[ERROR] ^
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16095) Support impersonation for AuthenticationFilter
Eric Yang created HADOOP-16095:
--
Summary: Support impersonation for AuthenticationFilter
Key: HADOOP-16095
URL: https://issues.apache.org/jira/browse/HADOOP-16095
Project: Hadoop Common
Issue Type: New Feature
Components: security
Reporter: Eric Yang
Assignee: Eric Yang
External services or YARN service may need to call into WebHDFS or YARN REST
API on behave of the user using web protocols. It would be good to support
impersonation mechanism in AuthenticationFilter or similar extensions. The
general design is similar to UserGroupInformation.doAs in RPC layer.
The calling service credential is verified as a proxy user coming from a
trusted host using Hadoop proxy user ACL. If proxy user ACL allows proxy user
to become doAs user. HttpRequest object will report REMOTE_USER as doAs user.
This feature enables web application logic to be written with minimal changes
to call Hadoop API with UserGroupInformation wrapper.
h2. Http Request
A few possible options:
1. Using query parameter to pass doAs user:
{code}
POST /service?doAs=foobar
Authorization: [proxy user Kerberos token]
{code}
2. Use HTTP Header to pass doAs user:
{code}
POST /service
Authorization: [proxy user Kerberos token]
x-hadoop-doas: foobar
{code}
h2. Response
403 - Forbidden (Including impersonation is not allowed)
h2. Proxy User ACL requirement
Proxy user kerberos token maps to a service principal, such as
yarn/host1.example.com. The host part of the credential and HTTP request
origin are both validated with *hadoop.proxyuser.yarn.hosts* ACL. doAs user
group membership or identity is checked with either
*hadoop.proxyuser.yarn.groups* or *hadoop.proxyuser.yarn.users*. This governs
the caller is coming from authorized host and belong to authorized group.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16094) AuthenticationFilter can trigger NullPointerException in KerberosName class
Eric Yang created HADOOP-16094:
--
Summary: AuthenticationFilter can trigger NullPointerException in
KerberosName class
Key: HADOOP-16094
URL: https://issues.apache.org/jira/browse/HADOOP-16094
Project: Hadoop Common
Issue Type: Bug
Reporter: Eric Yang
Hadoop AuthenticationFilter example can fail with NullPointerException if
auth_to_local rules has not been parsed from Configuration object. This can
happen if the web application does not have any initialization code that leads
to triggering: UserGroupInformation.initialize(conf, boolean);
Stacktrace:
{code}
2019-02-05 20:08:05,668 [http-bio-8080-exec-11] DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter-
Authentication exception: java.lang.NullPointerException
org.apache.hadoop.security.authentication.client.AuthenticationException:
java.lang.NullPointerException
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:315)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:422)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.runWithPrincipal(KerberosAuthenticationHandler.java:352)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.access$000(KerberosAuthenticationHandler.java:64)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:304)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:300)
... 18 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15959) revert HADOOP-12751
[ https://issues.apache.org/jira/browse/HADOOP-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-15959. Resolution: Fixed The failed registry DNS unit test has been addressed in HADOOP-16031. Hence, close as resolved again. > revert HADOOP-12751 > --- > > Key: HADOOP-15959 > URL: https://issues.apache.org/jira/browse/HADOOP-15959 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 3.2.0, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.2.0, 2.7.8, 3.0.4, 3.1.2, 2.8.6, 2.9.3 > > Attachments: HADOOP-15959-001.patch, HADOOP-15959-branch-2-002.patch, > HADOOP-15959-branch-2.7-003.patch > > > HADOOP-12751 doesn't quite work right. Revert. > (this patch is so jenkins can do the test runs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15996) Plugin interface to support more complex usernames in Hadoop
Eric Yang created HADOOP-15996: -- Summary: Plugin interface to support more complex usernames in Hadoop Key: HADOOP-15996 URL: https://issues.apache.org/jira/browse/HADOOP-15996 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Eric Yang Hadoop does not allow support of @ character in username in recent security mailing list vote to revert HADOOP-12751. Hadoop auth_to_local rule must match to authorize user to login to Hadoop cluster. This design does not work well in multi-realm environment where identical username between two realms do not map to the same user. There is also possibility that lossy regex can incorrect map users. In the interest of supporting multi-realms, it maybe preferred to pass principal name without rewrite to uniquely distinguish users. This jira is to revisit if Hadoop can support full principal names without rewrite and provide a plugin to override Hadoop's default implementation of auth_to_local for multi-realm use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15922) DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL
[
https://issues.apache.org/jira/browse/HADOOP-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HADOOP-15922:
> DelegationTokenAuthenticationFilter get wrong doAsUser since it does not
> decode URL
> ---
>
> Key: HADOOP-15922
> URL: https://issues.apache.org/jira/browse/HADOOP-15922
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15922.001.patch, HADOOP-15922.002.patch,
> HADOOP-15922.003.patch, HADOOP-15922.004.patch, HADOOP-15922.005.patch
>
>
> DelegationTokenAuthenticationFilter get wrong doAsUser when proxy user from
> client is complete kerberos name (e.g., user/hostn...@realm.com, actually it
> is acceptable), because DelegationTokenAuthenticationFilter does not decode
> DOAS parameter in URL which is encoded by {{URLEncoder}} at client.
> e.g. KMS as example:
> a. KMSClientProvider creates connection to KMS Server using
> DelegationTokenAuthenticatedURL#openConnection.
> b. If KMSClientProvider is a doAsUser, KMSClientProvider will put {{doas}}
> with url encoded user as one parameter of http request.
> {code:java}
> // proxyuser
> if (doAs != null) {
> extraParams.put(DO_AS, URLEncoder.encode(doAs, "UTF-8"));
> }
> {code}
> c. when KMS server receives the request, it does not decode the proxy user.
> As result, KMS Server will get the wrong proxy user if this proxy user is
> complete Kerberos Name or it includes some special character. Some other
> authentication and authorization exception will throws next to it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15896) Refine Kerberos based AuthenticationHandler to check proxyuser ACL
Eric Yang created HADOOP-15896:
--
Summary: Refine Kerberos based AuthenticationHandler to check
proxyuser ACL
Key: HADOOP-15896
URL: https://issues.apache.org/jira/browse/HADOOP-15896
Project: Hadoop Common
Issue Type: Bug
Reporter: Eric Yang
JWTRedirectAuthenticationHandler is based on KerberosAuthenticationHandler, and
authentication method in KerberosAuthenticationHandler basically do this:
{code}
String clientPrincipal = gssContext.getSrcName().toString();
KerberosName kerberosName = new KerberosName(clientPrincipal);
String userName = kerberosName.getShortName();
token = new AuthenticationToken(userName, clientPrincipal, getType());
response.setStatus(HttpServletResponse.SC_OK);
LOG.trace("SPNEGO completed for client principal [{}]",
clientPrincipal);
{code}
It obtains the short name of the client principal and respond OK. This is fine
for verifying end user. However, in proxy user case (knox), this
authentication is insufficient because knox principal name is:
knox/host1.example@example.com . KerberosAuthenticationHandler will gladly
confirm that knox is knox. Even if the knox/host1.example@example.com is
used from botnet.rogueresearchlab.tld host. KerberosAuthenticationHandler does
not need to change, but additional authentication with proxy user list should
take place in JWTRedirectAuthenticationHandler to properly fulfill the proxy
use case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-15821) Move Hadoop YARN Registry to Hadoop Registry
[ https://issues.apache.org/jira/browse/HADOOP-15821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-15821: > Move Hadoop YARN Registry to Hadoop Registry > > > Key: HADOOP-15821 > URL: https://issues.apache.org/jira/browse/HADOOP-15821 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15821.000.patch, HADOOP-15821.001.patch, > HADOOP-15821.002.patch, HADOOP-15821.003.patch, HADOOP-15821.004.patch, > HADOOP-15821.005.patch, HADOOP-15821.006.patch, HADOOP-15821.007.patch, > HADOOP-15821.008.patch, HADOOP-15821.009.patch > > > Currently, Hadoop YARN Registry is in YARN. However, this can be used by > other parts of the project (e.g., HDFS). In addition, it does not have any > real dependency to YARN. > We should move it into commons and make it Hadoop Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15670) UserGroupInformation TGT renewer thread doesn't use monotonically increasing time for calculating interval to sleep
[ https://issues.apache.org/jira/browse/HADOOP-15670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-15670. Resolution: Not A Problem > UserGroupInformation TGT renewer thread doesn't use monotonically increasing > time for calculating interval to sleep > --- > > Key: HADOOP-15670 > URL: https://issues.apache.org/jira/browse/HADOOP-15670 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.1.0, 2.9.1 >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre >Priority: Minor > > As per the [documentation of Time#now() > method|https://github.com/apache/hadoop/blob/74411ce0ce7336c0f7bb5793939fdd64a5dcdef6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Time.java#L49-L57], > it should not be used for calculating duration or interval to sleep. But the > TGT renewer thread in UserGroupInformation object doesn't follow this > recommendation, > [https://github.com/apache/hadoop/blob/74411ce0ce7336c0f7bb5793939fdd64a5dcdef6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L892-L899] > This should be fixed to use Time.monotonicNow() API instead. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15601) Change yarn.admin.acl setting to be more restricted
Eric Yang created HADOOP-15601: -- Summary: Change yarn.admin.acl setting to be more restricted Key: HADOOP-15601 URL: https://issues.apache.org/jira/browse/HADOOP-15601 Project: Hadoop Common Issue Type: Sub-task Components: security Reporter: Eric Yang Yarn.admin.acl is default to *, which means everyone is yarn administrator by default. It is probably better to default yarn.admin.acl to the user who runs yarn framework to prevent attacks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15600) Set default proxy user settings to non-routable IP addresses and default users group
Eric Yang created HADOOP-15600:
--
Summary: Set default proxy user settings to non-routable IP
addresses and default users group
Key: HADOOP-15600
URL: https://issues.apache.org/jira/browse/HADOOP-15600
Project: Hadoop Common
Issue Type: Sub-task
Components: security
Reporter: Eric Yang
The default setting to restrict the cluster nodes to communicate with peer
nodes are controlled by: hadoop.proxyuser.[hdfs.yarn].hosts, and
hadoop.proxyuser.[hdfs|yarn].groups. These settings are default to be opened
which allows any hosts to impersonate any user.
The proposal is to default settings to:
{code}
hadoop.proxyuser.hdfs.hosts
127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
hadoop.proxyuser.hdfs.groups
users
hadoop.proxyuser.yarn.hosts
127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
hadoop.proxyuser.yarn.groups
users
{code}
This will allow the cluster to default to a closed network and default "users"
group to reduce risks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15597) UserGroupInformation class throws NPE when Kerberos TGT expired
[
https://issues.apache.org/jira/browse/HADOOP-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-15597.
Resolution: Duplicate
> UserGroupInformation class throws NPE when Kerberos TGT expired
> ---
>
> Key: HADOOP-15597
> URL: https://issues.apache.org/jira/browse/HADOOP-15597
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.9.1, 3.0.1, 3.0.2, 2.9.2
>Reporter: Eric Yang
>Priority: Critical
>
> UserGroupInformation class throws NPE RuntimeException when tgt renewer can
> not determine expiration time:
> {code}
> Thread Thread[TGT Renewer for rm/host1.example@example.com,5,main] threw
> an Exception.
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The error occurs when Hadoop daemon processes uses UGI class to do service
> tgt renewal. The code is written that reattachMetrics() must be called by
> the main program to initialize UGI metrics. Metrics initialization is only
> called by resource manager. Other Hadoop processes do not call
> reattachMetrics(). The runtime exception could cause interruption to Hadoop
> services as observed in YARN RegistryDNS (YARN-8514). It would be nice if
> metrics initialization happens in UGI class without reliance on Hadoop
> program to make UGI metrics initialization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15597) UserGroupInformation class throws NPE when Kerberos TGT expired
Eric Yang created HADOOP-15597:
--
Summary: UserGroupInformation class throws NPE when Kerberos TGT
expired
Key: HADOOP-15597
URL: https://issues.apache.org/jira/browse/HADOOP-15597
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 3.0.2, 3.0.1, 2.9.1, 3.1.0, 3.0.0, 2.9.0, 2.9.2
Reporter: Eric Yang
UserGroupInformation class throws NPE RuntimeException when tgt renewer can not
determine expiration time:
{code}
Thread Thread[TGT Renewer for rm/host1.example@example.com,5,main] threw an
Exception.
java.lang.NullPointerException
at
javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
at
org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
at java.lang.Thread.run(Thread.java:745)
{code}
The error occurs when Hadoop daemon processes uses UGI class to do service tgt
renewal. The code is written that reattachMetrics() must be called by the main
program to initialize UGI metrics. Metrics initialization is only called by
resource manager. Other Hadoop processes do not call reattachMetrics(). The
runtime exception could cause interruption to Hadoop services as observed in
YARN RegistryDNS (YARN-8514). It would be nice if metrics initialization
happens in UGI class without reliance on Hadoop program to make UGI metrics
initialization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15588) Add proxy acl check for AuthenticationFilter
Eric Yang created HADOOP-15588: -- Summary: Add proxy acl check for AuthenticationFilter Key: HADOOP-15588 URL: https://issues.apache.org/jira/browse/HADOOP-15588 Project: Hadoop Common Issue Type: Sub-task Components: common Reporter: Eric Yang It would be nice if AuthenticationFilter can check proxy user and proxy hosts setting. This helps to determine if the user is coming from an authorized remote server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15587) Securing ASF Hadoop releases out of the box
Eric Yang created HADOOP-15587: -- Summary: Securing ASF Hadoop releases out of the box Key: HADOOP-15587 URL: https://issues.apache.org/jira/browse/HADOOP-15587 Project: Hadoop Common Issue Type: Wish Components: build, common, documentation Reporter: Eric Yang [Mail thread|http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201807.mbox/%3cdc06cefa-fe2b-4ca3-b9a9-1d6df0421...@hortonworks.com%3E] started by Steve Loughran on the mailing lists to change default Hadoop release to be more secure, a list of improvements to include: # Change default proxy acl settings to non-routable IPs. # Implement proxy acl check for HTTP protocol. # Change yarn.admin.acl setting to be more restricted. # Review settings that need to be lock down by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15284) Could not determine real path of mount
Eric Yang created HADOOP-15284:
--
Summary: Could not determine real path of mount
Key: HADOOP-15284
URL: https://issues.apache.org/jira/browse/HADOOP-15284
Project: Hadoop Common
Issue Type: Bug
Reporter: Eric Yang
Docker container is failing to launch in trunk. The root cause is:
{code}
[COMPINSTANCE sleeper-1 : container_1520032931921_0001_01_20]: [2018-03-02
23:26:09.196]Exception from container-launch.
Container id: container_1520032931921_0001_01_20
Exit code: 29
Exception message: image: hadoop/centos:latest is trusted in hadoop registry.
Could not determine real path of mount
'/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache'
Could not determine real path of mount
'/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache'
Invalid docker mount
'/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache:/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache',
realpath=/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache
Error constructing docker command, docker error code=12, error message='Invalid
docker mount'
Shell output: main : command provided 4
main : run as user is hbase
main : requested yarn user is hbase
Creating script paths...
Creating local dirs...
[2018-03-02 23:26:09.240]Diagnostic message from attempt 0 : [2018-03-02
23:26:09.240]
[2018-03-02 23:26:09.240]Container exited with a non-zero exit code 29.
[2018-03-02 23:26:39.278]Could not find
nmPrivate/application_1520032931921_0001/container_1520032931921_0001_01_20//container_1520032931921_0001_01_20.pid
in any of the directories
[COMPONENT sleeper]: Failed 11 times, exceeded the limit - 10. Shutting down
now...
{code}
The filecache cant not be mounted because it doesn't exist.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14077) Improve the patch of HADOOP-13119
[
https://issues.apache.org/jira/browse/HADOOP-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HADOOP-14077:
> Improve the patch of HADOOP-13119
> -
>
> Key: HADOOP-14077
> URL: https://issues.apache.org/jira/browse/HADOOP-14077
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
>Priority: Major
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-14077.001.patch, HADOOP-14077.002.patch,
> HADOOP-14077.003.patch
>
>
> For some links(such as "/jmx, /stack"), blocking the links in filter chain
> due to impersonation issue is not friendly for users. For example, user "sam"
> is not allowed to be impersonated by user "knox", and the link "/jmx" doesn't
> need any user to do authorization by default. It only needs user "knox" to do
> authentication, in this case, it's not right to block the access in SPNEGO
> filter. We intend to check impersonation permission when the method
> "getRemoteUser" of request is used, so that such kind of links("/jmx,
> /stack") would not be blocked by mistake.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15222) Refine proxy user authorization to support multiple ACL list
Eric Yang created HADOOP-15222: -- Summary: Refine proxy user authorization to support multiple ACL list Key: HADOOP-15222 URL: https://issues.apache.org/jira/browse/HADOOP-15222 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Eric Yang This Jira is responding to follow up work for HADOOP-14077. The original goal of HADOOP-14077 is to have ability to support multiple ACL lists. When checking for proxy user authorization in AuthenticationFilter to ensure there is a way to authorize normal users and admin users using separate proxy users ACL lists. This was suggested in HADOOP-14060 to configure AuthenticationFilterWithProxyUser this way: AuthenticationFilterWithProxyUser->StaticUserWebFilter->AuthenticationFIlterWithProxyUser This enables the second AuthenticationFilterWithProxyUser validates both credentials claim by proxy user, and end user. However, there is a side effect that unauthorized users are not properly rejected with 403 FORBIDDEN message if there is no other web filter configured to handle the required authorization work. This JIRA is intend to discuss the work of HADOOP-14077 by either combine StaticUserWebFilter + second AuthenticationFilterWithProxyUser into a AuthorizationFilterWithProxyUser as a final filter to evict unauthorized user, or revert both HADOOP-14077 and HADOOP-13119 to eliminate the false positive in user authorization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15162) UserGroupInformation.createRemoteUser hardcode authentication method to SIMPLE
[
https://issues.apache.org/jira/browse/HADOOP-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved HADOOP-15162.
Resolution: Not A Problem
Close this as not a problem. Bad assumption for SIMPLE security mode doesn't
check for proxy ACL. I verified that SIMPLE security mode also checks for
proxy ACL. UGI.createRemoteUser(remoteUser) has no effect to proxy ACL check.
Thanks to [~jlowe] and [~daryn] for advices and recommendations.
> UserGroupInformation.createRemoteUser hardcode authentication method to SIMPLE
> --
>
> Key: HADOOP-15162
> URL: https://issues.apache.org/jira/browse/HADOOP-15162
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Eric Yang
>
> {{UserGroupInformation.createRemoteUser(String user)}} is hard coded
> Authentication method to SIMPLE by HADOOP-10683. This by passed proxyuser
> ACL check, isSecurityEnabled check, and allow caller to impersonate as
> anyone. This method could be abused in the main code base, which can cause
> part of Hadoop to become insecure without proxyuser check for both SIMPLE or
> Kerberos enabled environment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15162) UserGroupInformation.createRmoteUser hardcode authentication method to SIMPLE
Eric Yang created HADOOP-15162:
--
Summary: UserGroupInformation.createRmoteUser hardcode
authentication method to SIMPLE
Key: HADOOP-15162
URL: https://issues.apache.org/jira/browse/HADOOP-15162
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Eric Yang
{{UserGroupInformation.createRemoteUser(String user)}} is hard coded
Authentication method to SIMPLE by HADOOP-10683. This by passed proxyuser ACL
check, isSecurityEnabled check, and allow caller to impersonate as anyone.
This method could be abused in the main code base, which can cause part of
Hadoop to become insecure without proxyuser check for both SIMPLE or Kerberos
enabled environment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-10054) ViewFsFileStatus.toString() is broken
[
https://issues.apache.org/jira/browse/HADOOP-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reopened HADOOP-10054:
This patch broke trunk development. Please run unit test:
{code}
mvn clean test -Dtest=TestViewFileSystemLocalFileSystem
{code}
> ViewFsFileStatus.toString() is broken
> -
>
> Key: HADOOP-10054
> URL: https://issues.apache.org/jira/browse/HADOOP-10054
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.0.5-alpha
>Reporter: Paul Han
>Assignee: Hanisha Koneru
>Priority: Minor
> Fix For: 3.0.1
>
> Attachments: HADOOP-10054.001.patch, HADOOP-10054.002.patch
>
>
> ViewFsFileStatus.toString is broken. Following code snippet :
> {code}
> FileStatus stat= somefunc(); // somefunc() returns an instance of
> ViewFsFileStatus
> System.out.println("path:" + stat.getPath());
> System.out.println(stat.toString());
> {code}
> produces the output:
> {code}
> path:viewfs://x.com/user/X/tmp-48
> ViewFsFileStatus{path=null; isDirectory=false; length=0; replication=0;
> blocksize=0; modification_time=0; access_time=0; owner=; group=;
> permission=rw-rw-rw-; isSymlink=false}
> {code}
> Note that "path=null" is not correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14967) Use jetty CORS filter for web interface
Eric Yang created HADOOP-14967:
--
Summary: Use jetty CORS filter for web interface
Key: HADOOP-14967
URL: https://issues.apache.org/jira/browse/HADOOP-14967
Project: Hadoop Common
Issue Type: Bug
Reporter: Eric Yang
Hadoop CORS filter only prevent iframe from embedding Hadoop UI. It would be
nice to use standard jetty CORS filter to improve control of CORS filtering. A
standard approach is to add this section of code in web.xml:
{code}
cross-origin
org.eclipse.jetty.servlets.CrossOriginFilter
allowedOrigins
*
allowedMethods
*
allowedHeaders
*
cross-origin
/*
{code}
and pom.xml:
{code}
org.eclipse.jetty
jetty-servlets
${jetty.version}
{code}
Hadoop web application are written with embedding Jetty. This is most likely
translate to a callable class to initialize web filter using standard jetty web
filter, when {{hadoop.http.cross-origin.enabled}} is set to true.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-9122) Add powermock library for writing better unit tests
[ https://issues.apache.org/jira/browse/HADOOP-9122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-9122: --- Assignee: Eric Yang (was: Radim Kolar) Reopening this old issue. Powermockito is great for creating better unit test cases for large scale project that code needs to interact with other code which may not be modifiable. There were some incompatibility issues between Mockito and PowerMock in the last couple years. Mockito 1.9.5 + Powermock 1.5.6, and commons-io 2.5 seems to be a reliable combination that works for Hadoop trunk code base. > Add powermock library for writing better unit tests > --- > > Key: HADOOP-9122 > URL: https://issues.apache.org/jira/browse/HADOOP-9122 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Radim Kolar >Assignee: Eric Yang >Priority: Critical > Attachments: powermock.txt > > > This library is extension of mockito and easymock libraries. It allows > testing of private methods, private variables and other otherwise untestable > stuff. > http://code.google.com/p/powermock/ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13119) Web UI authorization error accessing /logs/ when Kerberos
[ https://issues.apache.org/jira/browse/HADOOP-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-13119: > Web UI authorization error accessing /logs/ when Kerberos > - > > Key: HADOOP-13119 > URL: https://issues.apache.org/jira/browse/HADOOP-13119 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.4 >Reporter: Jeffrey E Rodriguez > > User Hadoop on secure mode. > login as kdc user, kinit. > start firefox and enable Kerberos > access http://localhost:50070/logs/ > Get 403 authorization errors. > only hdfs user could access logs. > Would expect as a user to be able to web interface logs link. > Same results if using curl: > curl -v --negotiate -u tester: http://localhost:50070/logs/ > HTTP/1.1 403 User tester is unauthorized to access this page. > so: > 1. either don't show links if hdfs user is able to access. > 2. provide mechanism to add users to web application realm. > 3. note that we are pass authentication so the issue is authorization to > /logs/ > suspect that /logs/ path is secure in webdescriptor so suspect users by > default don't have access to secure paths. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12110) Consolidate usage of JSON libraries
[ https://issues.apache.org/jira/browse/HADOOP-12110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-12110. Resolution: Invalid Opened for the wrong project. Sorry, close as invalid. > Consolidate usage of JSON libraries > --- > > Key: HADOOP-12110 > URL: https://issues.apache.org/jira/browse/HADOOP-12110 > Project: Hadoop Common > Issue Type: Bug >Reporter: Eric Yang >Assignee: Eric Yang > > Chukwa uses JSON jar from json.org and also json-simple from googlecode. It > would be nice if we only use one implementation of JSON to be consistent. > Mindev JSON-smart was also considered as replacement for JSON simple to > improve performance, but it doesn't handle some characters correctly. > Therefore, it's best to use JSON Simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12110) Consolidate usage of JSON libraries
Eric Yang created HADOOP-12110: -- Summary: Consolidate usage of JSON libraries Key: HADOOP-12110 URL: https://issues.apache.org/jira/browse/HADOOP-12110 Project: Hadoop Common Issue Type: Bug Reporter: Eric Yang Assignee: Eric Yang Chukwa uses JSON jar from json.org and also json-simple from googlecode. It would be nice if we only use one implementation of JSON to be consistent. Mindev JSON-smart was also considered as replacement for JSON simple to improve performance, but it doesn't handle some characters correctly. Therefore, it's best to use JSON Simple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-10759) Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-10759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-10759. Resolution: Fixed Allen, this JIRA is filed for Hadoop Common, YARN problems can be addressed in YARN JIRA. The hard coded value was introduced in HADOOP-5212. Before 0.21, the behavior of JVM heap size control is exactly same as this patch. It does not look like a new feature if we are just rollback to existing Hadoop behavior in 2009. We don't need to spread fear on this patch is going to the universe because it doesn't. This has been tested in stress test, and our clusters have been running this patch for almost a year before it was contributed to the community. > Remove hardcoded JAVA_HEAP_MAX in hadoop-config.sh > -- > > Key: HADOOP-10759 > URL: https://issues.apache.org/jira/browse/HADOOP-10759 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.4.0 > Environment: Linux64 >Reporter: sam liu >Priority: Minor > Fix For: 2.6.0 > > Attachments: HADOOP-10759.patch, HADOOP-10759.patch > > > In hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, there > is a hard code for Java parameter: 'JAVA_HEAP_MAX=-Xmx1000m'. It should be > removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8419) GzipCodec NPE upon reset with IBM JDK
[ https://issues.apache.org/jira/browse/HADOOP-8419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-8419. --- Resolution: Fixed Hadoop Commons and HDFS trunk builds have been stabilized. Mark this as fixed. > GzipCodec NPE upon reset with IBM JDK > - > > Key: HADOOP-8419 > URL: https://issues.apache.org/jira/browse/HADOOP-8419 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3 >Reporter: Luke Lu >Assignee: Yu Li > Labels: gzip, ibm-jdk > Fix For: 1.1.2 > > Attachments: HADOOP-8419-branch-1.patch, > HADOOP-8419-branch1-v2.patch, HADOOP-8419-trunk.patch, > HADOOP-8419-trunk-v2.patch > > > The GzipCodec will NPE upon reset after finish when the native zlib codec is > not loaded. When the native zlib is loaded the codec creates a > CompressorOutputStream that doesn't have the problem, otherwise, the > GZipCodec uses GZIPOutputStream which is extended to provide the resetState > method. Since IBM JDK 6 SR9 FP2 including the current JDK 6 SR10, > GZIPOutputStream#finish will release the underlying deflater, which causes > NPE upon reset. This seems to be an IBM JDK quirk as Sun JDK and OpenJDK > doesn't have this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7641) Add Apache License to template config files
Add Apache License to template config files --- Key: HADOOP-7641 URL: https://issues.apache.org/jira/browse/HADOOP-7641 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.23.0 Environment: Java 6, Linux Reporter: Eric Yang Assignee: Eric Yang Files in hadoop-common-project/hadoop-common/src/main/packages/templates/conf don't have Apache Software License in the header. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7637) Fair scheduler configuration file is not bundled in RPM
Fair scheduler configuration file is not bundled in RPM
---
Key: HADOOP-7637
URL: https://issues.apache.org/jira/browse/HADOOP-7637
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 0.20.205.0
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0
205 build of tar is fine, but rpm failed with:
{noformat}
[rpm] Processing files: hadoop-0.20.205.0-1
[rpm] warning: File listed twice: /usr/libexec
[rpm] warning: File listed twice: /usr/libexec/hadoop-config.sh
[rpm] warning: File listed twice: /usr/libexec/jsvc.i386
[rpm] Checking for unpackaged file(s): /usr/lib/rpm/check-files
/tmp/hadoop_package_build_hortonfo/BUILD
[rpm] error: Installed (but unpackaged) file(s) found:
[rpm]/etc/hadoop/fair-scheduler.xml
[rpm] File listed twice: /usr/libexec
[rpm] File listed twice: /usr/libexec/hadoop-config.sh
[rpm] File listed twice: /usr/libexec/jsvc.i386
[rpm] Installed (but unpackaged) file(s) found:
[rpm]/etc/hadoop/fair-scheduler.xml
[rpm]
[rpm]
[rpm] RPM build errors:
BUILD FAILED
/grid/0/dev/mfoley/hadoop-0.20-security-205/build.xml:1747: '/usr/bin/rpmbuild'
failed with exit code 1
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7388) Remove definition of HADOOP_HOME and HADOOP_PREFIX from hadoop-env.sh.template
[ https://issues.apache.org/jira/browse/HADOOP-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-7388. --- Resolution: Fixed Fix Version/s: 0.23.0 0.20.205.0 Resolved as part of HADOOP-7599. > Remove definition of HADOOP_HOME and HADOOP_PREFIX from hadoop-env.sh.template > -- > > Key: HADOOP-7388 > URL: https://issues.apache.org/jira/browse/HADOOP-7388 > Project: Hadoop Common > Issue Type: Improvement > Environment: Java 6, RHEL 5.5 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Trivial > Fix For: 0.20.205.0, 0.23.0 > > Attachments: HADOOP-7388.patch > > > The file structure layout proposed in HADOOP-6255 was designed to remove the > need of using HADOOP_HOME environment to locate hadoop bits. The file > structure layout should be able to map to /usr or system directories, > therefore HADOOP_HOME is renamed to HADOOP_PREFIX to be more concise. > HADOOP_PREFIX should not be exported to the user. If the user use > hadoop-setup-single-node.sh or hadoop-setup-conf.sh to configure hadoop, the > current scripts put HADOOP_PREFIX/HADOOP_HOME in hadoop-env.sh. The config > template generation code should remove reference of HADOOP_PREFIX/HADOOP_HOME > from hadoop-env.sh. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7626) Allow overwrite of HADOOP_CLASSPATH and HADOOP_OPTS
Allow overwrite of HADOOP_CLASSPATH and HADOOP_OPTS
---
Key: HADOOP-7626
URL: https://issues.apache.org/jira/browse/HADOOP-7626
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 0.20.205.0
Environment: Java, Linux
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.20.205.0
Quote email from Ashutosh Chauhan:
bq. There is a bug in hadoop-env.sh which prevents hcatalog server to start in
secure settings. Instead of adding classpath, it overrides them. I was not able
to verify where the bug belongs to, in HMS or in hadoop scripts. Looks like
hadoop-env.sh is generated from hadoop-env.sh.template in installation process
by HMS. Hand crafted patch follows:
bq. - export HADOOP_CLASSPATH=$f
bq. +export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f
bq. -export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true "
bq. +export HADOOP_OPTS="${HADOOP_OPTS} -Djava.net.preferIPv4Stack=true "
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7609) Debian package shows invalid hdfs user
[ https://issues.apache.org/jira/browse/HADOOP-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-7609. --- Resolution: Not A Problem This is not a real problem. > Debian package shows invalid hdfs user > -- > > Key: HADOOP-7609 > URL: https://issues.apache.org/jira/browse/HADOOP-7609 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 0.20.204.0, 0.23.0 > Environment: Java 6, Debian >Reporter: Eric Yang >Assignee: Eric Yang > > First time install debian package on Debian machine, there is a error message > showing: > invalid hdfs user. > invalid mapred user. > Looks like the users are not created during the installation. Not sure if > this is EC2 related or debian related. Investigating... -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7615) Binary layout does not put share/hadoop/contrib/*.jar into the class path
Binary layout does not put share/hadoop/contrib/*.jar into the class path - Key: HADOOP-7615 URL: https://issues.apache.org/jira/browse/HADOOP-7615 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.20.204.0, 0.23.0 Environment: Java, Linux Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.0 For contrib projects, contrib jar files are not included in HADOOP_CLASSPATH in the binary layout. Several projects jar files should be copied to $HADOOP_PREFIX/share/hadoop/lib for binary deployment. The interesting jar files to include in $HADOOP_PREFIX/share/hadoop/lib are: capacity-scheduler, thriftfs, fairscheduler. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7610) /etc/profile.d does not exist on Debian
/etc/profile.d does not exist on Debian
---
Key: HADOOP-7610
URL: https://issues.apache.org/jira/browse/HADOOP-7610
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 0.20.204.0, 0.23.0
Environment: Java 6, Debian
Reporter: Eric Yang
Assignee: Eric Yang
As part of post installation script, there is a symlink created in
/etc/profile.d/hadoop-env.sh to source /etc/hadoop/hadoop-env.sh. Therefore,
users do not need to configure HADOOP_* environment. Unfortunately,
/etc/profile.d only exists in Ubuntu. [Section 9.9 of the Debian
Policy|http://www.debian.org/doc/debian-policy/ch-opersys.html#s9.9] states:
{quote}
A program must not depend on environment variables to get reasonable defaults.
(That's because these environment variables would have to be set in a
system-wide configuration file like /etc/profile, which is not supported by all
shells.)
If a program usually depends on environment variables for its configuration,
the program should be changed to fall back to a reasonable default
configuration if these environment variables are not present. If this cannot be
done easily (e.g., if the source code of a non-free program is not available),
the program must be replaced by a small "wrapper" shell script which sets the
environment variables if they are not already defined, and calls the original
program.
Here is an example of a wrapper script for this purpose:
{noformat}
#!/bin/sh
BAR=${BAR:-/var/lib/fubar}
export BAR
exec /usr/lib/foo/foo "$@"
{noformat}
Furthermore, as /etc/profile is a configuration file of the base-files package,
other packages must not put any environment variables or other commands into
that file.
{quote}
Hence the default environment setup should skip for Debian.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7609) Debian package shows invalid hdfs user
Debian package shows invalid hdfs user -- Key: HADOOP-7609 URL: https://issues.apache.org/jira/browse/HADOOP-7609 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.20.204.0, 0.23.0 Environment: Java 6, Debian Reporter: Eric Yang Assignee: Eric Yang First time install debian package on Debian machine, there is a error message showing: invalid hdfs user. invalid mapred user. Looks like the users are not created during the installation. Not sure if this is EC2 related or debian related. Investigating... -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7603) Set default hdfs, mapred uid, and hadoop group gid for RPM packages
Set default hdfs, mapred uid, and hadoop group gid for RPM packages
---
Key: HADOOP-7603
URL: https://issues.apache.org/jira/browse/HADOOP-7603
Project: Hadoop Common
Issue Type: Bug
Environment: Java, Redhat EL, Ubuntu
Reporter: Eric Yang
Assignee: Eric Yang
Hadoop rpm package creates hdfs, mapped users, and hadoop group for
automatically setting up pid directory and log directory with proper
permission. The default headless users should have a fixed uid, and gid
numbers defined.
Searched through the standard uid and gid on both Redhat and Debian distro. It
looks like:
{noformat}
uid: 201 for hdfs
uid: 202 for mapred
gid: 49 for hadoop
{noformat}
would be free for use.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7599) Improve hadoop setup conf script to setup secure Hadoop cluster
Improve hadoop setup conf script to setup secure Hadoop cluster --- Key: HADOOP-7599 URL: https://issues.apache.org/jira/browse/HADOOP-7599 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.20.203.0 Environment: Java 6, RHEL 5.6 Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.0 Setting up a secure Hadoop cluster requires a lot of manual setup. The motivation of this jira is to provide setup scripts to automate setup secure Hadoop cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7596) Enable jsvc to work with Hadoop RPM package
Enable jsvc to work with Hadoop RPM package --- Key: HADOOP-7596 URL: https://issues.apache.org/jira/browse/HADOOP-7596 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.20.204.0 Environment: Java 6, RedHat EL 5.6 Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.20.205.0 For secure Hadoop 0.20.2xx cluster, datanode can only run with 32 bit jvm because Hadoop only packages 32 bit jsvc. The build process should download proper jsvc versions base on the build architecture. In addition, the shell script should be enhanced to locate hadoop jar files in the proper location. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-7583) mvn -Ptar and -Pbintar are broken
[ https://issues.apache.org/jira/browse/HADOOP-7583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-7583: --- Please use -Ptar to activate -Pdist and build tarball > mvn -Ptar and -Pbintar are broken > - > > Key: HADOOP-7583 > URL: https://issues.apache.org/jira/browse/HADOOP-7583 > Project: Hadoop Common > Issue Type: Bug >Reporter: Arun C Murthy >Priority: Blocker > > I can no longer get the artifacts via mvn -Ptar or -Pbintar. Related to > HADOOP-7560? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7563) hadoop-config.sh setup HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME incorrectly
hadoop-config.sh setup HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME incorrectly -- Key: HADOOP-7563 URL: https://issues.apache.org/jira/browse/HADOOP-7563 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.23.0 HADOOP_HDFS_HOME and HADOOP_MAPRED_HOME was set to HADOOP_PREFIX/share/hadoop/hdfs and HADOOP_PREFIX/share/hadoop/mapreduce. This setup confuses the location of hdfs and mapred scripts. Instead the script should look for hdfs and mapred script in HADOOP_PREFIX/bin. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7521) bintar created tarball should use a common directory for prefix
bintar created tarball should use a common directory for prefix
---
Key: HADOOP-7521
URL: https://issues.apache.org/jira/browse/HADOOP-7521
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 0.23.0
Environment: Java 6, Maven, Linux/Mac
Reporter: Eric Yang
The binary tarball contains the directory structure like:
{noformat}
hadoop-common-0.23.0-SNAPSHOT-bin/bin
/etc/hadoop
/libexec
/sbin
/share/hadoop/common
{noformat}
It would be nice to rename the prefix directory to a common directory where it
is common to all Hadoop stack software. Therefore, user can untar hbase,
hadoop, zookeeper, pig, hive all into the same location and run from the top
level directory without manually renaming them to the same directory again.
By default the prefix directory can be /usr. Hence, it could merge with the
base OS.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-7356) RPM packages broke bin/hadoop script for hadoop 0.20.205
[ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-7356: --- bq. I took out the support for running out of non-deployed developer directories. Owen, without support for non-deployed developer directories: "ant test-c++-libhdfs would -Dlibhdfs=true" fails because test native hdfs could not start hdfs via bin/hadoop shell script in source directory. > RPM packages broke bin/hadoop script for hadoop 0.20.205 > > > Key: HADOOP-7356 > URL: https://issues.apache.org/jira/browse/HADOOP-7356 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.20.204.0 > Environment: Java 6, Redhat EL 5.5 >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: 0.20.204.0, 0.23.0 > > Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, > HADOOP-7356.patch > > > hadoop-config.sh has been moved to libexec for binary package, but developers > prefers to have hadoop-config.sh in bin. Hadoo shell scripts should be > modified to support both scenarios. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7484) Update HDFS dependency of Java for deb package
[ https://issues.apache.org/jira/browse/HADOOP-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-7484. --- Resolution: Duplicate This is duplicate of HDFS-2192. > Update HDFS dependency of Java for deb package > -- > > Key: HADOOP-7484 > URL: https://issues.apache.org/jira/browse/HADOOP-7484 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 0.23.0 > Environment: Java 6, Ubuntu/Debian >Reporter: Eric Yang > Fix For: 0.23.0 > > > Java dependency for Debian package is specified as open JDK, but it should > depends on Sun version of Java. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7484) Update HDFS dependency of Java for deb package
Update HDFS dependency of Java for deb package -- Key: HADOOP-7484 URL: https://issues.apache.org/jira/browse/HADOOP-7484 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 0.23.0 Environment: Java 6, Ubuntu/Debian Reporter: Eric Yang Fix For: 0.23.0 Java dependency for Debian package is specified as open JDK, but it should depends on Sun version of Java. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7483) Multiple Java installed confuses Hadoop Debian package installer
Multiple Java installed confuses Hadoop Debian package installer Key: HADOOP-7483 URL: https://issues.apache.org/jira/browse/HADOOP-7483 Project: Hadoop Common Issue Type: Bug Environment: Java 6, Debian/Ubuntu Reporter: Eric Yang Assignee: Eric Yang When openjdk and sun java are both installed on the machine, Hadoop debian package can not determine correct Java to use. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7475) hadoop-setup-single-node.sh is broken
hadoop-setup-single-node.sh is broken
-
Key: HADOOP-7475
URL: https://issues.apache.org/jira/browse/HADOOP-7475
Project: Hadoop Common
Issue Type: Bug
Environment: Java 6
Reporter: Eric Yang
Assignee: Eric Yang
When running hadoop-setup-single-node.sh, the system can not find the templates
configuration directory:
{noformat}
cat: /usr/libexec/../templates/conf/core-site.xml: No such file or directory
cat: /usr/libexec/../templates/conf/hdfs-site.xml: No such file or directory
cat: /usr/libexec/../templates/conf/mapred-site.xml: No such file or directory
cat: /usr/libexec/../templates/conf/hadoop-env.sh: No such file or directory
chown: cannot access `hadoop-env.sh': No such file or directory
chmod: cannot access `hadoop-env.sh': No such file or directory
cp: cannot stat `*.xml': No such file or directory
cp: cannot stat `hadoop-env.sh': No such file or directory
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7436) Bundle Chukwa Metrics plugin in Hadoop
Bundle Chukwa Metrics plugin in Hadoop -- Key: HADOOP-7436 URL: https://issues.apache.org/jira/browse/HADOOP-7436 Project: Hadoop Common Issue Type: New Feature Environment: Java 6 Reporter: Eric Yang Assignee: Eric Yang For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics. It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base. It is similar to bundling Ganglia metrics plugin in Hadoop code base. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7417) Hadoop Management System (Umbrella)
Hadoop Management System (Umbrella) --- Key: HADOOP-7417 URL: https://issues.apache.org/jira/browse/HADOOP-7417 Project: Hadoop Common Issue Type: New Feature Environment: Java 6, Linux Reporter: Eric Yang Assignee: Eric Yang The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop. Prototype demo source code can be obtained from: http://github.com/macroadster/hms -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7388) Remove definition of HADOOP_HOME and HADOOP_PREFIX from hadoop-env.sh.template
Remove definition of HADOOP_HOME and HADOOP_PREFIX from hadoop-env.sh.template -- Key: HADOOP-7388 URL: https://issues.apache.org/jira/browse/HADOOP-7388 Project: Hadoop Common Issue Type: Improvement Environment: Java 6, RHEL 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Trivial The file structure layout proposed in HADOOP-6255 was designed to remove the need of using HADOOP_HOME environment to locate hadoop bits. The file structure layout should be able to map to /usr or system directories, therefore HADOOP_HOME is renamed to HADOOP_PREFIX to be more concise. HADOOP_PREFIX should not be exported to the user. If the user use hadoop-setup-single-node.sh or hadoop-setup-conf.sh to configure hadoop, the current scripts put HADOOP_PREFIX/HADOOP_HOME in hadoop-env.sh. The config template generation code should remove reference of HADOOP_PREFIX/HADOOP_HOME from hadoop-env.sh. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7387) Change rpm to ignore configuration files when uninstalling
Change rpm to ignore configuration files when uninstalling -- Key: HADOOP-7387 URL: https://issues.apache.org/jira/browse/HADOOP-7387 Project: Hadoop Common Issue Type: Improvement Environment: Java 6, RHEL 5.5 Reporter: Eric Yang Assignee: Eric Yang Priority: Minor When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave. It would be nice if config file does not get renamed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7371) Improve tarball distributions
Improve tarball distributions - Key: HADOOP-7371 URL: https://issues.apache.org/jira/browse/HADOOP-7371 Project: Hadoop Common Issue Type: Improvement Components: build Environment: Java 6, Redhat 5.5 Reporter: Eric Yang Assignee: Eric Yang Hadoop release tarball contains both raw source and binary. This leads users to use the release tarball as base for applying patches, to build custom Hadoop. This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate. To correct the problematic usage of the release tarball, the release build target should be defined as: "ant source" generates source release tarball. "ant binary" is binary release without source/javadoc jar files. "ant tar" is a mirror of binary release with source/javadoc jar files. Does this sound reasonable? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-7356) RPM packages broke bin/hadoop script for hadoop 0.20.205
[ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reopened HADOOP-7356: --- Reopen for not yet committed to 0.20.20x branch. > RPM packages broke bin/hadoop script for hadoop 0.20.205 > > > Key: HADOOP-7356 > URL: https://issues.apache.org/jira/browse/HADOOP-7356 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.20.204.0 > Environment: Java 6, Redhat EL 5.5 >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: 0.23.0 > > Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, > HADOOP-7356.patch > > > hadoop-config.sh has been moved to libexec for binary package, but developers > prefers to have hadoop-config.sh in bin. Hadoo shell scripts should be > modified to support both scenarios. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7356) RPM packages broke bin/hadoop script for hadoop 0.20.205
RPM packages broke bin/hadoop script for hadoop 0.20.205 Key: HADOOP-7356 URL: https://issues.apache.org/jira/browse/HADOOP-7356 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.20.204.0 Environment: Java 6, Redhat EL 5.5 Reporter: Eric Yang Assignee: Eric Yang hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin. Hadoo shell scripts should be modified to support both scenarios. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
