[jira] [Created] (HADOOP-7535) Hadoop Streaming map_input_file
Hadoop Streaming map_input_file --- Key: HADOOP-7535 URL: https://issues.apache.org/jira/browse/HADOOP-7535 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.20.203.0 Environment: Debian Squeeze Reporter: Jonathan Poon I'm currently trying to use the map_input_file environment variable to determine the file the stdin stream is coming from into a mapper code I've written. I used the following command to print the environment variables and see what map_input_file was given: hadoop jar /usr/local/hadoop/contrib/streaming/hadoop-streaming-0.20.203.0.jar -input A -input S -input F -output B -mapper "bash -c \"export\"" I get the following output for the map_input_file: declare -x map_input_file="hdfs://localhost:54310/user/poonj/A/A.txt" declare -x map_input_file="hdfs://localhost:54310/user/poonj/A/A.txt" declare -x map_input_file="hdfs://localhost:54310/user/poonj/F/F.txt" declare -x map_input_file="hdfs://localhost:54310/user/poonj/S/S.txt" I am under the assumption that the variable is only used once and should store the current file being processed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7534) Make test-patch.sh ensure that dependent projects can compile
[ https://issues.apache.org/jira/browse/HADOOP-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082761#comment-13082761 ] Aaron T. Myers commented on HADOOP-7534: Great point, Alejandro. Seems like with HADOOP-7533 we're almost all the way there, then. I'd still like to see two things: # Even when run from trunk/hadoop-hfds, test-patch could still make sure that M/R compiles. # Make sure that the Hudson pre-commit build runs from trunk/, and so will necessarily compile all the bits of all the sub-projects. > Make test-patch.sh ensure that dependent projects can compile > - > > Key: HADOOP-7534 > URL: https://issues.apache.org/jira/browse/HADOOP-7534 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: 0.23.0 >Reporter: Aaron T. Myers > > Every once in a while a change in Common causes HDFS or M/R to not compile, > or a change in HDFS causes M/R to not compile. With the project unsplit, it > wouldn't be difficult to make test-patch.sh simply ensure that all the Hadoop > sub-projects continue to compile for any given change. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082763#comment-13082763 ] Todd Lipcon commented on HADOOP-7531: - +1, lgtm assuming test-patch is clean. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch, hadoop-7531-3.patch, hadoop-7531-4.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HADOOP-7531: Attachment: hadoop-7531-4.patch Actually no, I think the only reason it checked for null was the mock in TestStream and that now stubs out getRequestURI. Updated patch. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch, hadoop-7531-3.patch, hadoop-7531-4.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082751#comment-13082751 ] Eli Collins edited comment on HADOOP-7531 at 8/10/11 11:21 PM: --- Actually no, I think the only reason it checked for null was the mock in TestStreamFile and HDFS-2235 now mocks getRequestURI so that's no longer the case. Updated patch. was (Author: eli): Actually no, I think the only reason it checked for null was the mock in TestStream and that now stubs out getRequestURI. Updated patch. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch, hadoop-7531-3.patch, hadoop-7531-4.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[
https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082746#comment-13082746
]
Todd Lipcon commented on HADOOP-7531:
-
hmm:
{code}
+
Preconditions.checkArgument(request.getRequestURI().startsWith(servletName));
+return null == request.getRequestURI() ? "/" :
{code}
The second line implies that request.getRequestURI() may be null, in which case
the first line would throw NPE. Is it actually possible to be null? If so,
separate the ternary into an if/else with the precondition inside the if, I
guess?
> Add servlet util methods for handling paths in requests
>
>
> Key: HADOOP-7531
> URL: https://issues.apache.org/jira/browse/HADOOP-7531
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 0.23.0
>
> Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch,
> hadoop-7531-2.patch, hadoop-7531-3.patch
>
>
> Common side of HDFS-2235.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7264) Bump avro version to at least 1.4.1
[ https://issues.apache.org/jira/browse/HADOOP-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu reassigned HADOOP-7264: --- Assignee: Luke Lu > Bump avro version to at least 1.4.1 > --- > > Key: HADOOP-7264 > URL: https://issues.apache.org/jira/browse/HADOOP-7264 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 0.21.0 >Reporter: Luke Lu >Assignee: Luke Lu > Labels: avro > Fix For: 0.23.0 > > > Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There > is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user > facing APIs) not sure about 1.5x though. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HADOOP-7531: Attachment: hadoop-7531-3.patch Yup, right patch this time. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch, hadoop-7531-3.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[
https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082711#comment-13082711
]
Todd Lipcon commented on HADOOP-7531:
-
Last thing... URIUtil.encodeQuery doesn't seem to do much:
{code}
groovy:000> URIUtil.encodeQuery("ugi=a&b=c")
===> ugi=a&b=c
{code}
I think what we really want is to be appending "ugi=" +
URIUtil.encodeWithinQuery(ugi)
> Add servlet util methods for handling paths in requests
>
>
> Key: HADOOP-7531
> URL: https://issues.apache.org/jira/browse/HADOOP-7531
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 0.23.0
>
> Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch,
> hadoop-7531-2.patch
>
>
> Common side of HDFS-2235.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082709#comment-13082709 ] Todd Lipcon commented on HADOOP-7531: - Another small thing: in getDecodedPath and getRawPath, I think you should do a Preconditions.checkArgument(request.getRequestURI().startsWith(servletName)). Otherwise the substring might chop something else off if the function is misused. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082707#comment-13082707 ] Todd Lipcon commented on HADOOP-7531: - hmm, sure you attached the right patch? I don't see the "UTF-8" arg in the latest upload. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7533) Allow test-patch to be run from any subproject directory
[ https://issues.apache.org/jira/browse/HADOOP-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082691#comment-13082691 ] Tom White commented on HADOOP-7533: --- Todd - it could do that (even though the file has a .properties extension, it's never read as a Java properties file, only sourced by bash). I'll leave it as a future enhancement if you don't mind. > Allow test-patch to be run from any subproject directory > - > > Key: HADOOP-7533 > URL: https://issues.apache.org/jira/browse/HADOOP-7533 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Tom White >Assignee: Tom White > Attachments: HADOOP-7533.patch > > > Currently dev-support/test-patch.sh can only be run from the top-level (and > only for hadoop-common). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7380) Add client failover functionality to o.a.h.io.(ipc|retry)
[
https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082686#comment-13082686
]
Eli Collins commented on HADOOP-7380:
-
Btw this is just the IPC plumbing for retry, fail-over is covered in HDFS-1973.
> Add client failover functionality to o.a.h.io.(ipc|retry)
> -
>
> Key: HADOOP-7380
> URL: https://issues.apache.org/jira/browse/HADOOP-7380
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ipc
>Affects Versions: 0.23.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.23.0
>
> Attachments: hadoop-7380-hdfs-example.patch, hadoop-7380.0.patch,
> hadoop-7380.1.patch, hadoop-7380.2.patch, hdfs-7380.3.patch
>
>
> Implementing client failover will likely require changes to {{o.a.h.io.ipc}}
> and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7499) Add method for doing a sanity check on hostnames in NetUtils
[ https://issues.apache.org/jira/browse/HADOOP-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Naisbitt updated HADOOP-7499: - Status: Open (was: Patch Available) So, we discovered that the interface change is incompatible. So, we are removing the incompatible change and the corresponding portion of code relying on that change here and implementing the sanity check only. > Add method for doing a sanity check on hostnames in NetUtils > > > Key: HADOOP-7499 > URL: https://issues.apache.org/jira/browse/HADOOP-7499 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 0.23.0 >Reporter: Jeffrey Naisbitt >Assignee: Jeffrey Naisbitt > Fix For: 0.23.0 > > Attachments: HADOOP-7499-v2.patch, HADOOP-7499.patch, > patchJavadocWarnings.txt > > > As part of MAPREDUCE-2489, we need a method in NetUtils to do a sanity check > on hostnames -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7533) Allow test-patch to be run from any subproject directory
[
https://issues.apache.org/jira/browse/HADOOP-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082681#comment-13082681
]
Todd Lipcon commented on HADOOP-7533:
-
Looks good, though I didn't try it out. One thought - could the top-level
test-patch.properties do something like:
{code}
TOT_RA=0
TOT_FINDBUGS=0
TOT_JAVADOC=0
for subproj in hadoop-common hadoop-hdfs hadoop-mapreduce ; do
OK_RELEASEAUDIT_WARNINGS=0
OK_FINDBUGS_WARNINGS=0
OK_JAVADOC_WARNINGS=0
source $subproj/dev-support/test-patch.properties
TOT_RA=$[$TOT_RA + $OK_RELEASEAUDIT_WARNINGS]
...
done
OK_RELEASEAUDIT_WARNINGS=$TOT_RA
...
{code}
?
> Allow test-patch to be run from any subproject directory
> -
>
> Key: HADOOP-7533
> URL: https://issues.apache.org/jira/browse/HADOOP-7533
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Tom White
>Assignee: Tom White
> Attachments: HADOOP-7533.patch
>
>
> Currently dev-support/test-patch.sh can only be run from the top-level (and
> only for hadoop-common).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7527) Make URL encoding consistent
[ https://issues.apache.org/jira/browse/HADOOP-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082680#comment-13082680 ] Luke Lu commented on HADOOP-7527: - bq. Mechanisms that cause the dev's code to fail in a safe way are far preferable to ones that fail with a XSS that lays unfixed for years. I agree that this is a reasonable stop gap solution until we moved to more secure web UI framework (cf. HADOOP-7532) ;) > Make URL encoding consistent > > > Key: HADOOP-7527 > URL: https://issues.apache.org/jira/browse/HADOOP-7527 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Eli Collins > > URL encoding is currently handled in at least 5 different ways. We should > make these consistent: > # Parameters are encoded when a URI object is created > # HttpServlet uses RequestQuoter to html escape parameter names and values > # StringEscapeUtils is used to escape parameters in ReconfigurationServlet > and DatanodeJspHelper > # URLEncoder and URLDecoder are used in multiple places > # encodePath from Jetty's URIUtil > We should also be consistent about how we pass file names in URLs, some times > they're passed in the path segment, sometimes they're passed in the query > fragment as parameters. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7472) RPC client should deal with the IP address changes
[ https://issues.apache.org/jira/browse/HADOOP-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082660#comment-13082660 ] Hudson commented on HADOOP-7472: Integrated in Hadoop-Common-trunk-Commit #721 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/721/]) HADOOP-7472. RPC client should deal with IP address change. Contributed by Kihwal Lee. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1156350 Files : * /hadoop/common/trunk/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java > RPC client should deal with the IP address changes > -- > > Key: HADOOP-7472 > URL: https://issues.apache.org/jira/browse/HADOOP-7472 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 0.20.205.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Minor > Fix For: 0.20.205.0, 0.23.0 > > Attachments: addr_change_dfs-1.patch.txt, > addr_change_dfs-2.patch.txt, addr_change_dfs-3.patch.txt, > addr_change_dfs.patch.txt, addr_change_dfs_0_20s-1.patch.txt, > addr_change_dfs_0_20s-2.patch.txt, addr_change_dfs_0_20s.patch.txt, > addr_change_dfs_trunk-1.patch.txt, addr_change_dfs_trunk-2.patch.txt, > addr_change_dfs_trunk-3.patch.txt, addr_change_dfs_trunk.patch.txt > > > The current RPC client implementation and the client-side callers assume that > the hostname-address mappings of servers never change. The resolved address > is stored in an immutable InetSocketAddress object above/outside RPC, and the > reconnect logic in the RPC Connection implementation also trusts the resolved > address that was passed down. > If the NN suffers a failure that requires migration, it may be started on a > different node with a different IP address. In this case, even if the > name-address mapping is updated in DNS, the cluster is stuck trying old > address until the whole cluster is restarted. > The RPC client-side should detect this situation and exit or try to recover. > Updating ConnectionId within the Client implementation may get the system > work for the moment, there always is a risk of the cached address:port become > connectable again unintentionally. The real solution will be notifying upper > layer of the address change so that they can re-resolve and retry or > re-architecture the system as discussed in HDFS-34. > For 0.20 lines, some type of compromise may be acceptable. For example, raise > a custom exception for some well-defined high-impact upper layer to do > re-resolve/retry, while other will have to restart. For TRUNK, the HA work > will most likely determine what needs to be done. So this Jira won't cover > the solutions for TRUNK. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7472) RPC client should deal with the IP address changes
[ https://issues.apache.org/jira/browse/HADOOP-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-7472: Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] > RPC client should deal with the IP address changes > -- > > Key: HADOOP-7472 > URL: https://issues.apache.org/jira/browse/HADOOP-7472 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 0.20.205.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Minor > Fix For: 0.20.205.0, 0.23.0 > > Attachments: addr_change_dfs-1.patch.txt, > addr_change_dfs-2.patch.txt, addr_change_dfs-3.patch.txt, > addr_change_dfs.patch.txt, addr_change_dfs_0_20s-1.patch.txt, > addr_change_dfs_0_20s-2.patch.txt, addr_change_dfs_0_20s.patch.txt, > addr_change_dfs_trunk-1.patch.txt, addr_change_dfs_trunk-2.patch.txt, > addr_change_dfs_trunk-3.patch.txt, addr_change_dfs_trunk.patch.txt > > > The current RPC client implementation and the client-side callers assume that > the hostname-address mappings of servers never change. The resolved address > is stored in an immutable InetSocketAddress object above/outside RPC, and the > reconnect logic in the RPC Connection implementation also trusts the resolved > address that was passed down. > If the NN suffers a failure that requires migration, it may be started on a > different node with a different IP address. In this case, even if the > name-address mapping is updated in DNS, the cluster is stuck trying old > address until the whole cluster is restarted. > The RPC client-side should detect this situation and exit or try to recover. > Updating ConnectionId within the Client implementation may get the system > work for the moment, there always is a risk of the cached address:port become > connectable again unintentionally. The real solution will be notifying upper > layer of the address change so that they can re-resolve and retry or > re-architecture the system as discussed in HDFS-34. > For 0.20 lines, some type of compromise may be acceptable. For example, raise > a custom exception for some well-defined high-impact upper layer to do > re-resolve/retry, while other will have to restart. For TRUNK, the HA work > will most likely determine what needs to be done. So this Jira won't cover > the solutions for TRUNK. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7472) RPC client should deal with the IP address changes
[ https://issues.apache.org/jira/browse/HADOOP-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-7472: Resolution: Fixed Status: Resolved (was: Patch Available) I committed the patch to trunk and 0.20.security. Thank you Kihwal. > RPC client should deal with the IP address changes > -- > > Key: HADOOP-7472 > URL: https://issues.apache.org/jira/browse/HADOOP-7472 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 0.20.205.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Minor > Fix For: 0.20.205.0, 0.23.0 > > Attachments: addr_change_dfs-1.patch.txt, > addr_change_dfs-2.patch.txt, addr_change_dfs-3.patch.txt, > addr_change_dfs.patch.txt, addr_change_dfs_0_20s-1.patch.txt, > addr_change_dfs_0_20s-2.patch.txt, addr_change_dfs_0_20s.patch.txt, > addr_change_dfs_trunk-1.patch.txt, addr_change_dfs_trunk-2.patch.txt, > addr_change_dfs_trunk-3.patch.txt, addr_change_dfs_trunk.patch.txt > > > The current RPC client implementation and the client-side callers assume that > the hostname-address mappings of servers never change. The resolved address > is stored in an immutable InetSocketAddress object above/outside RPC, and the > reconnect logic in the RPC Connection implementation also trusts the resolved > address that was passed down. > If the NN suffers a failure that requires migration, it may be started on a > different node with a different IP address. In this case, even if the > name-address mapping is updated in DNS, the cluster is stuck trying old > address until the whole cluster is restarted. > The RPC client-side should detect this situation and exit or try to recover. > Updating ConnectionId within the Client implementation may get the system > work for the moment, there always is a risk of the cached address:port become > connectable again unintentionally. The real solution will be notifying upper > layer of the address change so that they can re-resolve and retry or > re-architecture the system as discussed in HDFS-34. > For 0.20 lines, some type of compromise may be acceptable. For example, raise > a custom exception for some well-defined high-impact upper layer to do > re-resolve/retry, while other will have to restart. For TRUNK, the HA work > will most likely determine what needs to be done. So this Jira won't cover > the solutions for TRUNK. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7499) Add method for doing a sanity check on hostnames in NetUtils
[ https://issues.apache.org/jira/browse/HADOOP-7499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082548#comment-13082548 ] Hadoop QA commented on HADOOP-7499: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12489887/HADOOP-7499-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//artifact/trunk/target/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//console This message is automatically generated. > Add method for doing a sanity check on hostnames in NetUtils > > > Key: HADOOP-7499 > URL: https://issues.apache.org/jira/browse/HADOOP-7499 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 0.23.0 >Reporter: Jeffrey Naisbitt >Assignee: Jeffrey Naisbitt > Fix For: 0.23.0 > > Attachments: HADOOP-7499-v2.patch, HADOOP-7499.patch, > patchJavadocWarnings.txt > > > As part of MAPREDUCE-2489, we need a method in NetUtils to do a sanity check > on hostnames -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7498) Remove legacy TAR layout creation
[ https://issues.apache.org/jira/browse/HADOOP-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082529#comment-13082529 ] Eric Yang commented on HADOOP-7498: --- +1 for removing the legacy tar layout. With result of the discussion on HADOOP-7521, we should support using both isolated directory layout as well as merged layout. Hence, the shell script needs to check and act correctly to the corresponding layout. > Remove legacy TAR layout creation > - > > Key: HADOOP-7498 > URL: https://issues.apache.org/jira/browse/HADOOP-7498 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur > Fix For: 0.23.0 > > > Currently the build creates 2 different tarball layouts. > One is the legacy one, the layout used until 0.22 (ant tar & mvn package > -Ptar) > The other is new new one, the layout used in trunk that mimics the Unix > layout (ant binary & mvn package -Pbintar). > The legacy layout is of not use as all the scripts have been modified to work > with the new layout only. > We should thus remove the legacy layout generation. > In addition we could rename the current 'bintar' to just 'tar' -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7534) Make test-patch.sh ensure that dependent projects can compile
[ https://issues.apache.org/jira/browse/HADOOP-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082528#comment-13082528 ] Alejandro Abdelnur commented on HADOOP-7534: With HADOOP-7533 this should be taken care of. * If test-patch is run from trunk/ it would use compile and use all bits from the build. * If test-patch is run from trunk/hadoop-hdfs, it would compile only hadoop-hdfs and it would use hadoop-common from the Maven repo. > Make test-patch.sh ensure that dependent projects can compile > - > > Key: HADOOP-7534 > URL: https://issues.apache.org/jira/browse/HADOOP-7534 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: 0.23.0 >Reporter: Aaron T. Myers > > Every once in a while a change in Common causes HDFS or M/R to not compile, > or a change in HDFS causes M/R to not compile. With the project unsplit, it > wouldn't be difficult to make test-patch.sh simply ensure that all the Hadoop > sub-projects continue to compile for any given change. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7498) Remove legacy TAR layout creation
[ https://issues.apache.org/jira/browse/HADOOP-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082525#comment-13082525 ] Giridharan Kesavan commented on HADOOP-7498: +1 for removing the legacy tar layout and renaming bintar to tar. > Remove legacy TAR layout creation > - > > Key: HADOOP-7498 > URL: https://issues.apache.org/jira/browse/HADOOP-7498 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur > Fix For: 0.23.0 > > > Currently the build creates 2 different tarball layouts. > One is the legacy one, the layout used until 0.22 (ant tar & mvn package > -Ptar) > The other is new new one, the layout used in trunk that mimics the Unix > layout (ant binary & mvn package -Pbintar). > The legacy layout is of not use as all the scripts have been modified to work > with the new layout only. > We should thus remove the legacy layout generation. > In addition we could rename the current 'bintar' to just 'tar' -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7530) The landing pages for Apache Hadoop website has an incorrect spelling for the word "Foundation" in the Trademark statement
[ https://issues.apache.org/jira/browse/HADOOP-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HADOOP-7530: --- Assignee: Chris Smith Hi Chris, Thanks for filing the issue and uploading a patch. Looks like the patch is out of date, mind generating one against the current trunk? Thanks, Eli > The landing pages for Apache Hadoop website has an incorrect spelling for the > word "Foundation" in the Trademark statement > -- > > Key: HADOOP-7530 > URL: https://issues.apache.org/jira/browse/HADOOP-7530 > Project: Hadoop Common > Issue Type: Task > Components: documentation >Affects Versions: site > Environment: Apache Hadoop Website >Reporter: Chris Smith >Assignee: Chris Smith >Priority: Trivial > Labels: documentation, website > Fix For: site > > Attachments: HADOOP-7530.patch > > > The landing pages for at least these URLs has an incorrect spelling for the > word "Foundation" in the Trademark statement in the page footer: > http://hadoop.apache.org/ > http://hadoop.apache.org/common/ > http://hadoop.apache.org/hdfs/ > http://hadoop.apache.org/mapreduce/ > Needs someone with the necessary privilges to change the spelling from > "Foundataion" to "Foundation". -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082488#comment-13082488 ] Eli Collins commented on HADOOP-7531: - Btw that still requires catching exceptions, unlike Jetty's URIUtil all commons URIUtil encode/decode methods throw URIException, which is lame since most people want standard byte-by-byte encoding. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7531) Add servlet util methods for handling paths in requests
[ https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HADOOP-7531: Attachment: hadoop-7531-1.patch Agree specifying UTF-8 explicitly is better. Updated patch attached. > Add servlet util methods for handling paths in requests > > > Key: HADOOP-7531 > URL: https://issues.apache.org/jira/browse/HADOOP-7531 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Affects Versions: 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: hadoop-7531-1.patch, hadoop-7531-1.patch, > hadoop-7531-2.patch > > > Common side of HDFS-2235. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7534) Make test-patch.sh ensure that dependent projects can compile
Make test-patch.sh ensure that dependent projects can compile - Key: HADOOP-7534 URL: https://issues.apache.org/jira/browse/HADOOP-7534 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: 0.23.0 Reporter: Aaron T. Myers Every once in a while a change in Common causes HDFS or M/R to not compile, or a change in HDFS causes M/R to not compile. With the project unsplit, it wouldn't be difficult to make test-patch.sh simply ensure that all the Hadoop sub-projects continue to compile for any given change. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7533) Allow test-patch to be run from any subproject directory
[ https://issues.apache.org/jira/browse/HADOOP-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082458#comment-13082458 ] Aaron T. Myers commented on HADOOP-7533: +1, patch looks good to me. > Allow test-patch to be run from any subproject directory > - > > Key: HADOOP-7533 > URL: https://issues.apache.org/jira/browse/HADOOP-7533 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Tom White >Assignee: Tom White > Attachments: HADOOP-7533.patch > > > Currently dev-support/test-patch.sh can only be run from the top-level (and > only for hadoop-common). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7531) Add servlet util methods for handling paths in requests
[
https://issues.apache.org/jira/browse/HADOOP-7531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082346#comment-13082346
]
Daryn Sharp commented on HADOOP-7531:
-
bq. And we wouldn't let someone check that in.. =)
I'm worried because it's a latent land-mine. Over 20 years I've debugged so
many "it'll never happen" bugs that I strongly adhere to Murphy's Law. Someday
the charset will be changed, either in error or with legit purpose, or an
external jar will fiddle with the charset, or a deployment env will use an
incompatible charset, etc. When that day arrives, some poor developer will
spend too much time debugging why the web page acts goofy instead of
immediately seeing an invalid charset exception.
Perhaps the safest way is to not catch the exception and use
{{URIUtil.encodeXXX(String, "UTF-8")}}?
> Add servlet util methods for handling paths in requests
>
>
> Key: HADOOP-7531
> URL: https://issues.apache.org/jira/browse/HADOOP-7531
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 0.23.0
>
> Attachments: hadoop-7531-1.patch, hadoop-7531-2.patch
>
>
> Common side of HDFS-2235.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7493) [HDFS-362] Provide ShortWritable class in hadoop.
[ https://issues.apache.org/jira/browse/HADOOP-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HADOOP-7493: Attachment: HADOOP-7493.1.patch > [HDFS-362] Provide ShortWritable class in hadoop. > - > > Key: HADOOP-7493 > URL: https://issues.apache.org/jira/browse/HADOOP-7493 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HADOOP-7493.1.patch, HADOOP-7493.patch > > > As part of HDFS-362, Provide the ShortWritable class. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7493) [HDFS-362] Provide ShortWritable class in hadoop.
[ https://issues.apache.org/jira/browse/HADOOP-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HADOOP-7493: Status: Patch Available (was: Open) Updated patch against trunk. > [HDFS-362] Provide ShortWritable class in hadoop. > - > > Key: HADOOP-7493 > URL: https://issues.apache.org/jira/browse/HADOOP-7493 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HADOOP-7493.1.patch, HADOOP-7493.patch > > > As part of HDFS-362, Provide the ShortWritable class. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7493) [HDFS-362] Provide ShortWritable class in hadoop.
[ https://issues.apache.org/jira/browse/HADOOP-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HADOOP-7493: Status: Open (was: Patch Available) > [HDFS-362] Provide ShortWritable class in hadoop. > - > > Key: HADOOP-7493 > URL: https://issues.apache.org/jira/browse/HADOOP-7493 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HADOOP-7493.1.patch, HADOOP-7493.patch > > > As part of HDFS-362, Provide the ShortWritable class. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7530) The landing pages for Apache Hadoop website has an incorrect spelling for the word "Foundation" in the Trademark statement
[ https://issues.apache.org/jira/browse/HADOOP-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Smith updated HADOOP-7530: Fix Version/s: site Affects Version/s: site Release Note: Fix to the ..xsl files to correct the spelling of "Foundation" that appears on the home page of http://hadoop.apache.org and /common, /hdfs and /mapreduce sub-sites. Status: Patch Available (was: Open) > The landing pages for Apache Hadoop website has an incorrect spelling for the > word "Foundation" in the Trademark statement > -- > > Key: HADOOP-7530 > URL: https://issues.apache.org/jira/browse/HADOOP-7530 > Project: Hadoop Common > Issue Type: Task > Components: documentation >Affects Versions: site > Environment: Apache Hadoop Website >Reporter: Chris Smith >Priority: Trivial > Labels: documentation, website > Fix For: site > > Attachments: HADOOP-7530.patch > > > The landing pages for at least these URLs has an incorrect spelling for the > word "Foundation" in the Trademark statement in the page footer: > http://hadoop.apache.org/ > http://hadoop.apache.org/common/ > http://hadoop.apache.org/hdfs/ > http://hadoop.apache.org/mapreduce/ > Needs someone with the necessary privilges to change the spelling from > "Foundataion" to "Foundation". -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7530) The landing pages for Apache Hadoop website has an incorrect spelling for the word "Foundation" in the Trademark statement
[ https://issues.apache.org/jira/browse/HADOOP-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Smith updated HADOOP-7530: Attachment: HADOOP-7530.patch Patch for JIRA HADOOP-7530 to correct spelling mistakes for the word "Foundation" in the .xsl stylesheets that generate the .html pages. > The landing pages for Apache Hadoop website has an incorrect spelling for the > word "Foundation" in the Trademark statement > -- > > Key: HADOOP-7530 > URL: https://issues.apache.org/jira/browse/HADOOP-7530 > Project: Hadoop Common > Issue Type: Task > Components: documentation > Environment: Apache Hadoop Website >Reporter: Chris Smith >Priority: Trivial > Labels: documentation, website > Attachments: HADOOP-7530.patch > > > The landing pages for at least these URLs has an incorrect spelling for the > word "Foundation" in the Trademark statement in the page footer: > http://hadoop.apache.org/ > http://hadoop.apache.org/common/ > http://hadoop.apache.org/hdfs/ > http://hadoop.apache.org/mapreduce/ > Needs someone with the necessary privilges to change the spelling from > "Foundataion" to "Foundation". -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7489) Hadoop logs errors upon startup on OS X 10.7
[
https://issues.apache.org/jira/browse/HADOOP-7489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082244#comment-13082244
]
Matthew Buckett commented on HADOOP-7489:
-
I'm also seeing this on another project (on OS X 10.7) and it's happening when
I attempt to perform a login using a keytab file. No idea why yet.
> Hadoop logs errors upon startup on OS X 10.7
>
>
> Key: HADOOP-7489
> URL: https://issues.apache.org/jira/browse/HADOOP-7489
> Project: Hadoop Common
> Issue Type: Bug
> Environment: Mac OS X 10.7, Java 1.6.0_26
>Reporter: Bryan Keller
>Priority: Minor
>
> When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs
> the following errors:
> 2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from
> SCDynamicStore
> Hadoop does seem to function properly despite this.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
