[jira] [Commented] (HDFS-5382) Implement the UI of browsing filesystems in HTML 5 page
[
https://issues.apache.org/jira/browse/HDFS-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799942#comment-13799942
]
Philip Zeyliger commented on HDFS-5382:
---
{code}
+
+
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[ https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469050#comment-13469050 ] Philip Zeyliger commented on HDFS-347: -- That error message is typically a statement that the reserved disk space requirement makes it so that the datanodes have no more space. (Or, the datanodes really don't have any more space.) > DFS read performance suboptimal when client co-located on nodes with data > - > > Key: HDFS-347 > URL: https://issues.apache.org/jira/browse/HDFS-347 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, hdfs client, performance >Reporter: George Porter >Assignee: Colin Patrick McCabe > Attachments: all.tsv, BlockReaderLocal1.txt, HADOOP-4801.1.patch, > HADOOP-4801.2.patch, HADOOP-4801.3.patch, HDFS-347-016_cleaned.patch, > HDFS-347.016.patch, HDFS-347.017.clean.patch, HDFS-347.017.patch, > HDFS-347.018.clean.patch, HDFS-347.018.patch, HDFS-347-branch-20-append.txt, > hdfs-347.png, hdfs-347.txt, local-reads-doc > > > One of the major strategies Hadoop uses to get scalable data processing is to > move the code to the data. However, putting the DFS client on the same > physical node as the data blocks it acts on doesn't improve read performance > as much as expected. > After looking at Hadoop and O/S traces (via HADOOP-4049), I think the problem > is due to the HDFS streaming protocol causing many more read I/O operations > (iops) than necessary. Consider the case of a DFSClient fetching a 64 MB > disk block from the DataNode process (running in a separate JVM) running on > the same machine. The DataNode will satisfy the single disk block request by > sending data back to the HDFS client in 64-KB chunks. In BlockSender.java, > this is done in the sendChunk() method, relying on Java's transferTo() > method. Depending on the host O/S and JVM implementation, transferTo() is > implemented as either a sendfilev() syscall or a pair of mmap() and write(). > In either case, each chunk is read from the disk by issuing a separate I/O > operation for each chunk. The result is that the single request for a 64-MB > block ends up hitting the disk as over a thousand smaller requests for 64-KB > each. > Since the DFSClient runs in a different JVM and process than the DataNode, > shuttling data from the disk to the DFSClient also results in context > switches each time network packets get sent (in this case, the 64-kb chunk > turns into a large number of 1500 byte packet send operations). Thus we see > a large number of context switches for each block send operation. > I'd like to get some feedback on the best way to address this, but I think > providing a mechanism for a DFSClient to directly open data blocks that > happen to be on the same machine. It could do this by examining the set of > LocatedBlocks returned by the NameNode, marking those that should be resident > on the local host. Since the DataNode and DFSClient (probably) share the > same hadoop configuration, the DFSClient should be able to find the files > holding the block data, and it could directly open them and send data back to > the client. This would avoid the context switches imposed by the network > layer, and would allow for much larger read buffers than 64KB, which should > reduce the number of iops imposed by each read block operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3447) StandbyException should not be logged at ERROR level on server
[ https://issues.apache.org/jira/browse/HDFS-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Zeyliger updated HDFS-3447: -- Target Version/s: (was: ) Labels: newbie (was: ) > StandbyException should not be logged at ERROR level on server > -- > > Key: HDFS-3447 > URL: https://issues.apache.org/jira/browse/HDFS-3447 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Affects Versions: 2.0.0-alpha >Reporter: Todd Lipcon >Priority: Minor > Labels: newbie > > Currently, the standby NN will log StandbyExceptions at ERROR level any time > a client tries to connect to it. So, if the second NN in an HA pair is > active, the first NN will spew a lot of these errors in the log, as each > client gets redirected to the proper NN. Instead, this should be at INFO > level, and should probably be logged in a less "scary" manner (eg "Received > READ request from client 1.2.3.4, but in Standby state. Redirecting client to > other NameNode.") -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1958) Format confirmation prompt should be more lenient of its input
[ https://issues.apache.org/jira/browse/HDFS-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035895#comment-13035895 ] Philip Zeyliger commented on HDFS-1958: --- If we're editing this code, the fact that it can't be forced on the command line, and the fact that it treats an empty directory as one that already has a namenode image on it are also usability issues if you're scripting around namenode formatting. > Format confirmation prompt should be more lenient of its input > -- > > Key: HDFS-1958 > URL: https://issues.apache.org/jira/browse/HDFS-1958 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: hdfs-1958.txt > > > As reported on the mailing list, the namenode format prompt only accepts 'Y'. > We should also accept 'y' and 'yes' (non-case-sensitive). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1971) HA: Send block report from datanode to both active and standby namenodes
[ https://issues.apache.org/jira/browse/HDFS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088729#comment-13088729 ] Philip Zeyliger commented on HDFS-1971: --- I will return on the 24th. For urgent matters, please contact my teammates or Amr. Thanks, -- Philip > HA: Send block report from datanode to both active and standby namenodes > > > Key: HDFS-1971 > URL: https://issues.apache.org/jira/browse/HDFS-1971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, name-node >Reporter: Suresh Srinivas >Assignee: Sanjay Radia > Attachments: DualBlockReports.pdf > > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the block location information > current in the standby node. To do this, the proposed solution is to send > block reports from the datanodes to both the active and the standby namenode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5128) Allow multiple net interfaces to be used with HA namenode RPC server
[
https://issues.apache.org/jira/browse/HDFS-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749511#comment-13749511
]
Philip Zeyliger commented on HDFS-5128:
---
It would be slightly more consistent to expose this as "rpc bind address" and
have users specify 0.0.0.0 if they want to bind to the wildcard. The existing
bug and bad behavior is because {{dfs.namenode.servicerpc-address}} doubles as
both the address that clients should use to connect and the address that the
server should bind on. If you want to be fully flexible, you may as well
separate those two. Introducing a toggle for "bind to wildcard" is certainly
clear (and, in fact, it's exactly the toggle that we try to expose in Cloudera
Manager as a general practice), but since many other addresses in the Hadoop
ecosystem already require you to specify 0.0.0.0 to bind to the wildcard (e.g.,
that's what you do for the datanode), it might be more consistent to have a
bind address config option.
> Allow multiple net interfaces to be used with HA namenode RPC server
>
>
> Key: HDFS-5128
> URL: https://issues.apache.org/jira/browse/HDFS-5128
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Attachments: HDFS-5128.branch-2.patch, HDFS-5128.trunk.patch
>
>
> With the new RPCv9, NN RPC server can be bound to multiple IP addresses and
> serve clients connecting to different addressed using the same SPN. But the
> only way to have NN bind to multiple addresses is to set its RPC address to
> 0.0.0.0. This does not work well with HA especially when name nodes share the
> same config.
> I propose a new option for NN rpc server to bind to all interfaces,
> independent of the rpc address configuration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13763665#comment-13763665
]
Philip Zeyliger commented on HDFS-5122:
---
Haohui,
Your patch doesn't seem to handle the possibility that the first namenode isn't
the currently active one. So it's not just not handling failover during the
lifetime of the client, but it's not handling a case where the configuration
happens to have the currently active namenode first. It seems like the code
ought to at least try to access (with a timeout) the namenode to see if it's
active, no?
> WebHDFS should support logical service names in URIs
>
>
> Key: HDFS-5122
> URL: https://issues.apache.org/jira/browse/HDFS-5122
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.1.0-beta
>Reporter: Arpit Gupta
>Assignee: Haohui Mai
> Attachments: HDFS-5122.patch
>
>
> For example if the dfs.nameservices is set to arpit
> {code}
> hdfs dfs -ls webhdfs://arpit:50070/tmp
> or
> hdfs dfs -ls webhdfs://arpit/tmp
> {code}
> does not work
> You have to provide the exact active namenode hostname. On an HA cluster
> using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5237) Get rid of nodes' registration names
[
https://issues.apache.org/jira/browse/HDFS-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773074#comment-13773074
]
Philip Zeyliger commented on HDFS-5237:
---
I don't think this is a great idea. Hadoop's not been great at separating out
a few things:
* The bind addresses and ports for RPC and HTTP. (Particularly whether to bind
on 0.0.0.0 or on a single IP.)
* The way to identify this process to other people so that they can reach it
* How to use DNS to reach other folks
There are, I agree, very minimal defaults here. Bind to the wildcard.
Identify as {{hostname -f}}. Use default DNS settings. Don't talk to DNS
explicitly but use the normal OS ways to do it.
{{slave.host.name}} has, I think, made it possible to do some less frequent but
still important configurations. I see this most often as a host having 2 names
("management-name" and "data-name") where "management-name" is the one that's
hostname. I've seen it in use in real configurations (not just in tests) where
it did the trick.
Meanwhile, {{DNS.getHosts(String strInterface, String nameserver)}} is a very
odd way to configure the hostname that a process should identify as. It says,
"hey, use this other system and do a reverse lookup for yourself with these
parameters." Most of the time, though, when you're in this case, you're
already generating your configuration from something that knows exactly what
IP/name you want, and it would be simpler and less error-prone to just specify
it.
Cheers,
Philip
> Get rid of nodes' registration names
>
>
> Key: HDFS-5237
> URL: https://issues.apache.org/jira/browse/HDFS-5237
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Junping Du
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: HDFS-5237.patch
>
>
> Per discussion in HDFS-5208 and may be some other discussions before, Node's
> registration name is pretty confusing and shouldn't be used in production
> environment as topology resolving issues. So we remove related configuration
> dfs.datanode.hostname or its old one "slave.host.name".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5237) Get rid of nodes' registration names
[ https://issues.apache.org/jira/browse/HDFS-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773861#comment-13773861 ] Philip Zeyliger commented on HDFS-5237: --- Colin: I agree--totally reasonable for dfs.datanode.hostname to be a hostname. (Or, at least, something that you can connect to (ip or hostname).) > Get rid of nodes' registration names > > > Key: HDFS-5237 > URL: https://issues.apache.org/jira/browse/HDFS-5237 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-5237.patch > > > Per discussion in HDFS-5208 and may be some other discussions before, Node's > registration name is pretty confusing and shouldn't be used in production > environment as topology resolving issues. So we remove related configuration > dfs.datanode.hostname or its old one "slave.host.name". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1179) Implement a file change log
[ https://issues.apache.org/jira/browse/HDFS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873046#action_12873046 ] Philip Zeyliger commented on HDFS-1179: --- Conceptually, this could be a side output of whoever rolls up the image+edits files into a new image, yes? -- Philip > Implement a file change log > --- > > Key: HDFS-1179 > URL: https://issues.apache.org/jira/browse/HDFS-1179 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > The File Change Log (FCL) is a construct that records all metadata changes in > a file system (http://en.wikipedia.org/wiki/File_change_log). This JIRA is to > discuss various high-level approaches to implementing such a FCL for HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1435) Provide an option to store fsimage compressed
[ https://issues.apache.org/jira/browse/HDFS-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916793#action_12916793 ] Philip Zeyliger commented on HDFS-1435: --- Instead of changing the binary format, could you update code to either read fsimage or fsimage.gz, whichever is available? Obviously, one could start compressing at any point, but there's significant value to being able to use existing tools to decompress if anything goes awry. > Provide an option to store fsimage compressed > - > > Key: HDFS-1435 > URL: https://issues.apache.org/jira/browse/HDFS-1435 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.22.0 > > > Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network > bandwidth when secondary NN uploads a new fsimage to primary NN. > If we could store fsimage compressed, the problem could be greatly alleviated. > I plan to provide a new configuration hdfs.image.compressed with a default > value of false. If it is set to be true, fsimage is stored as compressed. > The fsimage will have a new layout with a new field "compressed" in its > header, indicating if the namespace is stored compressed or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1456) Provide builder for constructing instances of MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920835#action_12920835
]
Philip Zeyliger commented on HDFS-1456:
---
+1. I'm a big fan of this suggestion.
One thing that might be interesting would be to "cache" MiniDFSClusters. I
have a theory (not tested) that many of the tests would work just fine using
the same cluster over and over again. If we had some sort of default cached
instance, the tests might pass faster.
> Provide builder for constructing instances of MiniDFSCluster
>
>
> Key: HDFS-1456
> URL: https://issues.apache.org/jira/browse/HDFS-1456
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.22.0
>Reporter: Jakob Homan
>Assignee: Jakob Homan
> Fix For: 0.22.0
>
> Attachments: HDFS-1456.patch
>
>
> Time to fix a broken window. Of the 293 occurences of "new
> MiniDFSCluster("... most look something like:
> {noformat}cluster = new MiniDFSCluster(0, config, numDatanodes, true, false,
> true, null, null, null, null);{noformat}
> The largest constructor takes 10 parameters, and even the overloaded
> constructors can be difficult to read as their mainaly nulls or booleans.
> We should provide a Builder for constructing MiniDFSClusters to improve
> readability.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1547) Improve decommission mechanism
[ https://issues.apache.org/jira/browse/HDFS-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973399#action_12973399 ] Philip Zeyliger commented on HDFS-1547: --- Hi Suresh, For decomissioning, it's always seemed a bit odd that interacting with the system requires editing a file local to the namenode as well as sending an RPC. Have you considered adding RPCs (and CLI wrappers for them) for explicitly updating data node states, without going through a file? Thanks, -- Philip > Improve decommission mechanism > -- > > Key: HDFS-1547 > URL: https://issues.apache.org/jira/browse/HDFS-1547 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > > Current decommission mechanism driven using exclude file has several issues. > This bug proposes some changes in the mechanism for better manageability. See > the proposal in the next comment for more details. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HDFS-1750) fs -ls hftp://file not working
[
https://issues.apache.org/jira/browse/HDFS-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010890#comment-13010890
]
Philip Zeyliger commented on HDFS-1750:
---
I'm traveling and will return to the office on Monday, March 28th.
For urgent matters, please contact Aparna Ramani.
Thanks!
-- Philip
> fs -ls hftp://file not working
> --
>
> Key: HDFS-1750
> URL: https://issues.apache.org/jira/browse/HDFS-1750
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.21.1
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.204, 0.21.1, 0.22.0, 0.23.0
>
> Attachments: h1750_20110314.patch,
> h1750_20110314_0.20-security.patch, h1750_20110314_0.21.patch
>
>
> {noformat}
> hadoop dfs -touchz /tmp/file1 # create file. OK
> hadoop dfs -ls /tmp/file1 # OK
> hadoop dfs -ls hftp://namenode:50070/tmp/file1 # FAILED: not seeing the file
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1873) Federation Cluster Management Web Console
[ https://issues.apache.org/jira/browse/HDFS-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027497#comment-13027497 ] Philip Zeyliger commented on HDFS-1873: --- Hi, Absolutely excellent to have a web UI here; thanks for making it! A few comments on the JMX approach: (1) The namenode already has a port open for the Namenode RPC protocol and another one open for HTTP. Have you considered adding a getWhatever() API to the namenode RPC protocol or adding a Jetty servlet endpoint for the same? Either approach would require less configuration. We already have two perfectly good protocols; why bother with a third? ( (2) You provide a way to set the agent port, but, ultimately, there are a lot of properties involved in setting up JMX (http://download.oracle.com/javase/6/docs/technotes/guides/management/agent.html has ones for security, SSL, passwords and credentials, etc.) As you know, once you connect via JMX, you can do pretty much anything, including injecting code into the namenode. It seems that setting '-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false' (as you do in one of the patches in the subissues) by default is unwise from a security perspective. (3) The method you use to get the config from hdfs-site to the shell script ends up creating an extra JVM. That's not terrible: it's not that Hadoop startup time is particularly speedy to begin with, but we already have options in hadoop-env.sh (like heapsize), so the extra mechanism feels like it may be gratuitous. I looked around to see if there was a way to start (or open) the JMX agent port programatically, but I did not find one. Cheers, -- Philip > Federation Cluster Management Web Console > - > > Key: HDFS-1873 > URL: https://issues.apache.org/jira/browse/HDFS-1873 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang > Fix For: 0.23.0 > > Attachments: ClusterSummary.png, Decommission.png, HDFS-1873.patch > > > The Federation cluster management console provides > # Cluster summary information that shows overall cluster utilization. A list > of the name nodes that reports the used space, files and directories, blocks, > live and dead datanodes > of each name space. > # decommissioning status of all the data nodes who are decommissioning in > process or decommissioned. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-13797) WebHDFS redirects to datanodes by hostname, ignoring bind addresses
Philip Zeyliger created HDFS-13797:
--
Summary: WebHDFS redirects to datanodes by hostname, ignoring bind
addresses
Key: HDFS-13797
URL: https://issues.apache.org/jira/browse/HDFS-13797
Project: Hadoop HDFS
Issue Type: Task
Components: webhdfs
Reporter: Philip Zeyliger
In testing Impala, we run datanodes bound to {{127.0.0.1}}. This works great,
except for WebHDFS, where the namenode redirects to a datanode by hostname,
leading to a "connection refused" error (since the datanode isn't listening).
Our workaround in Impala is to map hostname to {{127.0.0.1}} in {{/etc/hosts}}
which isn't so hot. (The workaround is at
https://github.com/apache/impala/blob/3334c167a69a6506d595898dff73fb16410266e6/bin/bootstrap_system.sh#L187
.)
I think there's a bug here. Reading
https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html,
it looks like WebHDFS should perhaps respect
{{dfs.client.use.datanode.hostname}}, but I think it doesn't and always uses
the hostname. I think the relevant point where the namenode figures this out is
https://github.com/apache/hadoop/blob/ca20e0d7e9767a7362dddfea8ec19548947d3fd7/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L406
.
I recognize this is a non-production edge case: it makes no sense to listen
only on localhost except for testing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] Commented: (HDFS-599) Improve Namenode robustness by prioritizing datanode heartbeats over client requests
[ https://issues.apache.org/jira/browse/HDFS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753235#action_12753235 ] Philip Zeyliger commented on HDFS-599: -- In my experience, using different ports for QoS turns out to be compelling because you can tell fancy-pants switches to prioritize traffic at port X over traffic at port Y, but it's harder (or impossible) to get a switch to prioritize based on what's in the packet itself. Perhaps Hadoop could be largely agnostic over which of a handful of open ports data is coming in, relying on the authorization layer to kick out client requests on a port that's supposed to be only for elevated-priority requests. > Improve Namenode robustness by prioritizing datanode heartbeats over client > requests > > > Key: HDFS-599 > URL: https://issues.apache.org/jira/browse/HDFS-599 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: dhruba borthakur > > The namenode processes RPC requests from clients that are reading/writing to > files as well as heartbeats/block reports from datanodes. > Sometime, because of various reasons (Java GC runs, inconsistent performance > of NFS filer that stores HDFS transacttion logs, etc), the namenode > encounters transient slowness. For example, if the device that stores the > HDFS transaction logs becomes sluggish, the Namenode's ability to process > RPCs slows down to a certain extent. During this time, the RPCs from clients > as well as the RPCs from datanodes suffer in similar fashion. If the > underlying problem becomes worse, the NN's ability to process a heartbeat > from a DN is severly impacted, thus causing the NN to declare that the DN is > dead. Then the NN starts replicating blocks that used to reside on the > now-declared-dead datanode. This adds extra load to the NN. Then the > now-declared-datanode finally re-establishes contact with the NN, and sends a > block report. The block report processing on the NN is another heavyweight > activity, thus casing more load to the already overloaded namenode. > My proposal is tha the NN should try its best to continue processing RPCs > from datanodes and give lesser priority to serving client requests. The > Datanode RPCs are integral to the consistency and performance of the Hadoop > file system, and it is better to protect it at all costs. This will ensure > that NN recovers from the hiccup much faster than what it does now. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-620) HDFS code is extensively using system property 'test.build.data' which is likely to fail under Eclipse environment
[ https://issues.apache.org/jira/browse/HDFS-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755574#action_12755574 ] Philip Zeyliger commented on HDFS-620: -- FWIW, the Eclipse templates in https://issues.apache.org/jira/browse/HDFS-598 specify test.build.data, and a couple other things. That said, I'm very much +100 for standardizing on how to get at this sort of thing from code, as opposed to environment variables. > HDFS code is extensively using system property 'test.build.data' which is > likely to fail under Eclipse environment > -- > > Key: HDFS-620 > URL: https://issues.apache.org/jira/browse/HDFS-620 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Konstantin Boudnik > > Because some of the system properties aren't available in Eclipse environment > and there's a sufficient number usages of 'test.data.dir' system property the > chances are that some of the tests won't work in Eclipse. There were two > occasions of such behavior which were fixed already (see HDFS-601, HDFS-614). > Although many more are left intact for post append time frame -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-598) Eclipse launch task for HDFS
[ https://issues.apache.org/jira/browse/HDFS-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Zeyliger updated HDFS-598: - Attachment: HDFS-598-v2.patch Yep, Tom, didn't check that one after the backport. The move into hdfs and hdfs-with-mr confused it, too. I've updated the patch to work. > Eclipse launch task for HDFS > > > Key: HDFS-598 > URL: https://issues.apache.org/jira/browse/HDFS-598 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build > Environment: Eclipse 3.5 >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Trivial > Attachments: HDFS-598-v2.patch, hdfs-598.patch > > > Porting HDFS launch task from HADOOP-5911. See MAPREDUCE-905. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: HDFS-621
URL: https://issues.apache.org/jira/browse/HDFS-621
Project: Hadoop HDFS
Issue Type: New Feature
Components: test, tools
Reporter: Philip Zeyliger
Priority: Minor
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
"bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a cluster
(internally, it's using Mini{MR,HDFS}Cluster) with a specified number of
daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and then
simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it. It's
significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others might
find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests have
all the required libraries, so I've put it there. I could conceivably split
this into "minimr" and "minihdfs", but it's specifically the fact that they're
configured to talk to each other that I like about having them together. And
one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/HDFS-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Zeyliger updated HDFS-621:
-
Attachment: HDFS-621.patch
Uploading patch to start a single-process cluster.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: HDFS-621
> URL: https://issues.apache.org/jira/browse/HDFS-621
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: test, tools
>Reporter: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/HDFS-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Zeyliger updated HDFS-621:
-
Assignee: Philip Zeyliger
Status: Patch Available (was: Open)
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: HDFS-621
> URL: https://issues.apache.org/jira/browse/HDFS-621
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: test, tools
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/HDFS-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755732#action_12755732
]
Philip Zeyliger commented on HDFS-621:
--
bq. I guess you need a separated MAPREDUCE patch for MiniMR.
If you would, take a quick look to see how I use MiniMRCluster. Do you feel
I'm abusing the fact that hdfs-hdfswithmr-test exists?
bq. Is it better for the testing of the product?
It doesn't contribute anything to testing Hadoop. It does make it easier to
write tests for programs that are downstream of Hadoop. One might argue that
such facilities ought to be "top-level" and not shunted into a "-test" jar.
I'm open to suggestions. (I could also see this being shunted to contrib, if
folks have a strong preference.)
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: HDFS-621
> URL: https://issues.apache.org/jira/browse/HDFS-621
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: test, tools
>Reporter: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/HDFS-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755735#action_12755735
]
Philip Zeyliger commented on HDFS-621:
--
Owen,
I totally agree that LocalJobRunner should be maximally useful. That's great
for testing jobs.
Let's say I have a python class that knows how to interact with HDFS and MR.
It knows how to look at files, start jobs, etc. I call out to hadoop binaries
to interact with HDFS, and I want to capture all the details that occur when I
talk to my real cluster. For this, if I were in Java, I'd spin up a Mini*
cluster. Since I'm not in Java, I resort to spinning up a subprocess. I could
also mock everything out, but at the end of the day, I want an integration
test, and I really don't want to run it against a cluster that has to be setup
externally: I'd rather the cluster be spun up and shut down by my test itself.
I'm happy to throw this contrib/ if you feel strongly about it. I figure it'd
be useful to other folks.
-- Philip
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: HDFS-621
> URL: https://issues.apache.org/jira/browse/HDFS-621
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: test, tools
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/HDFS-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Zeyliger updated HDFS-621:
-
Attachment: HDFS-621-0.20-patch
Attaching the 0.20 version. Conveniently, where to place it is not a problem
there :)
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: HDFS-621
> URL: https://issues.apache.org/jira/browse/HDFS-621
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: test, tools
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-245) Create symbolic links in HDFS
[
https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756799#action_12756799
]
Philip Zeyliger commented on HDFS-245:
--
In my opinion (I'd say IME, but I've never sneaked one of these past code
review),using exceptions for flow control is shady. Joshua Bloch would say the
same ("Don't force client to use exceptions for control flow" see page 38,
http://74.125.155.132/search?q=cache:MmIXuflmcvcJ:lcsd05.cs.tamu.edu/slides/keynote.pdf+joshua+bloch+exceptions&cd=4&hl=en&ct=clnk&gl=us&client=firefox-a
or http://lcsd05.cs.tamu.edu/slides/keynote.pdf ).
If the worry is about boilerplate, it seems like this exception
would have to be checked --- otherwise, people would not know that
this exception is part of what they have to handle and implement ---
and nothing introduces misunderstood boilerplate quite as fast as
checked exceptions.
I can't think of many analogous interfaces here; the lack
of which probably makes this argument more contentious.
HTTP is vaguely similar: there's sometimes a full response,
and sometimes there's a redirection. In most APIs wrapping
HTTP that I've seen, you get a response object in response
to a request. TCP/IP errors (socket timeouts, name resolution, etc.)
throw exceptions, but nothing that has to do with HTTP does.
One inspects response.status to see if it's 200 (ok), 301 (redirect), etc.
-- Philip
> Create symbolic links in HDFS
> -
>
> Key: HDFS-245
> URL: https://issues.apache.org/jira/browse/HDFS-245
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: dhruba borthakur
>Assignee: Eli Collins
> Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch,
> symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch,
> symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt,
> symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch,
> symLink8.patch, symLink9.patch
>
>
> HDFS should support symbolic links. A symbolic link is a special type of file
> that contains a reference to another file or directory in the form of an
> absolute or relative path and that affects pathname resolution. Programs
> which read or write to files named by a symbolic link will behave as if
> operating directly on the target file. However, archiving utilities can
> handle symbolic links specially and manipulate them directly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756876#action_12756876 ] Philip Zeyliger commented on HDFS-245: -- I'm not up to date on the current lore of exceptions performance, but Java Exceptions tend to carry around stack frames of where they were thrown (Throwable has a handful of fields including StackTraceElement[]), and those are certainly heavier than any return value object we might construct. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-705) Create an adapter to access some of package-private methods of DataNode from tests
[
https://issues.apache.org/jira/browse/HDFS-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765781#action_12765781
]
Philip Zeyliger commented on HDFS-705:
--
In other places, I've seen things marked as public with a comment (we could
also do annotation) indicating that it ought to be used for testing only. If
the package is server.datanode, that's pretty clearly internal to HDFS, no?
I'm merely curious about the preferred coding style for this codebase.
Definitely something JSR 294 ("superpackages") will help address.
-- Philip
> Create an adapter to access some of package-private methods of DataNode from
> tests
> --
>
> Key: HDFS-705
> URL: https://issues.apache.org/jira/browse/HDFS-705
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: HDFS-705.patch
>
>
> Some of tests from packages other than
> {{org.apache.hadoop.hdfs.server.datanode}} need an access to package-private
> methods of that package. Proposed patch will add new adapter class which will
> have public getter methods to provide such access to outside of the package.
> This is TEST ONLY class: it never has to be used for ANY development purposes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-705) Create an adapter to access some of package-private methods of DataNode from tests
[
https://issues.apache.org/jira/browse/HDFS-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765797#action_12765797
]
Philip Zeyliger commented on HDFS-705:
--
Ok, thanks for the explanation.
> Create an adapter to access some of package-private methods of DataNode from
> tests
> --
>
> Key: HDFS-705
> URL: https://issues.apache.org/jira/browse/HDFS-705
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: HDFS-705.patch
>
>
> Some of tests from packages other than
> {{org.apache.hadoop.hdfs.server.datanode}} need an access to package-private
> methods of that package. Proposed patch will add new adapter class which will
> have public getter methods to provide such access to outside of the package.
> This is TEST ONLY class: it never has to be used for ANY development purposes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-705) Create an adapter to access some of package-private methods of DataNode from tests
[
https://issues.apache.org/jira/browse/HDFS-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765822#action_12765822
]
Philip Zeyliger commented on HDFS-705:
--
bq. In fact, I thought that it might be pretty cool idea to assert on
invocation class' name if it doesn't start with Test...
It's cute, but it seems unnecessary. If people are loading classes from
foo-test.jar, they deserve everything that goes wrong.
In a previous life, I simply stuck a VisibleForTesting annotation on things
(http://code.google.com/p/google-collections/source/browse/trunk/src/com/google/common/annotations/VisibleForTesting.java?spec=svn80&r=80)
and marked the method I needed public.
> Create an adapter to access some of package-private methods of DataNode from
> tests
> --
>
> Key: HDFS-705
> URL: https://issues.apache.org/jira/browse/HDFS-705
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: HDFS-705.patch, HDFS-705.patch
>
>
> Some of tests from packages other than
> {{org.apache.hadoop.hdfs.server.datanode}} need an access to package-private
> methods of that package. Proposed patch will add new adapter class which will
> have public getter methods to provide such access to outside of the package.
> This is TEST ONLY class: it never has to be used for ANY development purposes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-890) Have a way of creating datanodes that throws an meaningful exception on failure
[ https://issues.apache.org/jira/browse/HDFS-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798759#action_12798759 ] Philip Zeyliger commented on HDFS-890: -- Do you need to mark the old API as deprecated? Isn't the "documented" way of starting datanodes "bin/hdfs datanode"? If we can get away with it, fewer deprecated methods are better. I'm +1 having real exit codes. > Have a way of creating datanodes that throws an meaningful exception on > failure > --- > > Key: HDFS-890 > URL: https://issues.apache.org/jira/browse/HDFS-890 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > In HDFS-884, I proposed printing out more details on why things fail. This is > hard to test, because you need to subvert the log4j back end that your test > harness will itself have grabbed. > There is a way to make it testable, and to make it easier for anyone creating > datanodes in process to recognise and handle failure: have a static > CreateDatanode() method that throws exceptions when directories cannot be > created or other problems arise. Right now some problems trigger failure, > others just return a null reference saying "something went wrong but we won't > tell you what -hope you know where the logs go". > The HDFS-884 patch would be replaced by something that threw an exception; > the existing methods would catch this, log it and return null. The new method > would pass it straight up. > This is easier to test, better for others. If people think this is good, I > will code it up and mark the old API as deprecated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829185#action_12829185 ] Philip Zeyliger commented on HDFS-945: -- A solution is rate limiting (by client IP or user id or something else). It ain't fool-proof, but it would probably get the job done. > Make NameNode resilient to DoS attacks (malicious or otherwise) > --- > > Key: HDFS-945 > URL: https://issues.apache.org/jira/browse/HDFS-945 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Arun C Murthy > > We've seen defective applications cause havoc on the NameNode, for e.g. by > doing 100k+ 'listStatus' on very large directories (60k files) etc. > I'd like to start a discussion around how we prevent such, and possibly > malicious applications in the future, taking down the NameNode. > Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-991) Allow browsing the filesystem over http using delegation tokens
[
https://issues.apache.org/jira/browse/HDFS-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837963#action_12837963
]
Philip Zeyliger commented on HDFS-991:
--
Hi Owen,
When does the delegation token that the namenode provides
(redirectToRandomDataNode()) expire?
A lot of websites pass security tokens via cookies, because GET parameters tend
to get written down in referrer fields and such. So there's the potential that
someone will get their hands on your token. Am I right that the token lets
anyone read any data as if they were you? I'd be more comfortable if it were
cookie based (though that implies that your datanodes and your namenode are in
the same domain, which might not be workable), though I do see how GET is
simpler.
The web security part of me is also worried that this is liable to CSRF
(http://en.wikipedia.org/wiki/Cross-site_request_forgery) attacks. The key
there, I think, is to make sure that when the namenode is issuing tokens, it's
absolutely confident that it's issuing them to someone who is asking for them.
bq. URLEncoder.encode("/", "UTF-8")
Might be worthwhile to make this a constant. It's unlikely to change :)
bq. URL Creation
This isn't necessary this JIRA's to fix (nor is it introduced in this patch),
but the manual URL concatenation strikes me as a bit ugly. Most web frameworks
have utilities to add GET parameters and such and to build URLs for you. Not
sure if one is handy in our environment, but usually more readable than long
string concatenations.
bq. Tests
There aren't any :) Understandable in a preliminary patch.
> Allow browsing the filesystem over http using delegation tokens
> ---
>
> Key: HDFS-991
> URL: https://issues.apache.org/jira/browse/HDFS-991
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.22.0
>
> Attachments: h-991.patch
>
>
> Assuming the user authenticates to the NameNode in the browser, allow them to
> browse the file system by adding a delegation token the the url when it is
> redirected to a datanode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868722#action_12868722 ] Philip Zeyliger commented on HDFS-1150: --- Hi Jakob, This may be a naive question because I'm out of date on the HDFS security stuff. What mechanism is preventing me from starting a rogue datanode and participating in the cluster? Thanks, -- Philip > Verify datanodes' identities to clients in secure clusters > -- > > Key: HDFS-1150 > URL: https://issues.apache.org/jira/browse/HDFS-1150 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Affects Versions: 0.22.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: HDFS-1150-BF1-Y20.patch, > HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, > HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, > HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, > HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, > HDFS-1150-Y20S-Rough.txt > > > Currently we use block access tokens to allow datanodes to verify clients' > identities, however we don't have a way for clients to verify the > authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HDFS-3447) StandbyException should not be logged at ERROR level on server
[ https://issues.apache.org/jira/browse/HDFS-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282023#comment-13282023 ] Philip Zeyliger commented on HDFS-3447: --- In a failed over situation, even at 'info' level this will end up with one log message per client when there's a failover (which might be a reasonable steady state case, since there's no reason to prefer A over B), which seems like a lot. You might tone it down to DEBUG even. > StandbyException should not be logged at ERROR level on server > -- > > Key: HDFS-3447 > URL: https://issues.apache.org/jira/browse/HDFS-3447 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Affects Versions: 2.0.0-alpha >Reporter: Todd Lipcon >Priority: Minor > > Currently, the standby NN will log StandbyExceptions at ERROR level any time > a client tries to connect to it. So, if the second NN in an HA pair is > active, the first NN will spew a lot of these errors in the log, as each > client gets redirected to the proper NN. Instead, this should be at INFO > level, and should probably be logged in a less "scary" manner (eg "Received > READ request from client 1.2.3.4, but in Standby state. Redirecting client to > other NameNode.") -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-6982) nntop: top-like tool for name node users
[ https://issues.apache.org/jira/browse/HDFS-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14118786#comment-14118786 ] Philip Zeyliger commented on HDFS-6982: --- Looks neat! Since you're proposing this for inclusion in HDFS proper, I'd suggest that the implementation skips the auditing stuff and just works directly. Obviously, you would introduce a configuration property turning the feature on or off as desired, since people may have memory management concerns or performance concerns. It would also be useful to have a structured way to get the output for monitoring tools, which it sounds like you already have. Could you give some sample output for that mechanism? > nntop: top-like tool for name node users > - > > Key: HDFS-6982 > URL: https://issues.apache.org/jira/browse/HDFS-6982 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Maysam Yabandeh > Attachments: nntop-design-v1.pdf > > > In this jira we motivate the need for nntop, a tool that, similarly to what > top does in Linux, gives the list of top users of the HDFS name node and > gives insight about which users are sending majority of each traffic type to > the name node. This information turns out to be the most critical when the > name node is under pressure and the HDFS admin needs to know which user is > hammering the name node and with what kind of requests. Here we present the > design of nntop which has been in production at Twitter in the past 10 > months. nntop proved to have low cpu overhead (< 2% in a cluster of 4K > nodes), low memory footprint (less than a few MB), and quite efficient for > the write path (only two hash lookup for updating a metric). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7165) Separate block metrics for files with replication count 1
[
https://issues.apache.org/jira/browse/HDFS-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159566#comment-14159566
]
Philip Zeyliger commented on HDFS-7165:
---
Seems like we should also expose this in JMX. See
{{hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java}}.
> Separate block metrics for files with replication count 1
> -
>
> Key: HDFS-7165
> URL: https://issues.apache.org/jira/browse/HDFS-7165
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Andrew Wang
>Assignee: Zhe Zhang
> Attachments: HDFS-7165-20141003-v1.patch
>
>
> We see a lot of escalations because someone has written teragen output with a
> replication factor of 1, a DN goes down, and a bunch of missing blocks show
> up. These are normally false positives, since teragen output is disposable,
> and generally speaking, users should understand this is true for all repl=1
> files.
> It'd be nice to be able to separate out these repl=1 missing blocks from
> missing blocks with higher replication factors..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7165) Separate block metrics for files with replication count 1
[
https://issues.apache.org/jira/browse/HDFS-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167019#comment-14167019
]
Philip Zeyliger commented on HDFS-7165:
---
The code looks fine to me. Thanks!
I'm having trouble with the naming of {{getNumberOfMissingUnreplicatedBlocks}}
vs. {{getNumberOfMissingBlocks}}. Perhaps simply
{{getNumberOfMissingBlocksWithReplicationOne}}? I assume you're trying to say
that "unreplicated" means "is intended to not be replicated; had just that one
base copy," but the usage seems inconsistent. Of course, I'll defer to others
with more experience in this code base.
bq. + required uint64 missing_unreplicated_blocks = 7;
Does making this "required" in the protocol cause issues? It's obviously not
existed in previous versions, but it's desirable that new clients can discern
that this isn't set, and old clients continue to work.
> Separate block metrics for files with replication count 1
> -
>
> Key: HDFS-7165
> URL: https://issues.apache.org/jira/browse/HDFS-7165
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Andrew Wang
>Assignee: Zhe Zhang
> Attachments: HDFS-7165-20141003-v1.patch, HDFS-7165-20141009-v1.patch
>
>
> We see a lot of escalations because someone has written teragen output with a
> replication factor of 1, a DN goes down, and a bunch of missing blocks show
> up. These are normally false positives, since teragen output is disposable,
> and generally speaking, users should understand this is true for all repl=1
> files.
> It'd be nice to be able to separate out these repl=1 missing blocks from
> missing blocks with higher replication factors..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7283) Bump DataNode OOM log from WARN to ERROR
[
https://issues.apache.org/jira/browse/HDFS-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181865#comment-14181865
]
Philip Zeyliger commented on HDFS-7283:
---
Do we know how this behaves when you use an
{code}
-XX:OnOutOfMemoryError=/some/kill/command
{code}
Java option?
> Bump DataNode OOM log from WARN to ERROR
>
>
> Key: HDFS-7283
> URL: https://issues.apache.org/jira/browse/HDFS-7283
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Labels: supportability
>
> When the DataNode OOMs, it logs the following WARN message which should be
> bumped up to ERROR because DataNode OOM often leads to DN process abortion.
> {code}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is out of
> memory. Will retry in 30 seconds.
> 4751 java.lang.OutOfMemoryError: unable to create new native thread"
> {code}
> Thanks to Roland Teague for identifying this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7283) Bump DataNode OOM log from WARN to ERROR
[
https://issues.apache.org/jira/browse/HDFS-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181933#comment-14181933
]
Philip Zeyliger commented on HDFS-7283:
---
Changing the log message to ERROR sounds like a great idea. I've taken,
however, to running my datanodes so that they die completely on OutOfMemory
because depending on where the exception happens, some things can't recover.
> Bump DataNode OOM log from WARN to ERROR
>
>
> Key: HDFS-7283
> URL: https://issues.apache.org/jira/browse/HDFS-7283
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha
>Reporter: Stephen Chu
>Assignee: Stephen Chu
>Priority: Trivial
> Labels: supportability
>
> When the DataNode OOMs, it logs the following WARN message which should be
> bumped up to ERROR because DataNode OOM often leads to DN process abortion.
> {code}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is out of
> memory. Will retry in 30 seconds.
> 4751 java.lang.OutOfMemoryError: unable to create new native thread"
> {code}
> Thanks to Roland Teague for identifying this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
