[jira] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847137#comment-15847137 ] John Zhuge commented on HDFS-10860: --- Thanks [~xiaochen] for the review. bq. HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private Fixed bq. MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params? GOT NPE without the check for daemonlog request ServerSetup.md.vm: bq. NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)? `httpfs.sh` is a wrapper of `hdfs httpfs`, not `hdfs` which has many other sub-commands. bq. keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat... Fixed bq. It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'. Fixed bq. Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context. Fixed in section 'Start/Stop HttpFS' bq. In Deprecated Environment Variables, please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File' Variables are sorted by name but ok with your suggestion. bq. TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable. Added to {{TestStartStop}} bq. index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys Fixed :) bq. Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param. Will investigate > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847205#comment-15847205 ] John Zhuge commented on HDFS-10860: --- bq. MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params? No, I was wrong about NPE without the check for daemonlog request. Will investigate further, > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11350) Document the missing settings relevant to DataNode disk IO statistics
[ https://issues.apache.org/jira/browse/HDFS-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847207#comment-15847207 ] Arpit Agarwal commented on HDFS-11350: -- Hi [~linyiqun], I want to hold off documenting these settings for now. > Document the missing settings relevant to DataNode disk IO statistics > - > > Key: HDFS-11350 > URL: https://issues.apache.org/jira/browse/HDFS-11350 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Minor > Attachments: HDFS-11350.001.patch > > > HDFS-11299 and HDFS-11339 introduced some new setting relevant to profiling > hooks to expose latency statistics around DataNode disk IO. These settings > should be intended for users but they are not documented. Totally three > relevant settings: > 1.dfs.datanode.enable.fileio.profiling > 2.dfs.datanode.enable.fileio.fault.injection > 3.dfs.datanode.fileio.profiling.sampling.fraction > Actually, only the setting {{dfs.datanode.enable.fileio.fault.injection}} > don't need to be documented. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847211#comment-15847211 ] Uma Maheswara Rao G commented on HDFS-11243: {quote} ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java:81: final static int DNA_DROP_SPS_WORK_COMMAND = 13; {quote} Intentionally did not fix this checksyle warning because I declared the constants same as other constants in the file. So, I thought need not fix this. > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847222#comment-15847222 ] John Zhuge commented on HDFS-10860: --- Confirmed NPE without the check for daemonlog request, and for /jmx, /conf, and /logLevel. > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847225#comment-15847225 ] Andrew Wang commented on HDFS-10983: Hi Manoj, We reused the replication field so the in-memory representation would be compact. You're right though that this isn't necessary for the on-disk representation; {{replication}} is an optional field, and if we add another optional field for the EC policy, it won't be any extra space. I'm okay with doing this, though we should do it in another JIRA since this one is oriented toward OIV. Let's make sure to have the appropriate asserts in place to make sure both fields aren't set for the same INodeField. > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-11243: Resolution: Fixed Fix Version/s: HDFS-10285 Target Version/s: (was: HDFS-10285) Status: Resolved (was: Patch Available) > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-10285 > > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847248#comment-15847248 ] Rakesh R commented on HDFS-11243: - Thank you [~umamaheswararao]. Committed to branch HDFS-10285 > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-10285 > > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847307#comment-15847307 ] Manoj Govindassamy commented on HDFS-10983: --- Thanks [~andrew.wang]. Filed HDFS-11382 to track persistence of EC Policy ID in a new optional field. > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11382) Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage
Manoj Govindassamy created HDFS-11382: - Summary: Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage Key: HDFS-11382 URL: https://issues.apache.org/jira/browse/HDFS-11382 Project: Hadoop HDFS Issue Type: Task Components: hdfs Affects Versions: 3.0.0-alpha1 Reporter: Manoj Govindassamy Assignee: Manoj Govindassamy For Erasure Coded files, replication field in INodeFile message is re-used to store the EC Policy ID. *FSDirWriteFileOp#addFile* {noformat} private static INodesInPath addFile( FSDirectory fsd, INodesInPath existing, byte[] localName, PermissionStatus permissions, short replication, long preferredBlockSize, String clientName, String clientMachine) throws IOException { .. .. .. try { ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp. getErasureCodingPolicy(fsd.getFSNamesystem(), existing); if (ecPolicy != null) { replication = ecPolicy.getId(); <=== } final BlockType blockType = ecPolicy != null? BlockType.STRIPED : BlockType.CONTIGUOUS; INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions, modTime, modTime, replication, preferredBlockSize, blockType); newNode.setLocalName(localName); newNode.toUnderConstruction(clientName, clientMachine); newiip = fsd.addINode(existing, newNode, permissions.getPermission()); {noformat} With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly getting the EC ID from the replication field and then uses the right Policy to construct the blocks. *FSImageFormatPBINode#Loader#loadInodeFile* {noformat} ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ? ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) : null; {noformat} The original intention was to re-use the replication field so the in-memory representation would be compact. But, this isn't necessary for the on-disk representation. replication is an optional field, and if we add another optional field for the EC policy, it won't be any extra space. Also, we need to make sure to have the appropriate asserts in place to make sure both fields aren't set for the same INodeField. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11382) Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage
[ https://issues.apache.org/jira/browse/HDFS-11382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11382: --- Hadoop Flags: Incompatible change > Persist Erasure Coding Policy ID in a new optional field in INodeFile in > FSImage > > > Key: HDFS-11382 > URL: https://issues.apache.org/jira/browse/HDFS-11382 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > > For Erasure Coded files, replication field in INodeFile message is re-used > to store the EC Policy ID. > *FSDirWriteFileOp#addFile* > {noformat} > private static INodesInPath addFile( > FSDirectory fsd, INodesInPath existing, byte[] localName, > PermissionStatus permissions, short replication, long > preferredBlockSize, > String clientName, String clientMachine) > throws IOException { > .. .. .. > try { > ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp. > getErasureCodingPolicy(fsd.getFSNamesystem(), existing); > if (ecPolicy != null) { > replication = ecPolicy.getId(); <=== > } > final BlockType blockType = ecPolicy != null? > BlockType.STRIPED : BlockType.CONTIGUOUS; > INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions, > modTime, modTime, replication, preferredBlockSize, blockType); > newNode.setLocalName(localName); > newNode.toUnderConstruction(clientName, clientMachine); > newiip = fsd.addINode(existing, newNode, permissions.getPermission()); > {noformat} > With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly > getting the EC ID from the replication field and then uses the right Policy > to construct the blocks. > *FSImageFormatPBINode#Loader#loadInodeFile* > {noformat} > ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ? > ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) : > null; > {noformat} > The original intention was to re-use the replication field so the in-memory > representation would be compact. But, this isn't necessary for the on-disk > representation. replication is an optional field, and if we add another > optional field for the EC policy, it won't be any extra space. > Also, we need to make sure to have the appropriate asserts in place to make > sure both fields aren't set for the same INodeField. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10629) Federation Router
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847495#comment-15847495 ] Lei Guo commented on HDFS-10629: [~elgoiri] [~jakace], do we have rough idea about the overhead introduced via router? > Federation Router > - > > Key: HDFS-10629 > URL: https://issues.apache.org/jira/browse/HDFS-10629 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10629.000.patch, HDFS-10629.001.patch, > HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch, > HDFS-10629-HDFS-10467-004.patch, HDFS-10629-HDFS-10467-005.patch, > HDFS-10629-HDFS-10467-006.patch, HDFS-10629-HDFS-10467-007.patch > > > Component that routes calls from the clients to the right Namespace. It > implements {{ClientProtocol}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7859: -- Priority: Blocker (was: Major) Target Version/s: 3.0.0-alpha3 Marking ec must do's as blockers for alpha3 > Erasure Coding: Persist erasure coding policies in NameNode > --- > > Key: HDFS-7859 > URL: https://issues.apache.org/jira/browse/HDFS-7859 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Xinwei Qin >Priority: Blocker > Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do > Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, > HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch, > HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, > HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, > HDFS-7859-HDFS-7285.003.patch > > > In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we > persist EC schemas in NameNode centrally and reliably, so that EC zones can > reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-8112) Enforce authorization policy to protect administration operations for EC zone and schemas
[ https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8112: -- Priority: Blocker (was: Major) Marking ec must do's as blockers for alpha3 > Enforce authorization policy to protect administration operations for EC zone > and schemas > - > > Key: HDFS-8112 > URL: https://issues.apache.org/jira/browse/HDFS-8112 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Kai Zheng >Assignee: Rakesh R >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > > We should allow to enforce authorization policy to protect administration > operations for EC zone and schemas as such behaviors would impact too much > for a system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11314) Validate client-provided EC schema on the NameNode
[ https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11314: --- Priority: Blocker (was: Major) Marking ec must do's as blockers for alpha3 > Validate client-provided EC schema on the NameNode > -- > > Key: HDFS-11314 > URL: https://issues.apache.org/jira/browse/HDFS-11314 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > > Filing based on discussion in HDFS-8095. A user might specify a policy that > is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster > only has 10 nodes. The NN should only allow the client to choose from a > pre-approved list determined by the cluster administrator. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded
[ https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10971: --- Priority: Blocker (was: Major) Marking ec must do's as blockers for alpha3 > Distcp should not copy replication factor if source file is erasure coded > - > > Key: HDFS-10971 > URL: https://issues.apache.org/jira/browse/HDFS-10971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: distcp >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-10971.testcase.patch > > > The current erasure coding implementation uses replication factor field to > store erasure coding policy. > Distcp copies the source file's replication factor to the destination if > {{-pr}} is specified. However, if the source file is EC, the replication > factor (which is EC policy) should not be replicated to the destination file. > When a HdfsFileStatus is converted to FileStatus, the replication factor is > set to 0 if it's an EC file. > In fact, I will attach a test case that shows trying to replicate the > replication factor of an EC file results in an IOException: "Requested > replication factor of 0 is less than the required minimum of 1 for > /tmp/dst/dest2" -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-8430) Erasure coding: compute file checksum for striped files (stripe by stripe)
[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8430: -- Priority: Blocker (was: Major) Target Version/s: 3.0.0-alpha3 Component/s: erasure-coding Marking ec must do's as blockers for alpha3 > Erasure coding: compute file checksum for striped files (stripe by stripe) > -- > > Key: HDFS-8430 > URL: https://issues.apache.org/jira/browse/HDFS-8430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: HDFS-7285 >Reporter: Walter Su >Assignee: Kai Zheng >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-8430-poc1.patch > > > HADOOP-3981 introduces a distributed file checksum algorithm. It's designed > for replicated block. > {{DFSClient.getFileChecksum()}} need some updates, so it can work for striped > block group. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11152) Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors
[ https://issues.apache.org/jira/browse/HDFS-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11152: --- Priority: Blocker (was: Major) Component/s: erasure-coding Marking ec must do's as blockers for alpha3 > Start erasure coding policy ID number from 1 instead of 0 to void potential > unexpected errors > - > > Key: HDFS-11152 > URL: https://issues.apache.org/jira/browse/HDFS-11152 > Project: Hadoop HDFS > Issue Type: Task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: SammiChen >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11152-v1.patch, HDFS-11152-v2.patch > > > This task will change erasure coding policy ID number starting from 1 instead > of current 0, to avoid some potential unexpected errors in codes since 0 is > default value for integer variables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847657#comment-15847657 ] John Zhuge commented on HDFS-10860: --- bq. Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param. /conf, /jmx, /logLevel, and /stacks do report "Authentication required" when {{user.name}} is not specified. /logs do not require authentication. This does not seem right, will file another JIRA to follow up. > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-10860: -- Attachment: HDFS-10860.007.patch Patch 007 - Xiao’s comments - Test simple request in TestHttpFSServerWebServer#testStartStop TESTING DONE - TestHttpFSServerWebServer - HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode - Verify ServerSetup.html and static index.html > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch, HDFS-10860.007.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10629) Federation Router
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847663#comment-15847663 ] Hadoop QA commented on HDFS-10629: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 3s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 33s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 394 unchanged - 0 fixed = 396 total (was 394) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 22s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 96m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs | | | Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be atomic in org.apache.hadoop.hdfs.server.federation.router.ConnectionManager.getConnection(UserGroupInformation, String) At ConnectionManager.java:may not be atomic in org.apache.hadoop.hdfs.server.federation.router.ConnectionManager.getConnection(UserGroupInformation, String) At ConnectionManager.java:[line 151] | | | org.apache.hadoop.hdfs.server.federation.router.Router.initAndStartRouter(Configuration, boolean) invokes System.exit(...), which shuts down the entire virtual machine At Router.java:shuts down the entire virtual machine At Router.java:[line 123] | | Failed junit tests | hadoop.tools.TestHdfsConfigFields | | | hadoop.hdfs.TestRollingUpgrade | | | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-10629 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832545/HDFS-10629-HDFS-10467-007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 8d4de08371eb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:2
[jira] [Commented] (HDFS-10629) Federation Router
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847662#comment-15847662 ] Inigo Goiri commented on HDFS-10629: [~grey], I just uploaded a figure showing the latency of the version we tested 4 months ago. Executing listing requests against one Namenode we got a latency of a little less than 1 ms and a saturation point of around 40k requests per seconds. Adding a Router in front, the latency almost double but still under 2ms; the saturation point was a little worse too. When comparing to 4 Namenodes and 12 Router for 4 Namenodes, both latencies and saturation points are worse but much better than for a single subcluster. Note that this is the worst case for us, as this was one of the cheapest operations. When reading data, the overhead is negligible. In addition, we have done some optimizations but the order of magnitude should be the same. > Federation Router > - > > Key: HDFS-10629 > URL: https://issues.apache.org/jira/browse/HDFS-10629 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10629.000.patch, HDFS-10629.001.patch, > HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch, > HDFS-10629-HDFS-10467-004.patch, HDFS-10629-HDFS-10467-005.patch, > HDFS-10629-HDFS-10467-006.patch, HDFS-10629-HDFS-10467-007.patch, > routerlatency.png > > > Component that routes calls from the clients to the right Namespace. It > implements {{ClientProtocol}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10629) Federation Router
[ https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-10629: --- Attachment: routerlatency.png Latency of a Router with a Namenode compared to a single Namenode and latency of 12 Routers going to 4 Namenodes compared to clients going to 4 Namenodes directly. > Federation Router > - > > Key: HDFS-10629 > URL: https://issues.apache.org/jira/browse/HDFS-10629 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Inigo Goiri >Assignee: Jason Kace > Attachments: HDFS-10629.000.patch, HDFS-10629.001.patch, > HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch, > HDFS-10629-HDFS-10467-004.patch, HDFS-10629-HDFS-10467-005.patch, > HDFS-10629-HDFS-10467-006.patch, HDFS-10629-HDFS-10467-007.patch, > routerlatency.png > > > Component that routes calls from the clients to the right Namespace. It > implements {{ClientProtocol}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11100) Recursively deleting file protected by sticky bit should fail
[ https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11100: -- Status: In Progress (was: Patch Available) > Recursively deleting file protected by sticky bit should fail > - > > Key: HDFS-11100 > URL: https://issues.apache.org/jira/browse/HDFS-11100 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Critical > Labels: permissions > Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch, > HDFS-11100.003.patch, hdfs_cmds > > > Recursively deleting a directory that contains files or directories protected > by sticky bit should fail but it doesn't in HDFS. In the case below, > {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive > deleting {{/tmp/test/sticky_dir}} should fail. > {noformat} > + hdfs dfs -ls -R /tmp/test > drwxrwxrwt - jzhuge supergroup 0 2016-11-03 18:08 > /tmp/test/sticky_dir > -rwxrwxrwx 1 jzhuge supergroup 0 2016-11-03 18:08 > /tmp/test/sticky_dir/f2 > + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2 > rm: Permission denied by sticky bit: user=hadoop, > path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx, > parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt > + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir > Deleted /tmp/test/sticky_dir > {noformat} > Centos 6.4 behavior: > {noformat} > $ ls -lR /tmp/test > /tmp/test: > total 4 > drwxrwxrwt 2 systest systest 4096 Nov 3 18:36 sbit > /tmp/test/sbit: > total 0 > -rw-rw-rw- 1 systest systest 0 Nov 2 13:45 f2 > $ sudo -u mapred rm -fr /tmp/test/sbit > rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted > $ chmod -t /tmp/test/sbit > $ sudo -u mapred rm -fr /tmp/test/sbit > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847693#comment-15847693 ] Hadoop QA commented on HDFS-11370: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m 37s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 93m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HDFS-11370 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12850160/HDFS-11370.5.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ea6d512c4399 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 258991d | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/18298/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/18298/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Optimize NamenodeFsck#getReplicaInfo > > > Key: HDFS-11370 > URL: https://issues.apache.org/jira/browse/HDFS-11370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch, > HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-
[jira] [Commented] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847699#comment-15847699 ] Uma Maheswara Rao G commented on HDFS-11243: Thank you [~rakeshr] for the review and commit > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-10285 > > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11210) Enhance key rolling to be atomic
[ https://issues.apache.org/jira/browse/HDFS-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-11210: - Attachment: HDFS-11210.02.patch Thanks for the review Andrew, really appreciate the quality review of yours. :) Good call to {{invalidate}} cache, don't know what I was thinking when naming it... bq. High-level, is KeyProvider#clearCache intended for use by end-users? Since {{invalidateCache}} is added to KMS, and the keyprovider API, end-users can definitely use it. I try to hide this detail as much as possible, but agree {{hadoop key}} command addition would be handy, and no harm. So added this and a test in patch 2. All comments addressed I think. Also added a new test in {{TestKMS}} with mock, to fail-before-pass-after the invalidation. The rollover draining test is race by nature, so IMHO the separate mock testing would be clearer. > Enhance key rolling to be atomic > > > Key: HDFS-11210 > URL: https://issues.apache.org/jira/browse/HDFS-11210 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption, kms >Affects Versions: 2.6.5 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-11210.01.patch, HDFS-11210.02.patch > > > To support re-encrypting EDEK, we need to make sure after a key is rolled, no > old version EDEKs are used anymore. This includes various caches when > generating EDEK. > This is not true currently, simply because no such requirements / necessities > before. > This includes > - Client Provider(s), and corresponding cache(s). > When LoadBalancingKMSCP is used, we need to clear all KMSCPs. > - KMS server instance(s), and corresponding cache(s) > When KMS HA is configured with multiple KMS instances, only 1 will receive > the {{rollNewVersion}} request, we need to make sure other instances are > rolled too. > - The Client instance inside NN(s), and corresponding cache(s) > When {{hadoop key roll}} is succeeded, the client provider inside NN should > be drained too. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11210) Enhance key rolling to be guarantee new KeyVersion is returned from generateEncryptedKeys after a key version is rolled
[ https://issues.apache.org/jira/browse/HDFS-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-11210: - Summary: Enhance key rolling to be guarantee new KeyVersion is returned from generateEncryptedKeys after a key version is rolled (was: Enhance key rolling to be atomic) > Enhance key rolling to be guarantee new KeyVersion is returned from > generateEncryptedKeys after a key version is rolled > --- > > Key: HDFS-11210 > URL: https://issues.apache.org/jira/browse/HDFS-11210 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption, kms >Affects Versions: 2.6.5 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-11210.01.patch, HDFS-11210.02.patch > > > To support re-encrypting EDEK, we need to make sure after a key is rolled, no > old version EDEKs are used anymore. This includes various caches when > generating EDEK. > This is not true currently, simply because no such requirements / necessities > before. > This includes > - Client Provider(s), and corresponding cache(s). > When LoadBalancingKMSCP is used, we need to clear all KMSCPs. > - KMS server instance(s), and corresponding cache(s) > When KMS HA is configured with multiple KMS instances, only 1 will receive > the {{rollNewVersion}} request, we need to make sure other instances are > rolled too. > - The Client instance inside NN(s), and corresponding cache(s) > When {{hadoop key roll}} is succeeded, the client provider inside NN should > be drained too. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[ https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847736#comment-15847736 ] Chris Douglas commented on HDFS-6984: - bq. serialization that omits fields for efficiency isn't supportable, and is why I'd prefer a PathHandle. With this constraint, we either expand the API surface- adding functions that accept PathHandle instances, rather than FileStatus- or all fields are required in FileStatus objects. I see your point w.r.t. confused semantics for partial FileStatus objects. In completeness, we could change the return type to {{Optional}} for FileStatus fields, but that's a nonstarter from a compatibility perspective. We can pick this up in HDFS-7878. bq. I don't feel that strongly about removing Writable, but the nowritable patch is simple and to the point Well... sure. Deleting code is direct. :) I don't mind moving the protobuf serialization to a utility library. Would that move this forward? bq. Cross-serialization doesn't have an immediate usecase. HDFS-7878 IMO needs a serializable PathHandle, not a full FileStatus. The goal was not to make cross-serialization a requirement, but to make future efficiency possible. > In Hadoop 3, make FileStatus serialize itself via protobuf > -- > > Key: HDFS-6984 > URL: https://issues.apache.org/jira/browse/HDFS-6984 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Colin P. McCabe >Assignee: Colin P. McCabe > Labels: BB2015-05-TBR > Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch, > HDFS-6984.003.patch, HDFS-6984.nowritable.patch > > > FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this > to serialize it and send it over the wire. But in Hadoop 2 and later, we > have the protobuf {{HdfsFileStatusProto}} which serves to serialize this > information. The protobuf form is preferable, since it allows us to add new > fields in a backwards-compatible way. Another issue is that already a lot of > subclasses of FileStatus don't override the Writable methods of the > superclass, breaking the interface contract that read(status.write) should be > equal to the original status. > In Hadoop 3, we should just make FileStatus serialize itself via protobuf so > that we don't have to deal with these issues. It's probably too late to do > this in Hadoop 2, since user code may be relying on the existing FileStatus > serialization there. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11377) Balancer hung due to "No mover threads available"
[ https://issues.apache.org/jira/browse/HDFS-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847789#comment-15847789 ] Manoj Govindassamy commented on HDFS-11377: --- Thanks [~zhaoyunjiong]. In {{Dispatcher#dispatchBlockMoves}}, even though the return failure code for {{waitForMoveCompletion}} is not handled, the method returns total bytes moved and no bytes moved case is handled by the caller. LGTM. +1 (unbinding). > Balancer hung due to "No mover threads available" > - > > Key: HDFS-11377 > URL: https://issues.apache.org/jira/browse/HDFS-11377 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: HDFS-11377.001.patch > > > When running balancer on large cluster which have more than 3000 Datanodes, > it might be hung due to "No mover threads available". > The stack trace shows it waiting forever like below. > {code} > "main" #1 prio=5 os_prio=0 tid=0x7ff6cc014800 nid=0x6b2c waiting on > condition [0x7ff6d1bad000] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1043) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1017) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:981) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:611) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:663) > at > org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:776) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:905) > {code} > In the log, there are lots of WARN about "No mover threads available". > {quote} > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_13700554102_1112815018180 with size=268435456 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through > 10.115.67.137:50010 > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_4009558842_1103118359883 with size=268435456 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through > 10.115.67.137:50010 > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_13881956058_1112996460026 with size=133509566 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through 10.115.67.36:50010 > {quote} > What happened here is, when there are no mover threads available, > DDatanode.isPendingQEmpty() will return false, so Balancer hung. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847826#comment-15847826 ] Manoj Govindassamy commented on HDFS-11370: --- Thanks [~tasanuma0829]. bq. I think it is the same risk as that of BlockUnderConstructionFeature#getExpectedStorageLocations() at least. {{BlockUnderConstructionFeature#getExpectedStorageLocations()}} and few other public methods in the same class work on the {{replicas}} array without proper synchronization but they never expose the cursor on the array to the caller. With {{getExpectedStorageLocationsIterator()}} we will be exposing a public iterator with the cursor exposed, but not thread safe. As a general practice, its advisable not to introduce any new non thread safe public iterators. One workaround here i can think of is, since the replicas is a very small array, we can have a copy of the replicas in the iterator at the time of construction and iterate over the same. Your thoughts please ? > Optimize NamenodeFsck#getReplicaInfo > > > Key: HDFS-11370 > URL: https://issues.apache.org/jira/browse/HDFS-11370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch, > HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch > > > We can optimize the logic of calculating the number of storages in > {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847827#comment-15847827 ] Xiao Chen commented on HDFS-10860: -- Thanks [~jzhuge]. As discussed, there're really 2 issues with HttpServer2 security: # authentication should be needed to access everything # based on 1, what level of authorization should be there for each of these servlet. Looking at HttpServer2 code, it seems there's a {{hasAdministratorAccess}} method. So possibly we just need some configurations set. Will play with it a bit. > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch, HDFS-10860.007.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847847#comment-15847847 ] Jing Zhao commented on HDFS-11370: -- Thanks for the discussion, [~manojg] and [~tasanuma0829]. I agree it's better to achieve thread safe for the new {{getExpectedStorageLocationsIterator}}. But currently almost all block related classes, from Block to BlockInfo to BlockUnderConstructionFeature, does not provide thread-safe guarantee and depend on external mechanism such as the FSNamesystem lock for protection. So I do not think we need to address this issue here, but maybe we can add a java doc for {{getExpectedStorageLocationsIterator}} explaining that the method is not thread-safe by itself. > Optimize NamenodeFsck#getReplicaInfo > > > Key: HDFS-11370 > URL: https://issues.apache.org/jira/browse/HDFS-11370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch, > HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch > > > We can optimize the logic of calculating the number of storages in > {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847850#comment-15847850 ] Manoj Govindassamy commented on HDFS-11370: --- bq. maybe we can add a java doc for {{getExpectedStorageLocationsIterator}} explaining that the method is not thread-safe by itself. Sounds good to me. Thanks [~jingzhao]. > Optimize NamenodeFsck#getReplicaInfo > > > Key: HDFS-11370 > URL: https://issues.apache.org/jira/browse/HDFS-11370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch, > HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch > > > We can optimize the logic of calculating the number of storages in > {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847851#comment-15847851 ] Hadoop QA commented on HDFS-10860: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | | {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 2s{color} | {color:blue} Shelldocs was not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 59s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 18s{color} | {color:green} The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-assemblies {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-assemblies in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 3s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m 20s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 18s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}158m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad
[jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847858#comment-15847858 ] Andrew Wang commented on HDFS-7859: --- I re-read through the history of this JIRA, and it seems like we've debated a couple times whether it's useful to persist this information, with the current status of the patch being to only persist user-added policies. I think this is not very useful as is, and potentially dangerous. We let the user specify any ECPolicy they want, without much field validation. This means the user could specify an ID/name that we already use, or an ID/name we might want to hardcode later. Even with validation, this makes upgrade difficult. Given that we haven't finished the pluggable EC policy work, we also don't know what fields might be required to fully specify an EC policy. This patch does let the user configure different parameters for Reed Solomon, but we already provide what we think are a good set of hardcoded policies to choose from. IMO where some persistence would be useful is for HDFS-11314. We'd like to restrict the set of EC policies that can be used on a cluster, since fault tolerance depends on the # of nodes and racks. This would be limiting from the set of hardcoded policies though, rather than adding new policies. [~drankye], [~zhz], thoughts on this? > Erasure Coding: Persist erasure coding policies in NameNode > --- > > Key: HDFS-7859 > URL: https://issues.apache.org/jira/browse/HDFS-7859 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Xinwei Qin >Priority: Blocker > Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do > Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, > HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch, > HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, > HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, > HDFS-7859-HDFS-7285.003.patch > > > In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we > persist EC schemas in NameNode centrally and reliably, so that EC zones can > reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4025) QJM: Sychronize past log segments to JNs that missed them
[ https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDFS-4025: - Attachment: HDFS-4025.009.patch Patch v09 has some changes. * Checkstyle corrections * Minor optimizations * LastSyncedTxId has been dropped. It's possible that a checkpoint is done during sync interval and some segments are purged. To ensure correct lastSyncedTxId, we would need the checkpointing information from the NameNode. We can skip this doing this for now and always sync from the first transaction id. This improvement can be added later on. Thank you [~jingzhao] for guiding me through with this. > QJM: Sychronize past log segments to JNs that missed them > - > > Key: HDFS-4025 > URL: https://issues.apache.org/jira/browse/HDFS-4025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Hanisha Koneru > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: HDFS-4025.000.patch, HDFS-4025.001.patch, > HDFS-4025.002.patch, HDFS-4025.003.patch, HDFS-4025.004.patch, > HDFS-4025.005.patch, HDFS-4025.006.patch, HDFS-4025.007.patch, > HDFS-4025.008.patch, HDFS-4025.009.patch > > > Currently, if a JournalManager crashes and misses some segment of logs, and > then comes back, it will be re-added as a valid part of the quorum on the > next log roll. However, it will not have a complete history of log segments > (i.e any individual JN may have gaps in its transaction history). This > mirrors the behavior of the NameNode when there are multiple local > directories specified. > However, it would be better if a background thread noticed these gaps and > "filled them in" by grabbing the segments from other JournalNodes. This > increases the resilience of the system when JournalNodes get reformatted or > otherwise lose their local disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
Misha Dmitriev created HDFS-11383: - Summary: String duplication in org.apache.hadoop.fs.BlockLocation Key: HDFS-11383 URL: https://issues.apache.org/jira/browse/HDFS-11383 Project: Hadoop HDFS Issue Type: Improvement Reporter: Misha Dmitriev I am working on Hive performance, investigating the problem of high memory pressure when (a) a table consists of a high number (thousands) of partitions and (b) multiple queries run against it concurrently. It turns out that a lot of memory is wasted due to data duplication. One source of duplicate strings is class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds, topologyPaths, hosts, names, may collectively use up to 6% of memory in my benchmark, causing (together with other problematic classes) a huge memory spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are wasted due to duplication. I think we need to add calls to String.intern() in the BlockLocation constructor, like: {code} this.hosts = internStringsInArray(hosts); ... private void internStringsInArray(String[] sar) { for (int i = 0; i < sar.length; i++) { sar[i] = sar[i].intern(); } } {code} String.intern() performs very well starting from JDK 7. I've found some articles explaining the progress that was made by the HotSpot JVM developers in this area, verified that with benchmarks myself, and finally added quite a bit of interning to one of the Cloudera products without any issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4025) QJM: Sychronize past log segments to JNs that missed them
[ https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847912#comment-15847912 ] Hadoop QA commented on HDFS-4025: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 551 unchanged - 0 fixed = 552 total (was 551) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 91m 43s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HDFS-4025 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12850353/HDFS-4025.009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux b420f9c6046f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 3e06475 | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/18301/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/18301/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/18301/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > QJM: Sychronize past log segments to JNs that missed them > - > > Key: HDFS-4025 > URL: https://issues.apache.org/jira/browse/HDFS-4025 >
[jira] [Updated] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-11370: Attachment: HDFS-11370.6.patch Thanks for your thoughtful comments, [~manojg] and [~jingzhao]! I definitely agree with you. I uploaded a new patch including the java doc. > Optimize NamenodeFsck#getReplicaInfo > > > Key: HDFS-11370 > URL: https://issues.apache.org/jira/browse/HDFS-11370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Minor > Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch, > HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch, HDFS-11370.6.patch > > > We can optimize the logic of calculating the number of storages in > {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847993#comment-15847993 ] Manoj Govindassamy commented on HDFS-10983: --- [~andrew.wang], [~jojochuang], Here are the proposals. Please let me know your thoughts on the below. 1. OIV HTTP server does expose a read-only WebHDFS API which can be queried to print all file details. 1.a: Users can also get JSON formatted FileStatuses via HTTP REST API, which can very well be extended. Here is the proposal for {noformat} curl -i http://127.0.0.1:5978/webhdfs/v1/ec?op=getfilestatus {"FileStatus": {"owner":"manoj","replication":0,"hdfs.erasurecoding.policy":"XOR-2-1-64k", "length":0,"permission":"755","type":"DIRECTORY", "blockSize":0,"pathSuffix":"","modificationTime":1485921930732,"childrenNum":2,"accessTime":0,"group":"supergroup","fileId":16406} } curl -i http://127.0.0.1:5978/webhdfs/v1/ec/file.txt?op=getfilestatus {"FileStatus": {"owner":"manoj","replication":3,"blockType":"STRIPED", "length":0,"permission":"644","type":"FILE","blockSize":134217728, "pathSuffix":"","modificationTime":1485921930729,"childrenNum":0,"accessTime":1485921930710,"group":"supergroup","fileId":16407} } {noformat} 1.b: But,. when it is queried over shell, webhdfs returns only {{FileStatus}} as return type which doesn't carry any EC related details. So, I am not sure if we can make the following one to print extra details on EC file/dir. {noformat} hdfs dfs -ls webhdfs://127.0.0.1:5978/ {noformat} 2. {{OIV XML processor}} already has support for EC. Please review the output below. {noformat} 1 2 16406 3 DIRECTORY 4 ec 5 1485918336816 6 manoj:supergroup:0755 7 8 9 SYSTEM 10 hdfs.erasurecoding.policy <=== 11 XOR-2-1-64k 12 13 14 -1 15 -1 16 17 18 16407 19 FILE 20 EmptyECFile.txt 21 3 22 1485918336813 23 1485918336796 24 134217728 25 manoj:supergroup:0644 26 0 27 28 STRIPED<=== 29 30 {noformat} 2. {{OIV Delimited processor}} doesn't have support for EC. Here is the proposal for the new Header ("BlockType") and value ("CONTIGUOUS"/"STRIPED"). {noformat} PathReplication ModificationTimeAccessTime PreferredBlockSize BlockType BlocksCount FileSizeNSQUOTA DSQUOTA Permission UserNameGroupName / 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 9223372036854775807 -1 drwxr-xr-x manoj supergroup /dir0 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 -1 -1 drwxr-xr-x manoj supergroup /dir0/file0 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /dir0/file1 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /dir0/file2 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /dir0/file3 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /emptydir 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 -1 -1 drwxr-xr-x manoj supergroup /ec 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 -1 -1 drwxr-xr-x manoj supergroup /ec/EmptyECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728 STRIPED 0 0 0 0 -rw-r--r-- manoj supergroup /ec/SmallECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728 STRIPED 1 0 0 0 -rw-r--r-- manoj supergroup {noformat} > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For
[jira] [Comment Edited] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847993#comment-15847993 ] Manoj Govindassamy edited comment on HDFS-10983 at 2/1/17 4:11 AM: --- [~andrew.wang], [~jojochuang], Here are the proposals. Please let me know your thoughts on the below. 1. OIV HTTP server does expose a read-only WebHDFS API which can be queried to print all file details. 1.a: Users can also get JSON formatted FileStatuses via HTTP REST API, which can very well be extended. Here is the proposal for REST API output. Added "hdfs.erasurecoding.policy" for Directory and "blockType" for File. {noformat} curl -i http://127.0.0.1:5978/webhdfs/v1/ec?op=getfilestatus {"FileStatus": {"owner":"manoj","replication":0,"hdfs.erasurecoding.policy":"XOR-2-1-64k", "length":0,"permission":"755","type":"DIRECTORY", "blockSize":0,"pathSuffix":"","modificationTime":1485921930732, "childrenNum":2,"accessTime":0,"group":"supergroup","fileId":16406} } curl -i http://127.0.0.1:5978/webhdfs/v1/ec/file.txt?op=getfilestatus {"FileStatus": {"owner":"manoj","replication":3,"blockType":"STRIPED", "length":0,"permission":"644","type":"FILE","blockSize":134217728, "pathSuffix":"","modificationTime":1485921930729,"childrenNum":0, "accessTime":1485921930710,"group":"supergroup","fileId":16407} } {noformat} 1.b: But,. when it is queried over shell, webhdfs returns only {{FileStatus}} as return type which doesn't carry any EC related details. So, I am not sure if we can make the following one to print extra details on EC file/dir. {noformat} hdfs dfs -ls webhdfs://127.0.0.1:5978/ {noformat} 2. {{OIV XML processor}} already has support for EC. Please review the output below. {noformat} 1 2 16406 3 DIRECTORY 4 ec 5 1485918336816 6 manoj:supergroup:0755 7 8 9 SYSTEM 10 hdfs.erasurecoding.policy <=== 11 XOR-2-1-64k 12 13 14 -1 15 -1 16 17 18 16407 19 FILE 20 EmptyECFile.txt 21 3 22 1485918336813 23 1485918336796 24 134217728 25 manoj:supergroup:0644 26 0 27 28 STRIPED<=== 29 30 {noformat} 2. {{OIV Delimited processor}} doesn't have support for EC. Here is the proposal for the new Header ("BlockType") and value ("CONTIGUOUS"/"STRIPED"). {noformat} PathReplication ModificationTimeAccessTime PreferredBlockSize BlockType BlocksCount FileSizeNSQUOTA DSQUOTA Permission UserNameGroupName / 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 9223372036854775807 -1 drwxr-xr-x manoj supergroup /dir0 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 -1 -1 drwxr-xr-x manoj supergroup /dir0/file0 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /dir0/file1 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /dir0/file2 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /dir0/file3 3 2017-01-31 18:572017-01-31 18:57134217728 CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup /emptydir 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 -1 -1 drwxr-xr-x manoj supergroup /ec 0 2017-01-31 18:571969-12-31 16:000 NA 0 0 -1 -1 drwxr-xr-x manoj supergroup /ec/EmptyECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728 STRIPED 0 0 0 0 -rw-r--r-- manoj supergroup /ec/SmallECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728 STRIPED 1 0 0 0 -rw-r--r-- manoj supergroup {noformat} was (Author: manojg): [~andrew.wang], [~jojochuang], Here are the proposals. Please let me know your thoughts on the below. 1. OIV HTTP server does expose a read-only WebHDFS API which can be queried to print all file details. 1.a: Users can also get JSON formatted FileStatuses via HTTP REST API, which can very well be extended. Here is the proposal for {noformat} curl -i http://127.0.0.1:5978/webhdfs/v1/ec?op=getfilestatus {"FileStatus": {"owner":"manoj","replication":0,"hdfs.erasurecoding.policy":"XOR-2-1-64k", "length":0,"permission":"755","type":"DIRECTORY", "blockSize":0,"pathSuffix":"","modificationTime":1485921930732,"childrenNum":2,"accessTime":0,"group":"supergroup","fileId":16406} } curl -i http://127.0.0.1:5978/webhdfs/v1/ec/file.txt?op=getfilestatus {"FileStatus": {"owner":"manoj","replication":3,"bl
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[ https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848049#comment-15848049 ] Hadoop QA commented on HDFS-11370: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 7s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}120m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HDFS-11370 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12850367/HDFS-11370.6.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux fec09a0d67a1 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 3e06475 | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/18302/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/18302/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/18302/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Optimize NamenodeFsck#getReplicaInfo > > > Key: HDFS-11370 > URL: https://issues.apache.org/jira/browse/HDFS-11370 > Project: Hadoop HDFS > Issue Type: Improvement
[jira] [Updated] (HDFS-11335) Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf
[ https://issues.apache.org/jira/browse/HDFS-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-11335: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3 2.9.0 Status: Resolved (was: Patch Available) LGTM as well. +1. Committed to trunk and branch-2 Thanks [~manojg] for working on this patch and [~linyiqun] for the reviews! > Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage > from DNConf > -- > > Key: HDFS-11335 > URL: https://issues.apache.org/jira/browse/HDFS-11335 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11335.01.patch, HDFS-11335.02.patch, > HDFS-11335.03.patch > > > DataNode configuration {{DNConf}} reading in > {{HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY}} > (_dfs.client.slow.io.warning.threshold.ms_) is redundant as this threshold > limit is needed only for {{DfsClientConf}}. Better to remove the unwanted > usage in DNConf. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11340) DataNode reconfigure for disks doesn't remove the failed volumes
[ https://issues.apache.org/jira/browse/HDFS-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848074#comment-15848074 ] Lei (Eddy) Xu commented on HDFS-11340: -- Thanks for the explanation, [~manojg]. It makes sense. > DataNode reconfigure for disks doesn't remove the failed volumes > > > Key: HDFS-11340 > URL: https://issues.apache.org/jira/browse/HDFS-11340 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Attachments: HDFS-11340.01.patch > > > Say a DataNode (uuid:xyz) has disks D1 and D2. When D1 turns bad, JMX query > on FSDatasetState-xyz for "NumFailedVolumes" attr rightly shows the failed > volume count as 1 and the "FailedStorageLocations" attr has the failed > storage location as "D1". > It is possible to add or remove disks to this DataNode by running > {{reconfigure}} command. Let the failed disk D1 be removed from the conf and > the new conf has only one good disk D2. Upon running the reconfigure command > for this DataNode with this new disk conf, the expectation is DataNode would > no more have "NumFailedVolumes" or "FailedStorageLocations". But, even after > removing the failed disk from the conf and a successful reconfigure, DataNode > continues to show the "NumFailedVolumes" as 1 and "FailedStorageLocations" as > "D1" and it never gets reset. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11377) Balancer hung due to "No mover threads available"
[ https://issues.apache.org/jira/browse/HDFS-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848076#comment-15848076 ] Yiqun Lin commented on HDFS-11377: -- Thanks [~zhaoyunjiong] for reporting this and thanks [~manojg] for the review. It's a nice finding. The patch also looks good to me. One minor comment, can you also remove the unused variable {{MAX_NO_PENDING_MOVE_ITERATIONS}} in {{Dispatcher}}? This hardcoded value has been replaced by option {{-idleiterations}}. +1 once addressed. > Balancer hung due to "No mover threads available" > - > > Key: HDFS-11377 > URL: https://issues.apache.org/jira/browse/HDFS-11377 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: HDFS-11377.001.patch > > > When running balancer on large cluster which have more than 3000 Datanodes, > it might be hung due to "No mover threads available". > The stack trace shows it waiting forever like below. > {code} > "main" #1 prio=5 os_prio=0 tid=0x7ff6cc014800 nid=0x6b2c waiting on > condition [0x7ff6d1bad000] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1043) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1017) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:981) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:611) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:663) > at > org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:776) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:905) > {code} > In the log, there are lots of WARN about "No mover threads available". > {quote} > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_13700554102_1112815018180 with size=268435456 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through > 10.115.67.137:50010 > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_4009558842_1103118359883 with size=268435456 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through > 10.115.67.137:50010 > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_13881956058_1112996460026 with size=133509566 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through 10.115.67.36:50010 > {quote} > What happened here is, when there are no mover threads available, > DDatanode.isPendingQEmpty() will return false, so Balancer hung. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11335) Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf
[ https://issues.apache.org/jira/browse/HDFS-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848080#comment-15848080 ] Hudson commented on HDFS-11335: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11193 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11193/]) HDFS-11335. Remove (lei: rev bec9b7aa1dd3ed95b8783597135f8d90b3cc8dcd) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java > Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage > from DNConf > -- > > Key: HDFS-11335 > URL: https://issues.apache.org/jira/browse/HDFS-11335 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HDFS-11335.01.patch, HDFS-11335.02.patch, > HDFS-11335.03.patch > > > DataNode configuration {{DNConf}} reading in > {{HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY}} > (_dfs.client.slow.io.warning.threshold.ms_) is redundant as this threshold > limit is needed only for {{DfsClientConf}}. Better to remove the unwanted > usage in DNConf. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11377) Balancer hung due to no available mover threads
[ https://issues.apache.org/jira/browse/HDFS-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11377: - Target Version/s: 2.9.0, 3.0.0-alpha3 Component/s: balancer & mover Summary: Balancer hung due to no available mover threads (was: Balancer hung due to "No mover threads available") > Balancer hung due to no available mover threads > --- > > Key: HDFS-11377 > URL: https://issues.apache.org/jira/browse/HDFS-11377 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: HDFS-11377.001.patch > > > When running balancer on large cluster which have more than 3000 Datanodes, > it might be hung due to "No mover threads available". > The stack trace shows it waiting forever like below. > {code} > "main" #1 prio=5 os_prio=0 tid=0x7ff6cc014800 nid=0x6b2c waiting on > condition [0x7ff6d1bad000] >java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1043) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1017) > at > org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:981) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:611) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:663) > at > org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:776) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:905) > {code} > In the log, there are lots of WARN about "No mover threads available". > {quote} > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_13700554102_1112815018180 with size=268435456 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through > 10.115.67.137:50010 > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_4009558842_1103118359883 with size=268435456 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through > 10.115.67.137:50010 > 2017-01-26 15:36:40,085 WARN > org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads > available: skip moving blk_13881956058_1112996460026 with size=133509566 from > 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through 10.115.67.36:50010 > {quote} > What happened here is, when there are no mover threads available, > DDatanode.isPendingQEmpty() will return false, so Balancer hung. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[ https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848089#comment-15848089 ] John Zhuge commented on HDFS-10860: --- {{HttpServer2#isInstrumentationAccessAllowed}} requires the following conditions to return false: * hadoop.security.instrumentation.requires.admin set to true * hadoop.security.authorization set to true * remote user is not set or {{HttpServer2#setACL}} is called with an ACL that disallows the remote user Looks like we need another configuration property to set administrators for HttpFS server instrumentation access, e.g., {{hadoop.httpfs.http.administrators}}, similar to {{dfs.cluster.administrators}}. Will need to fix KMS to call {{HttpServer2#setACL}}. Also gotta make sure all users of {{HttpServer2}} call {{setACL}}. > Switch HttpFS from Tomcat to Jetty > -- > > Key: HDFS-10860 > URL: https://issues.apache.org/jira/browse/HDFS-10860 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch, > HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch, > HDFS-10860.006.patch, HDFS-10860.007.patch > > > The Tomcat 6 we are using will reach EOL at the end of 2017. While there are > other good options, I would propose switching to {{Jetty 9}} for the > following reasons: > * Easier migration. Both Tomcat and Jetty are based on {{Servlet > Containers}}, so we don't have to change client code that much. It would > require more work to switch to {{JAX-RS}}. > * Well established. > * Good performance and scalability. > Other alternatives: > * Jersey + Grizzly > * Tomcat 8 > Your opinions will be greatly appreciated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org