[jira] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847137#comment-15847137
]
John Zhuge commented on HDFS-10860:
---
Thanks [~xiaochen] for the review.
bq. HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private
Fixed
bq. MDCFilter:java: why do we need to check getPathInfo() being null here? Do
we need to check other params?
GOT NPE without the check for daemonlog request
ServerSetup.md.vm:
bq. NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of
`hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which
is the script being called)?
`httpfs.sh` is a wrapper of `hdfs httpfs`, not `hdfs` which has many other
sub-commands.
bq. keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat...
Fixed
bq. It seems the paragraph under the keytool line doesn't need the change.
Since the example is given 'As the `httpfs` Unix user'.
Fixed
bq. Let's also have a note in the doc that tomcat is deprecated, together with
its old .sh configurations. We can point to the Deprecated Environment
Variables section you added later, to give more context.
Fixed in section 'Start/Stop HttpFS'
bq. In Deprecated Environment Variables, please move HTTPFS_TEMP to the first
line to be consistent with the config code, and to better group the
'Configuration File'
Variables are sorted by name but ok with your suggestion.
bq. TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a
simple request after startup, to make sure the server is reachable.
Added to {{TestStartStop}}
bq. index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list
all keys
Fixed :)
bq. Sorry should have asked this in the KMS jira. The new index.html page for
loglevel/jmx/conf/logs looks nice and handy. But how does the security work
here? Have you tested this in a kerberized environment? I tried locally in a
pseudo-authenticated setup, it seems I can read/set everything even without the
user.name= param.
Will investigate
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847205#comment-15847205
]
John Zhuge commented on HDFS-10860:
---
bq. MDCFilter:java: why do we need to check getPathInfo() being null here? Do
we need to check other params?
No, I was wrong about NPE without the check for daemonlog request. Will
investigate further,
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11350) Document the missing settings relevant to DataNode disk IO statistics
[
https://issues.apache.org/jira/browse/HDFS-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847207#comment-15847207
]
Arpit Agarwal commented on HDFS-11350:
--
Hi [~linyiqun], I want to hold off documenting these settings for now.
> Document the missing settings relevant to DataNode disk IO statistics
> -
>
> Key: HDFS-11350
> URL: https://issues.apache.org/jira/browse/HDFS-11350
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11350.001.patch
>
>
> HDFS-11299 and HDFS-11339 introduced some new setting relevant to profiling
> hooks to expose latency statistics around DataNode disk IO. These settings
> should be intended for users but they are not documented. Totally three
> relevant settings:
> 1.dfs.datanode.enable.fileio.profiling
> 2.dfs.datanode.enable.fileio.fault.injection
> 3.dfs.datanode.fileio.profiling.sampling.fraction
> Actually, only the setting {{dfs.datanode.enable.fileio.fault.injection}}
> don't need to be documented.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[
https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847211#comment-15847211
]
Uma Maheswara Rao G commented on HDFS-11243:
{quote}
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java:81:
final static int DNA_DROP_SPS_WORK_COMMAND = 13;
{quote}
Intentionally did not fix this checksyle warning because I declared the
constants same as other constants in the file. So, I thought need not fix this.
> [SPS]: Add a protocol command from NN to DN for dropping the SPS work and
> queues
> -
>
> Key: HDFS-11243
> URL: https://issues.apache.org/jira/browse/HDFS-11243
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-11243-HDFS-10285-00.patch,
> HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch
>
>
> This JIRA is for adding a protocol command from Namenode to Datanode for
> dropping SPS work. and Also for dropping in progress queues.
> Use case is: when admin deactivated SPS at NN, then internally NN should
> issue a command to DNs for dropping in progress queues as well. This command
> can be packed via heartbeat.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847222#comment-15847222
]
John Zhuge commented on HDFS-10860:
---
Confirmed NPE without the check for daemonlog request, and for /jmx, /conf, and
/logLevel.
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10983) OIV tool should make an EC file explicit
[
https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847225#comment-15847225
]
Andrew Wang commented on HDFS-10983:
Hi Manoj,
We reused the replication field so the in-memory representation would be
compact. You're right though that this isn't necessary for the on-disk
representation; {{replication}} is an optional field, and if we add another
optional field for the EC policy, it won't be any extra space.
I'm okay with doing this, though we should do it in another JIRA since this one
is oriented toward OIV. Let's make sure to have the appropriate asserts in
place to make sure both fields aren't set for the same INodeField.
> OIV tool should make an EC file explicit
>
>
> Key: HDFS-10983
> URL: https://issues.apache.org/jira/browse/HDFS-10983
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-nice-to-have
>
> The OIV tool's webhdfs interface does not print if a file is striped or not.
> Also, it prints the file's EC policy ID as replication factor, which is
> inconsistent to the output of a typical webhdfs call to the cluster, which
> always shows replication factor of 0 for EC files.
> Not just webhdfs, but delimiter output does not print if a file is stripped
> or not either.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-11243: Resolution: Fixed Fix Version/s: HDFS-10285 Target Version/s: (was: HDFS-10285) Status: Resolved (was: Patch Available) > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-10285 > > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847248#comment-15847248 ] Rakesh R commented on HDFS-11243: - Thank you [~umamaheswararao]. Committed to branch HDFS-10285 > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-10285 > > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847307#comment-15847307 ] Manoj Govindassamy commented on HDFS-10983: --- Thanks [~andrew.wang]. Filed HDFS-11382 to track persistence of EC Policy ID in a new optional field. > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11382) Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage
Manoj Govindassamy created HDFS-11382:
-
Summary: Persist Erasure Coding Policy ID in a new optional field
in INodeFile in FSImage
Key: HDFS-11382
URL: https://issues.apache.org/jira/browse/HDFS-11382
Project: Hadoop HDFS
Issue Type: Task
Components: hdfs
Affects Versions: 3.0.0-alpha1
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
For Erasure Coded files, replication field in INodeFile message is re-used to
store the EC Policy ID.
*FSDirWriteFileOp#addFile*
{noformat}
private static INodesInPath addFile(
FSDirectory fsd, INodesInPath existing, byte[] localName,
PermissionStatus permissions, short replication, long preferredBlockSize,
String clientName, String clientMachine)
throws IOException {
.. .. ..
try {
ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
if (ecPolicy != null) {
replication = ecPolicy.getId(); <===
}
final BlockType blockType = ecPolicy != null?
BlockType.STRIPED : BlockType.CONTIGUOUS;
INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
modTime, modTime, replication, preferredBlockSize, blockType);
newNode.setLocalName(localName);
newNode.toUnderConstruction(clientName, clientMachine);
newiip = fsd.addINode(existing, newNode, permissions.getPermission());
{noformat}
With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly
getting the EC ID from the replication field and then uses the right Policy to
construct the blocks.
*FSImageFormatPBINode#Loader#loadInodeFile*
{noformat}
ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :
null;
{noformat}
The original intention was to re-use the replication field so the in-memory
representation would be compact. But, this isn't necessary for the on-disk
representation. replication is an optional field, and if we add another
optional field for the EC policy, it won't be any extra space.
Also, we need to make sure to have the appropriate asserts in place to make
sure both fields aren't set for the same INodeField.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11382) Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage
[
https://issues.apache.org/jira/browse/HDFS-11382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11382:
---
Hadoop Flags: Incompatible change
> Persist Erasure Coding Policy ID in a new optional field in INodeFile in
> FSImage
>
>
> Key: HDFS-11382
> URL: https://issues.apache.org/jira/browse/HDFS-11382
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>
> For Erasure Coded files, replication field in INodeFile message is re-used
> to store the EC Policy ID.
> *FSDirWriteFileOp#addFile*
> {noformat}
> private static INodesInPath addFile(
> FSDirectory fsd, INodesInPath existing, byte[] localName,
> PermissionStatus permissions, short replication, long
> preferredBlockSize,
> String clientName, String clientMachine)
> throws IOException {
> .. .. ..
> try {
> ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
> getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
> if (ecPolicy != null) {
> replication = ecPolicy.getId(); <===
> }
> final BlockType blockType = ecPolicy != null?
> BlockType.STRIPED : BlockType.CONTIGUOUS;
> INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
> modTime, modTime, replication, preferredBlockSize, blockType);
> newNode.setLocalName(localName);
> newNode.toUnderConstruction(clientName, clientMachine);
> newiip = fsd.addINode(existing, newNode, permissions.getPermission());
> {noformat}
> With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly
> getting the EC ID from the replication field and then uses the right Policy
> to construct the blocks.
> *FSImageFormatPBINode#Loader#loadInodeFile*
> {noformat}
> ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
> ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :
> null;
> {noformat}
> The original intention was to re-use the replication field so the in-memory
> representation would be compact. But, this isn't necessary for the on-disk
> representation. replication is an optional field, and if we add another
> optional field for the EC policy, it won't be any extra space.
> Also, we need to make sure to have the appropriate asserts in place to make
> sure both fields aren't set for the same INodeField.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10629) Federation Router
[
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847495#comment-15847495
]
Lei Guo commented on HDFS-10629:
[~elgoiri] [~jakace], do we have rough idea about the overhead introduced via
router?
> Federation Router
> -
>
> Key: HDFS-10629
> URL: https://issues.apache.org/jira/browse/HDFS-10629
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10629.000.patch, HDFS-10629.001.patch,
> HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch,
> HDFS-10629-HDFS-10467-004.patch, HDFS-10629-HDFS-10467-005.patch,
> HDFS-10629-HDFS-10467-006.patch, HDFS-10629-HDFS-10467-007.patch
>
>
> Component that routes calls from the clients to the right Namespace. It
> implements {{ClientProtocol}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7859: -- Priority: Blocker (was: Major) Target Version/s: 3.0.0-alpha3 Marking ec must do's as blockers for alpha3 > Erasure Coding: Persist erasure coding policies in NameNode > --- > > Key: HDFS-7859 > URL: https://issues.apache.org/jira/browse/HDFS-7859 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Xinwei Qin >Priority: Blocker > Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do > Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, > HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch, > HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, > HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, > HDFS-7859-HDFS-7285.003.patch > > > In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we > persist EC schemas in NameNode centrally and reliably, so that EC zones can > reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-8112) Enforce authorization policy to protect administration operations for EC zone and schemas
[ https://issues.apache.org/jira/browse/HDFS-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8112: -- Priority: Blocker (was: Major) Marking ec must do's as blockers for alpha3 > Enforce authorization policy to protect administration operations for EC zone > and schemas > - > > Key: HDFS-8112 > URL: https://issues.apache.org/jira/browse/HDFS-8112 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Kai Zheng >Assignee: Rakesh R >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > > We should allow to enforce authorization policy to protect administration > operations for EC zone and schemas as such behaviors would impact too much > for a system. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11314) Validate client-provided EC schema on the NameNode
[ https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11314: --- Priority: Blocker (was: Major) Marking ec must do's as blockers for alpha3 > Validate client-provided EC schema on the NameNode > -- > > Key: HDFS-11314 > URL: https://issues.apache.org/jira/browse/HDFS-11314 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > > Filing based on discussion in HDFS-8095. A user might specify a policy that > is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster > only has 10 nodes. The NN should only allow the client to choose from a > pre-approved list determined by the cluster administrator. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-10971) Distcp should not copy replication factor if source file is erasure coded
[
https://issues.apache.org/jira/browse/HDFS-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-10971:
---
Priority: Blocker (was: Major)
Marking ec must do's as blockers for alpha3
> Distcp should not copy replication factor if source file is erasure coded
> -
>
> Key: HDFS-10971
> URL: https://issues.apache.org/jira/browse/HDFS-10971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: distcp
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-10971.testcase.patch
>
>
> The current erasure coding implementation uses replication factor field to
> store erasure coding policy.
> Distcp copies the source file's replication factor to the destination if
> {{-pr}} is specified. However, if the source file is EC, the replication
> factor (which is EC policy) should not be replicated to the destination file.
> When a HdfsFileStatus is converted to FileStatus, the replication factor is
> set to 0 if it's an EC file.
> In fact, I will attach a test case that shows trying to replicate the
> replication factor of an EC file results in an IOException: "Requested
> replication factor of 0 is less than the required minimum of 1 for
> /tmp/dst/dest2"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-8430) Erasure coding: compute file checksum for striped files (stripe by stripe)
[
https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-8430:
--
Priority: Blocker (was: Major)
Target Version/s: 3.0.0-alpha3
Component/s: erasure-coding
Marking ec must do's as blockers for alpha3
> Erasure coding: compute file checksum for striped files (stripe by stripe)
> --
>
> Key: HDFS-8430
> URL: https://issues.apache.org/jira/browse/HDFS-8430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Kai Zheng
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-8430-poc1.patch
>
>
> HADOOP-3981 introduces a distributed file checksum algorithm. It's designed
> for replicated block.
> {{DFSClient.getFileChecksum()}} need some updates, so it can work for striped
> block group.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] (HDFS-11152) Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors
[ https://issues.apache.org/jira/browse/HDFS-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11152: --- Priority: Blocker (was: Major) Component/s: erasure-coding Marking ec must do's as blockers for alpha3 > Start erasure coding policy ID number from 1 instead of 0 to void potential > unexpected errors > - > > Key: HDFS-11152 > URL: https://issues.apache.org/jira/browse/HDFS-11152 > Project: Hadoop HDFS > Issue Type: Task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: SammiChen >Assignee: SammiChen >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11152-v1.patch, HDFS-11152-v2.patch > > > This task will change erasure coding policy ID number starting from 1 instead > of current 0, to avoid some potential unexpected errors in codes since 0 is > default value for integer variables. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847657#comment-15847657
]
John Zhuge commented on HDFS-10860:
---
bq. Sorry should have asked this in the KMS jira. The new index.html page for
loglevel/jmx/conf/logs looks nice and handy. But how does the security work
here? Have you tested this in a kerberized environment? I tried locally in a
pseudo-authenticated setup, it seems I can read/set everything even without the
user.name= param.
/conf, /jmx, /logLevel, and /stacks do report "Authentication required" when
{{user.name}} is not specified.
/logs do not require authentication. This does not seem right, will file
another JIRA to follow up.
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10860:
--
Attachment: HDFS-10860.007.patch
Patch 007
- Xiao’s comments
- Test simple request in TestHttpFSServerWebServer#testStartStop
TESTING DONE
- TestHttpFSServerWebServer
- HttpFS Bats regression tests
https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode
- Verify ServerSetup.html and static index.html
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch, HDFS-10860.007.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10629) Federation Router
[
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847663#comment-15847663
]
Hadoop QA commented on HDFS-10629:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
3s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 394 unchanged - 0 fixed = 396 total (was 394) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be
atomic in
org.apache.hadoop.hdfs.server.federation.router.ConnectionManager.getConnection(UserGroupInformation,
String) At ConnectionManager.java:may not be atomic in
org.apache.hadoop.hdfs.server.federation.router.ConnectionManager.getConnection(UserGroupInformation,
String) At ConnectionManager.java:[line 151] |
| |
org.apache.hadoop.hdfs.server.federation.router.Router.initAndStartRouter(Configuration,
boolean) invokes System.exit(...), which shuts down the entire virtual machine
At Router.java:shuts down the entire virtual machine At Router.java:[line
123] |
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10629 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832545/HDFS-10629-HDFS-10467-007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8d4de08371eb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:2
[jira] [Commented] (HDFS-10629) Federation Router
[
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847662#comment-15847662
]
Inigo Goiri commented on HDFS-10629:
[~grey], I just uploaded a figure showing the latency of the version we tested
4 months ago.
Executing listing requests against one Namenode we got a latency of a little
less than 1 ms and a saturation point of around 40k requests per seconds.
Adding a Router in front, the latency almost double but still under 2ms; the
saturation point was a little worse too.
When comparing to 4 Namenodes and 12 Router for 4 Namenodes, both latencies and
saturation points are worse but much better than for a single subcluster.
Note that this is the worst case for us, as this was one of the cheapest
operations.
When reading data, the overhead is negligible.
In addition, we have done some optimizations but the order of magnitude should
be the same.
> Federation Router
> -
>
> Key: HDFS-10629
> URL: https://issues.apache.org/jira/browse/HDFS-10629
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10629.000.patch, HDFS-10629.001.patch,
> HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch,
> HDFS-10629-HDFS-10467-004.patch, HDFS-10629-HDFS-10467-005.patch,
> HDFS-10629-HDFS-10467-006.patch, HDFS-10629-HDFS-10467-007.patch,
> routerlatency.png
>
>
> Component that routes calls from the clients to the right Namespace. It
> implements {{ClientProtocol}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10629) Federation Router
[
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Inigo Goiri updated HDFS-10629:
---
Attachment: routerlatency.png
Latency of a Router with a Namenode compared to a single Namenode and latency
of 12 Routers going to 4 Namenodes compared to clients going to 4 Namenodes
directly.
> Federation Router
> -
>
> Key: HDFS-10629
> URL: https://issues.apache.org/jira/browse/HDFS-10629
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10629.000.patch, HDFS-10629.001.patch,
> HDFS-10629-HDFS-10467-002.patch, HDFS-10629-HDFS-10467-003.patch,
> HDFS-10629-HDFS-10467-004.patch, HDFS-10629-HDFS-10467-005.patch,
> HDFS-10629-HDFS-10467-006.patch, HDFS-10629-HDFS-10467-007.patch,
> routerlatency.png
>
>
> Component that routes calls from the clients to the right Namespace. It
> implements {{ClientProtocol}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11100) Recursively deleting file protected by sticky bit should fail
[
https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-11100:
--
Status: In Progress (was: Patch Available)
> Recursively deleting file protected by sticky bit should fail
> -
>
> Key: HDFS-11100
> URL: https://issues.apache.org/jira/browse/HDFS-11100
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
> Labels: permissions
> Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch,
> HDFS-11100.003.patch, hdfs_cmds
>
>
> Recursively deleting a directory that contains files or directories protected
> by sticky bit should fail but it doesn't in HDFS. In the case below,
> {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive
> deleting {{/tmp/test/sticky_dir}} should fail.
> {noformat}
> + hdfs dfs -ls -R /tmp/test
> drwxrwxrwt - jzhuge supergroup 0 2016-11-03 18:08
> /tmp/test/sticky_dir
> -rwxrwxrwx 1 jzhuge supergroup 0 2016-11-03 18:08
> /tmp/test/sticky_dir/f2
> + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2
> rm: Permission denied by sticky bit: user=hadoop,
> path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx,
> parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt
> + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir
> Deleted /tmp/test/sticky_dir
> {noformat}
> Centos 6.4 behavior:
> {noformat}
> $ ls -lR /tmp/test
> /tmp/test:
> total 4
> drwxrwxrwt 2 systest systest 4096 Nov 3 18:36 sbit
> /tmp/test/sbit:
> total 0
> -rw-rw-rw- 1 systest systest 0 Nov 2 13:45 f2
> $ sudo -u mapred rm -fr /tmp/test/sbit
> rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted
> $ chmod -t /tmp/test/sbit
> $ sudo -u mapred rm -fr /tmp/test/sbit
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847693#comment-15847693
]
Hadoop QA commented on HDFS-11370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m
37s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850160/HDFS-11370.5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ea6d512c4399 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 258991d |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18298/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18298/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize NamenodeFsck#getReplicaInfo
>
>
> Key: HDFS-11370
> URL: https://issues.apache.org/jira/browse/HDFS-11370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch,
> HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-
[jira] [Commented] (HDFS-11243) [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues
[ https://issues.apache.org/jira/browse/HDFS-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847699#comment-15847699 ] Uma Maheswara Rao G commented on HDFS-11243: Thank you [~rakeshr] for the review and commit > [SPS]: Add a protocol command from NN to DN for dropping the SPS work and > queues > - > > Key: HDFS-11243 > URL: https://issues.apache.org/jira/browse/HDFS-11243 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS-10285 > > Attachments: HDFS-11243-HDFS-10285-00.patch, > HDFS-11243-HDFS-10285-01.patch, HDFS-11243-HDFS-10285-02.patch > > > This JIRA is for adding a protocol command from Namenode to Datanode for > dropping SPS work. and Also for dropping in progress queues. > Use case is: when admin deactivated SPS at NN, then internally NN should > issue a command to DNs for dropping in progress queues as well. This command > can be packed via heartbeat. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11210) Enhance key rolling to be atomic
[
https://issues.apache.org/jira/browse/HDFS-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11210:
-
Attachment: HDFS-11210.02.patch
Thanks for the review Andrew, really appreciate the quality review of yours. :)
Good call to {{invalidate}} cache, don't know what I was thinking when naming
it...
bq. High-level, is KeyProvider#clearCache intended for use by end-users?
Since {{invalidateCache}} is added to KMS, and the keyprovider API, end-users
can definitely use it. I try to hide this detail as much as possible, but agree
{{hadoop key}} command addition would be handy, and no harm. So added this and
a test in patch 2.
All comments addressed I think. Also added a new test in {{TestKMS}} with mock,
to fail-before-pass-after the invalidation. The rollover draining test is race
by nature, so IMHO the separate mock testing would be clearer.
> Enhance key rolling to be atomic
>
>
> Key: HDFS-11210
> URL: https://issues.apache.org/jira/browse/HDFS-11210
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: encryption, kms
>Affects Versions: 2.6.5
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11210.01.patch, HDFS-11210.02.patch
>
>
> To support re-encrypting EDEK, we need to make sure after a key is rolled, no
> old version EDEKs are used anymore. This includes various caches when
> generating EDEK.
> This is not true currently, simply because no such requirements / necessities
> before.
> This includes
> - Client Provider(s), and corresponding cache(s).
> When LoadBalancingKMSCP is used, we need to clear all KMSCPs.
> - KMS server instance(s), and corresponding cache(s)
> When KMS HA is configured with multiple KMS instances, only 1 will receive
> the {{rollNewVersion}} request, we need to make sure other instances are
> rolled too.
> - The Client instance inside NN(s), and corresponding cache(s)
> When {{hadoop key roll}} is succeeded, the client provider inside NN should
> be drained too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11210) Enhance key rolling to be guarantee new KeyVersion is returned from generateEncryptedKeys after a key version is rolled
[
https://issues.apache.org/jira/browse/HDFS-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-11210:
-
Summary: Enhance key rolling to be guarantee new KeyVersion is returned
from generateEncryptedKeys after a key version is rolled (was: Enhance key
rolling to be atomic)
> Enhance key rolling to be guarantee new KeyVersion is returned from
> generateEncryptedKeys after a key version is rolled
> ---
>
> Key: HDFS-11210
> URL: https://issues.apache.org/jira/browse/HDFS-11210
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: encryption, kms
>Affects Versions: 2.6.5
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-11210.01.patch, HDFS-11210.02.patch
>
>
> To support re-encrypting EDEK, we need to make sure after a key is rolled, no
> old version EDEKs are used anymore. This includes various caches when
> generating EDEK.
> This is not true currently, simply because no such requirements / necessities
> before.
> This includes
> - Client Provider(s), and corresponding cache(s).
> When LoadBalancingKMSCP is used, we need to clear all KMSCPs.
> - KMS server instance(s), and corresponding cache(s)
> When KMS HA is configured with multiple KMS instances, only 1 will receive
> the {{rollNewVersion}} request, we need to make sure other instances are
> rolled too.
> - The Client instance inside NN(s), and corresponding cache(s)
> When {{hadoop key roll}} is succeeded, the client provider inside NN should
> be drained too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847736#comment-15847736
]
Chris Douglas commented on HDFS-6984:
-
bq. serialization that omits fields for efficiency isn't supportable, and is
why I'd prefer a PathHandle.
With this constraint, we either expand the API surface- adding functions that
accept PathHandle instances, rather than FileStatus- or all fields are required
in FileStatus objects. I see your point w.r.t. confused semantics for partial
FileStatus objects. In completeness, we could change the return type to
{{Optional}} for FileStatus fields, but that's a nonstarter from a
compatibility perspective.
We can pick this up in HDFS-7878.
bq. I don't feel that strongly about removing Writable, but the nowritable
patch is simple and to the point
Well... sure. Deleting code is direct. :) I don't mind moving the protobuf
serialization to a utility library. Would that move this forward?
bq. Cross-serialization doesn't have an immediate usecase. HDFS-7878 IMO needs
a serializable PathHandle, not a full FileStatus.
The goal was not to make cross-serialization a requirement, but to make future
efficiency possible.
> In Hadoop 3, make FileStatus serialize itself via protobuf
> --
>
> Key: HDFS-6984
> URL: https://issues.apache.org/jira/browse/HDFS-6984
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Labels: BB2015-05-TBR
> Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch,
> HDFS-6984.003.patch, HDFS-6984.nowritable.patch
>
>
> FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this
> to serialize it and send it over the wire. But in Hadoop 2 and later, we
> have the protobuf {{HdfsFileStatusProto}} which serves to serialize this
> information. The protobuf form is preferable, since it allows us to add new
> fields in a backwards-compatible way. Another issue is that already a lot of
> subclasses of FileStatus don't override the Writable methods of the
> superclass, breaking the interface contract that read(status.write) should be
> equal to the original status.
> In Hadoop 3, we should just make FileStatus serialize itself via protobuf so
> that we don't have to deal with these issues. It's probably too late to do
> this in Hadoop 2, since user code may be relying on the existing FileStatus
> serialization there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11377) Balancer hung due to "No mover threads available"
[
https://issues.apache.org/jira/browse/HDFS-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847789#comment-15847789
]
Manoj Govindassamy commented on HDFS-11377:
---
Thanks [~zhaoyunjiong]. In {{Dispatcher#dispatchBlockMoves}}, even though the
return failure code for {{waitForMoveCompletion}} is not handled, the method
returns total bytes moved and no bytes moved case is handled by the caller.
LGTM. +1 (unbinding).
> Balancer hung due to "No mover threads available"
> -
>
> Key: HDFS-11377
> URL: https://issues.apache.org/jira/browse/HDFS-11377
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-11377.001.patch
>
>
> When running balancer on large cluster which have more than 3000 Datanodes,
> it might be hung due to "No mover threads available".
> The stack trace shows it waiting forever like below.
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7ff6cc014800 nid=0x6b2c waiting on
> condition [0x7ff6d1bad000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1043)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1017)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:981)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:611)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:663)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:776)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:905)
> {code}
> In the log, there are lots of WARN about "No mover threads available".
> {quote}
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_13700554102_1112815018180 with size=268435456 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through
> 10.115.67.137:50010
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_4009558842_1103118359883 with size=268435456 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through
> 10.115.67.137:50010
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_13881956058_1112996460026 with size=133509566 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through 10.115.67.36:50010
> {quote}
> What happened here is, when there are no mover threads available,
> DDatanode.isPendingQEmpty() will return false, so Balancer hung.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847826#comment-15847826
]
Manoj Govindassamy commented on HDFS-11370:
---
Thanks [~tasanuma0829].
bq. I think it is the same risk as that of
BlockUnderConstructionFeature#getExpectedStorageLocations() at least.
{{BlockUnderConstructionFeature#getExpectedStorageLocations()}} and few other
public methods in the same class work on the {{replicas}} array without proper
synchronization but they never expose the cursor on the array to the caller.
With {{getExpectedStorageLocationsIterator()}} we will be exposing a public
iterator with the cursor exposed, but not thread safe. As a general practice,
its advisable not to introduce any new non thread safe public iterators. One
workaround here i can think of is, since the replicas is a very small array, we
can have a copy of the replicas in the iterator at the time of construction and
iterate over the same. Your thoughts please ?
> Optimize NamenodeFsck#getReplicaInfo
>
>
> Key: HDFS-11370
> URL: https://issues.apache.org/jira/browse/HDFS-11370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch,
> HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch
>
>
> We can optimize the logic of calculating the number of storages in
> {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847827#comment-15847827
]
Xiao Chen commented on HDFS-10860:
--
Thanks [~jzhuge].
As discussed, there're really 2 issues with HttpServer2 security:
# authentication should be needed to access everything
# based on 1, what level of authorization should be there for each of these
servlet.
Looking at HttpServer2 code, it seems there's a {{hasAdministratorAccess}}
method. So possibly we just need some configurations set. Will play with it a
bit.
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch, HDFS-10860.007.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847847#comment-15847847
]
Jing Zhao commented on HDFS-11370:
--
Thanks for the discussion, [~manojg] and [~tasanuma0829]. I agree it's better
to achieve thread safe for the new {{getExpectedStorageLocationsIterator}}. But
currently almost all block related classes, from Block to BlockInfo to
BlockUnderConstructionFeature, does not provide thread-safe guarantee and
depend on external mechanism such as the FSNamesystem lock for protection. So I
do not think we need to address this issue here, but maybe we can add a java
doc for {{getExpectedStorageLocationsIterator}} explaining that the method is
not thread-safe by itself.
> Optimize NamenodeFsck#getReplicaInfo
>
>
> Key: HDFS-11370
> URL: https://issues.apache.org/jira/browse/HDFS-11370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch,
> HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch
>
>
> We can optimize the logic of calculating the number of storages in
> {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847850#comment-15847850
]
Manoj Govindassamy commented on HDFS-11370:
---
bq. maybe we can add a java doc for {{getExpectedStorageLocationsIterator}}
explaining that the method is not thread-safe by itself.
Sounds good to me. Thanks [~jingzhao].
> Optimize NamenodeFsck#getReplicaInfo
>
>
> Key: HDFS-11370
> URL: https://issues.apache.org/jira/browse/HDFS-11370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch,
> HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch
>
>
> We can optimize the logic of calculating the number of storages in
> {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847851#comment-15847851
]
Hadoop QA commented on HDFS-10860:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
2s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
18s{color} | {color:green} The patch generated 0 new + 564 unchanged - 8 fixed
= 564 total (was 572) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} hadoop-assemblies in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m
20s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
18s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad
[jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
[ https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847858#comment-15847858 ] Andrew Wang commented on HDFS-7859: --- I re-read through the history of this JIRA, and it seems like we've debated a couple times whether it's useful to persist this information, with the current status of the patch being to only persist user-added policies. I think this is not very useful as is, and potentially dangerous. We let the user specify any ECPolicy they want, without much field validation. This means the user could specify an ID/name that we already use, or an ID/name we might want to hardcode later. Even with validation, this makes upgrade difficult. Given that we haven't finished the pluggable EC policy work, we also don't know what fields might be required to fully specify an EC policy. This patch does let the user configure different parameters for Reed Solomon, but we already provide what we think are a good set of hardcoded policies to choose from. IMO where some persistence would be useful is for HDFS-11314. We'd like to restrict the set of EC policies that can be used on a cluster, since fault tolerance depends on the # of nodes and racks. This would be limiting from the set of hardcoded policies though, rather than adding new policies. [~drankye], [~zhz], thoughts on this? > Erasure Coding: Persist erasure coding policies in NameNode > --- > > Key: HDFS-7859 > URL: https://issues.apache.org/jira/browse/HDFS-7859 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Xinwei Qin >Priority: Blocker > Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do > Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch, > HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch, > HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch, > HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch, > HDFS-7859-HDFS-7285.003.patch > > > In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we > persist EC schemas in NameNode centrally and reliably, so that EC zones can > reference them by name efficiently. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4025) QJM: Sychronize past log segments to JNs that missed them
[ https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDFS-4025: - Attachment: HDFS-4025.009.patch Patch v09 has some changes. * Checkstyle corrections * Minor optimizations * LastSyncedTxId has been dropped. It's possible that a checkpoint is done during sync interval and some segments are purged. To ensure correct lastSyncedTxId, we would need the checkpointing information from the NameNode. We can skip this doing this for now and always sync from the first transaction id. This improvement can be added later on. Thank you [~jingzhao] for guiding me through with this. > QJM: Sychronize past log segments to JNs that missed them > - > > Key: HDFS-4025 > URL: https://issues.apache.org/jira/browse/HDFS-4025 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Hanisha Koneru > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: HDFS-4025.000.patch, HDFS-4025.001.patch, > HDFS-4025.002.patch, HDFS-4025.003.patch, HDFS-4025.004.patch, > HDFS-4025.005.patch, HDFS-4025.006.patch, HDFS-4025.007.patch, > HDFS-4025.008.patch, HDFS-4025.009.patch > > > Currently, if a JournalManager crashes and misses some segment of logs, and > then comes back, it will be re-added as a valid part of the quorum on the > next log roll. However, it will not have a complete history of log segments > (i.e any individual JN may have gaps in its transaction history). This > mirrors the behavior of the NameNode when there are multiple local > directories specified. > However, it would be better if a background thread noticed these gaps and > "filled them in" by grabbing the segments from other JournalNodes. This > increases the resilience of the system when JournalNodes get reformatted or > otherwise lose their local disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11383) String duplication in org.apache.hadoop.fs.BlockLocation
Misha Dmitriev created HDFS-11383:
-
Summary: String duplication in org.apache.hadoop.fs.BlockLocation
Key: HDFS-11383
URL: https://issues.apache.org/jira/browse/HDFS-11383
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Misha Dmitriev
I am working on Hive performance, investigating the problem of high memory
pressure when (a) a table consists of a high number (thousands) of partitions
and (b) multiple queries run against it concurrently. It turns out that a lot
of memory is wasted due to data duplication. One source of duplicate strings is
class org.apache.hadoop.fs.BlockLocation. Its fields such as storageIds,
topologyPaths, hosts, names, may collectively use up to 6% of memory in my
benchmark, causing (together with other problematic classes) a huge memory
spike. Of these 6% of memory taken by BlockLocation strings, more than 5% are
wasted due to duplication.
I think we need to add calls to String.intern() in the BlockLocation
constructor, like:
{code}
this.hosts = internStringsInArray(hosts);
...
private void internStringsInArray(String[] sar) {
for (int i = 0; i < sar.length; i++) {
sar[i] = sar[i].intern();
}
}
{code}
String.intern() performs very well starting from JDK 7. I've found some
articles explaining the progress that was made by the HotSpot JVM developers in
this area, verified that with benchmarks myself, and finally added quite a bit
of interning to one of the Cloudera products without any issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4025) QJM: Sychronize past log segments to JNs that missed them
[
https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847912#comment-15847912
]
Hadoop QA commented on HDFS-4025:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 551 unchanged - 0 fixed = 552 total (was 551) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m
35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-4025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850353/HDFS-4025.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux b420f9c6046f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3e06475 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18301/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18301/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18301/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> QJM: Sychronize past log segments to JNs that missed them
> -
>
> Key: HDFS-4025
> URL: https://issues.apache.org/jira/browse/HDFS-4025
>
[jira] [Updated] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-11370:
Attachment: HDFS-11370.6.patch
Thanks for your thoughtful comments, [~manojg] and [~jingzhao]! I definitely
agree with you. I uploaded a new patch including the java doc.
> Optimize NamenodeFsck#getReplicaInfo
>
>
> Key: HDFS-11370
> URL: https://issues.apache.org/jira/browse/HDFS-11370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-11370.1.patch, HDFS-11370.2.patch,
> HDFS-11370.3.patch, HDFS-11370.4.patch, HDFS-11370.5.patch, HDFS-11370.6.patch
>
>
> We can optimize the logic of calculating the number of storages in
> {{NamenodeFsck#getReplicaInfo}}. This is a follow-on task of HDFS-11124.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10983) OIV tool should make an EC file explicit
[
https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847993#comment-15847993
]
Manoj Govindassamy commented on HDFS-10983:
---
[~andrew.wang], [~jojochuang],
Here are the proposals. Please let me know your thoughts on the below.
1. OIV HTTP server does expose a read-only WebHDFS API which can be queried to
print all file details.
1.a: Users can also get JSON formatted FileStatuses via HTTP REST API, which
can very well be extended. Here is the proposal for
{noformat}
curl -i http://127.0.0.1:5978/webhdfs/v1/ec?op=getfilestatus
{"FileStatus":
{"owner":"manoj","replication":0,"hdfs.erasurecoding.policy":"XOR-2-1-64k",
"length":0,"permission":"755","type":"DIRECTORY",
"blockSize":0,"pathSuffix":"","modificationTime":1485921930732,"childrenNum":2,"accessTime":0,"group":"supergroup","fileId":16406}
}
curl -i http://127.0.0.1:5978/webhdfs/v1/ec/file.txt?op=getfilestatus
{"FileStatus":
{"owner":"manoj","replication":3,"blockType":"STRIPED",
"length":0,"permission":"644","type":"FILE","blockSize":134217728,
"pathSuffix":"","modificationTime":1485921930729,"childrenNum":0,"accessTime":1485921930710,"group":"supergroup","fileId":16407}
}
{noformat}
1.b: But,. when it is queried over shell, webhdfs returns only {{FileStatus}}
as return type which doesn't carry any EC related details. So, I am not sure if
we can make the following one to print extra details on EC file/dir.
{noformat}
hdfs dfs -ls webhdfs://127.0.0.1:5978/
{noformat}
2. {{OIV XML processor}} already has support for EC. Please review the output
below.
{noformat}
1
2 16406
3 DIRECTORY
4 ec
5 1485918336816
6 manoj:supergroup:0755
7
8
9 SYSTEM
10 hdfs.erasurecoding.policy <===
11 XOR-2-1-64k
12
13
14 -1
15 -1
16
17
18 16407
19 FILE
20 EmptyECFile.txt
21 3
22 1485918336813
23 1485918336796
24 134217728
25 manoj:supergroup:0644
26 0
27
28 STRIPED<===
29
30
{noformat}
2. {{OIV Delimited processor}} doesn't have support for EC. Here is the
proposal for the new Header ("BlockType") and value ("CONTIGUOUS"/"STRIPED").
{noformat}
PathReplication ModificationTimeAccessTime
PreferredBlockSize BlockType BlocksCount FileSizeNSQUOTA DSQUOTA
Permission UserNameGroupName
/ 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 9223372036854775807 -1 drwxr-xr-x manoj supergroup
/dir0 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 -1 -1 drwxr-xr-x manoj supergroup
/dir0/file0 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/dir0/file1 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/dir0/file2 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/dir0/file3 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/emptydir 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 -1 -1 drwxr-xr-x manoj supergroup
/ec 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 -1 -1 drwxr-xr-x manoj supergroup
/ec/EmptyECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728
STRIPED 0 0 0 0 -rw-r--r-- manoj supergroup
/ec/SmallECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728
STRIPED 1 0 0 0 -rw-r--r-- manoj supergroup
{noformat}
> OIV tool should make an EC file explicit
>
>
> Key: HDFS-10983
> URL: https://issues.apache.org/jira/browse/HDFS-10983
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-nice-to-have
>
> The OIV tool's webhdfs interface does not print if a file is striped or not.
> Also, it prints the file's EC policy ID as replication factor, which is
> inconsistent to the output of a typical webhdfs call to the cluster, which
> always shows replication factor of 0 for EC files.
> Not just webhdfs, but delimiter output does not print if a file is stripped
> or not either.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For
[jira] [Comment Edited] (HDFS-10983) OIV tool should make an EC file explicit
[
https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847993#comment-15847993
]
Manoj Govindassamy edited comment on HDFS-10983 at 2/1/17 4:11 AM:
---
[~andrew.wang], [~jojochuang],
Here are the proposals. Please let me know your thoughts on the below.
1. OIV HTTP server does expose a read-only WebHDFS API which can be queried to
print all file details.
1.a: Users can also get JSON formatted FileStatuses via HTTP REST API, which
can very well be extended. Here is the proposal for REST API output. Added
"hdfs.erasurecoding.policy" for Directory and "blockType" for File.
{noformat}
curl -i http://127.0.0.1:5978/webhdfs/v1/ec?op=getfilestatus
{"FileStatus":
{"owner":"manoj","replication":0,"hdfs.erasurecoding.policy":"XOR-2-1-64k",
"length":0,"permission":"755","type":"DIRECTORY",
"blockSize":0,"pathSuffix":"","modificationTime":1485921930732,
"childrenNum":2,"accessTime":0,"group":"supergroup","fileId":16406}
}
curl -i http://127.0.0.1:5978/webhdfs/v1/ec/file.txt?op=getfilestatus
{"FileStatus":
{"owner":"manoj","replication":3,"blockType":"STRIPED",
"length":0,"permission":"644","type":"FILE","blockSize":134217728,
"pathSuffix":"","modificationTime":1485921930729,"childrenNum":0,
"accessTime":1485921930710,"group":"supergroup","fileId":16407}
}
{noformat}
1.b: But,. when it is queried over shell, webhdfs returns only {{FileStatus}}
as return type which doesn't carry any EC related details. So, I am not sure if
we can make the following one to print extra details on EC file/dir.
{noformat}
hdfs dfs -ls webhdfs://127.0.0.1:5978/
{noformat}
2. {{OIV XML processor}} already has support for EC. Please review the output
below.
{noformat}
1
2 16406
3 DIRECTORY
4 ec
5 1485918336816
6 manoj:supergroup:0755
7
8
9 SYSTEM
10 hdfs.erasurecoding.policy <===
11 XOR-2-1-64k
12
13
14 -1
15 -1
16
17
18 16407
19 FILE
20 EmptyECFile.txt
21 3
22 1485918336813
23 1485918336796
24 134217728
25 manoj:supergroup:0644
26 0
27
28 STRIPED<===
29
30
{noformat}
2. {{OIV Delimited processor}} doesn't have support for EC. Here is the
proposal for the new Header ("BlockType") and value ("CONTIGUOUS"/"STRIPED").
{noformat}
PathReplication ModificationTimeAccessTime
PreferredBlockSize BlockType BlocksCount FileSizeNSQUOTA DSQUOTA
Permission UserNameGroupName
/ 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 9223372036854775807 -1 drwxr-xr-x manoj supergroup
/dir0 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 -1 -1 drwxr-xr-x manoj supergroup
/dir0/file0 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/dir0/file1 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/dir0/file2 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/dir0/file3 3 2017-01-31 18:572017-01-31 18:57134217728
CONTIGUOUS 1 1 0 0 -rw-r--r-- manoj supergroup
/emptydir 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 -1 -1 drwxr-xr-x manoj supergroup
/ec 0 2017-01-31 18:571969-12-31 16:000
NA 0 0 -1 -1 drwxr-xr-x manoj supergroup
/ec/EmptyECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728
STRIPED 0 0 0 0 -rw-r--r-- manoj supergroup
/ec/SmallECFile.txt 3 2017-01-31 18:572017-01-31 18:57134217728
STRIPED 1 0 0 0 -rw-r--r-- manoj supergroup
{noformat}
was (Author: manojg):
[~andrew.wang], [~jojochuang],
Here are the proposals. Please let me know your thoughts on the below.
1. OIV HTTP server does expose a read-only WebHDFS API which can be queried to
print all file details.
1.a: Users can also get JSON formatted FileStatuses via HTTP REST API, which
can very well be extended. Here is the proposal for
{noformat}
curl -i http://127.0.0.1:5978/webhdfs/v1/ec?op=getfilestatus
{"FileStatus":
{"owner":"manoj","replication":0,"hdfs.erasurecoding.policy":"XOR-2-1-64k",
"length":0,"permission":"755","type":"DIRECTORY",
"blockSize":0,"pathSuffix":"","modificationTime":1485921930732,"childrenNum":2,"accessTime":0,"group":"supergroup","fileId":16406}
}
curl -i http://127.0.0.1:5978/webhdfs/v1/ec/file.txt?op=getfilestatus
{"FileStatus":
{"owner":"manoj","replication":3,"bl
[jira] [Commented] (HDFS-11370) Optimize NamenodeFsck#getReplicaInfo
[
https://issues.apache.org/jira/browse/HDFS-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848049#comment-15848049
]
Hadoop QA commented on HDFS-11370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850367/HDFS-11370.6.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fec09a0d67a1 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3e06475 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18302/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18302/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18302/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize NamenodeFsck#getReplicaInfo
>
>
> Key: HDFS-11370
> URL: https://issues.apache.org/jira/browse/HDFS-11370
> Project: Hadoop HDFS
> Issue Type: Improvement
[jira] [Updated] (HDFS-11335) Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf
[
https://issues.apache.org/jira/browse/HDFS-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-11335:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
LGTM as well. +1. Committed to trunk and branch-2
Thanks [~manojg] for working on this patch and [~linyiqun] for the reviews!
> Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage
> from DNConf
> --
>
> Key: HDFS-11335
> URL: https://issues.apache.org/jira/browse/HDFS-11335
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HDFS-11335.01.patch, HDFS-11335.02.patch,
> HDFS-11335.03.patch
>
>
> DataNode configuration {{DNConf}} reading in
> {{HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY}}
> (_dfs.client.slow.io.warning.threshold.ms_) is redundant as this threshold
> limit is needed only for {{DfsClientConf}}. Better to remove the unwanted
> usage in DNConf.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11340) DataNode reconfigure for disks doesn't remove the failed volumes
[
https://issues.apache.org/jira/browse/HDFS-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848074#comment-15848074
]
Lei (Eddy) Xu commented on HDFS-11340:
--
Thanks for the explanation, [~manojg]. It makes sense.
> DataNode reconfigure for disks doesn't remove the failed volumes
>
>
> Key: HDFS-11340
> URL: https://issues.apache.org/jira/browse/HDFS-11340
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11340.01.patch
>
>
> Say a DataNode (uuid:xyz) has disks D1 and D2. When D1 turns bad, JMX query
> on FSDatasetState-xyz for "NumFailedVolumes" attr rightly shows the failed
> volume count as 1 and the "FailedStorageLocations" attr has the failed
> storage location as "D1".
> It is possible to add or remove disks to this DataNode by running
> {{reconfigure}} command. Let the failed disk D1 be removed from the conf and
> the new conf has only one good disk D2. Upon running the reconfigure command
> for this DataNode with this new disk conf, the expectation is DataNode would
> no more have "NumFailedVolumes" or "FailedStorageLocations". But, even after
> removing the failed disk from the conf and a successful reconfigure, DataNode
> continues to show the "NumFailedVolumes" as 1 and "FailedStorageLocations" as
> "D1" and it never gets reset.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11377) Balancer hung due to "No mover threads available"
[
https://issues.apache.org/jira/browse/HDFS-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848076#comment-15848076
]
Yiqun Lin commented on HDFS-11377:
--
Thanks [~zhaoyunjiong] for reporting this and thanks [~manojg] for the review.
It's a nice finding. The patch also looks good to me. One minor comment, can
you also remove the unused variable {{MAX_NO_PENDING_MOVE_ITERATIONS}} in
{{Dispatcher}}? This hardcoded value has been replaced by option
{{-idleiterations}}. +1 once addressed.
> Balancer hung due to "No mover threads available"
> -
>
> Key: HDFS-11377
> URL: https://issues.apache.org/jira/browse/HDFS-11377
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-11377.001.patch
>
>
> When running balancer on large cluster which have more than 3000 Datanodes,
> it might be hung due to "No mover threads available".
> The stack trace shows it waiting forever like below.
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7ff6cc014800 nid=0x6b2c waiting on
> condition [0x7ff6d1bad000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1043)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1017)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:981)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:611)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:663)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:776)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:905)
> {code}
> In the log, there are lots of WARN about "No mover threads available".
> {quote}
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_13700554102_1112815018180 with size=268435456 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through
> 10.115.67.137:50010
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_4009558842_1103118359883 with size=268435456 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through
> 10.115.67.137:50010
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_13881956058_1112996460026 with size=133509566 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through 10.115.67.36:50010
> {quote}
> What happened here is, when there are no mover threads available,
> DDatanode.isPendingQEmpty() will return false, so Balancer hung.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11335) Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf
[
https://issues.apache.org/jira/browse/HDFS-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848080#comment-15848080
]
Hudson commented on HDFS-11335:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11193 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11193/])
HDFS-11335. Remove (lei: rev bec9b7aa1dd3ed95b8783597135f8d90b3cc8dcd)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
> Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage
> from DNConf
> --
>
> Key: HDFS-11335
> URL: https://issues.apache.org/jira/browse/HDFS-11335
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HDFS-11335.01.patch, HDFS-11335.02.patch,
> HDFS-11335.03.patch
>
>
> DataNode configuration {{DNConf}} reading in
> {{HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY}}
> (_dfs.client.slow.io.warning.threshold.ms_) is redundant as this threshold
> limit is needed only for {{DfsClientConf}}. Better to remove the unwanted
> usage in DNConf.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11377) Balancer hung due to no available mover threads
[
https://issues.apache.org/jira/browse/HDFS-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11377:
-
Target Version/s: 2.9.0, 3.0.0-alpha3
Component/s: balancer & mover
Summary: Balancer hung due to no available mover threads (was:
Balancer hung due to "No mover threads available")
> Balancer hung due to no available mover threads
> ---
>
> Key: HDFS-11377
> URL: https://issues.apache.org/jira/browse/HDFS-11377
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.7.3
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-11377.001.patch
>
>
> When running balancer on large cluster which have more than 3000 Datanodes,
> it might be hung due to "No mover threads available".
> The stack trace shows it waiting forever like below.
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x7ff6cc014800 nid=0x6b2c waiting on
> condition [0x7ff6d1bad000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1043)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1017)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:981)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:611)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:663)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:776)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:905)
> {code}
> In the log, there are lots of WARN about "No mover threads available".
> {quote}
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_13700554102_1112815018180 with size=268435456 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through
> 10.115.67.137:50010
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_4009558842_1103118359883 with size=268435456 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through
> 10.115.67.137:50010
> 2017-01-26 15:36:40,085 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: No mover threads
> available: skip moving blk_13881956058_1112996460026 with size=133509566 from
> 10.115.67.137:50010:DISK to 10.140.21.55:50010:DISK through 10.115.67.36:50010
> {quote}
> What happened here is, when there are no mover threads available,
> DDatanode.isPendingQEmpty() will return false, so Balancer hung.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10860) Switch HttpFS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HDFS-10860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848089#comment-15848089
]
John Zhuge commented on HDFS-10860:
---
{{HttpServer2#isInstrumentationAccessAllowed}} requires the following
conditions to return false:
* hadoop.security.instrumentation.requires.admin set to true
* hadoop.security.authorization set to true
* remote user is not set or {{HttpServer2#setACL}} is called with an ACL that
disallows the remote user
Looks like we need another configuration property to set administrators for
HttpFS server instrumentation access, e.g.,
{{hadoop.httpfs.http.administrators}}, similar to
{{dfs.cluster.administrators}}.
Will need to fix KMS to call {{HttpServer2#setACL}}. Also gotta make sure all
users of {{HttpServer2}} call {{setACL}}.
> Switch HttpFS from Tomcat to Jetty
> --
>
> Key: HDFS-10860
> URL: https://issues.apache.org/jira/browse/HDFS-10860
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-10860.001.patch, HDFS-10860.002.patch,
> HDFS-10860.003.patch, HDFS-10860.004.patch, HDFS-10860.005.patch,
> HDFS-10860.006.patch, HDFS-10860.007.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have to change client code that much. It would
> require more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
