[jira] [Created] (HADOOP-15887) Add an option to avoid writing data locally in Distcp
Tao Jie created HADOOP-15887: Summary: Add an option to avoid writing data locally in Distcp Key: HADOOP-15887 URL: https://issues.apache.org/jira/browse/HADOOP-15887 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0, 2.8.2 Reporter: Tao Jie Assignee: Tao Jie When copying large amount of data from one cluster to another via Distcp, and the Distcp jobs run in the target cluster, the datanode local usage would be imbalanced. Because the default placement policy chooses the local node to store the first replication. In https://issues.apache.org/jira/browse/HDFS-3702 we add a flag in DFSClient to avoid replicating to the local datanode. We can make use of this flag in Distcp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15419) Should not obtain delegationTokens from all namenodes when using ViewFS
Tao Jie created HADOOP-15419: Summary: Should not obtain delegationTokens from all namenodes when using ViewFS Key: HADOOP-15419 URL: https://issues.apache.org/jira/browse/HADOOP-15419 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.2 Reporter: Tao Jie Today when submit a job to a viewfs cluster, the client will try to obtain delegation token from all namenodes under the viewfs while only one namespace is actually used in this job. It would create many unnecessary rpc call to the whole cluster. In viewfs situation, we can just obtain delegation token from specific namenode rather than all namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15253) Should update maxQueueSize when refresh call queue
Tao Jie created HADOOP-15253:
Summary: Should update maxQueueSize when refresh call queue
Key: HADOOP-15253
URL: https://issues.apache.org/jira/browse/HADOOP-15253
Project: Hadoop Common
Issue Type: Improvement
Reporter: Tao Jie
Assignee: Tao Jie
When calling {{dfsadmin -refreshCallQueue}} to update CallQueue instance,
{{maxQueueSize}} should also be updated.
In case of changing CallQueue instance to FairCallQueue, the length of each
queue in FairCallQueue would be 1/priorityLevels of original length of
DefaultCallQueue. So it would be helpful for us to set the length of callqueue
to a proper value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15194) Some properties for FairCallQueue should be reloadable
Tao Jie created HADOOP-15194: Summary: Some properties for FairCallQueue should be reloadable Key: HADOOP-15194 URL: https://issues.apache.org/jira/browse/HADOOP-15194 Project: Hadoop Common Issue Type: Improvement Reporter: Tao Jie When trying to make use of FairCallQueue for a large cluster, it is not easy to set the properties appropriately at the first time. User may adjust those properties frequently and restarting the namenode costs a lot for a large multi-user hadoop clluster. We should support to reload some of those properties in configuration without restarting. Related properties: ipc.8020.faircallqueue.multiplexer.weights ipc.8020.faircallqueue.decay-scheduler.thresholds ipc.8020.decay-scheduler.backoff.responsetime.thresholds -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15121) Encounter NullPointerException when using DecayRpcScheduler
Tao Jie created HADOOP-15121:
Summary: Encounter NullPointerException when using
DecayRpcScheduler
Key: HADOOP-15121
URL: https://issues.apache.org/jira/browse/HADOOP-15121
Project: Hadoop Common
Issue Type: Bug
Reporter: Tao Jie
I set ipc.8020.scheduler.impl to org.apache.hadoop.ipc.DecayRpcScheduler, but
got excetion in namenode:
{code}
2017-12-15 15:26:34,662 ERROR impl.MetricsSourceAdapter
(MetricsSourceAdapter.java:getMetrics(202)) - Error getting metrics from source
DecayRpcSchedulerMetrics2.ipc.8020
java.lang.NullPointerException
at
org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.getMetrics(DecayRpcScheduler.java:781)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:199)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:182)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:155)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:66)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:100)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
at
org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.registerMetrics2Source(DecayRpcScheduler.java:709)
at
org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.(DecayRpcScheduler.java:685)
at
org.apache.hadoop.ipc.DecayRpcScheduler$MetricsProxy.getInstance(DecayRpcScheduler.java:693)
at
org.apache.hadoop.ipc.DecayRpcScheduler.(DecayRpcScheduler.java:236)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.CallQueueManager.createScheduler(CallQueueManager.java:102)
at
org.apache.hadoop.ipc.CallQueueManager.(CallQueueManager.java:76)
at org.apache.hadoop.ipc.Server.(Server.java:2612)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:958)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:374)
at
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:349)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:800)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:415)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:755)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:697)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:905)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:884)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678)
{code}
It seems that {{metricsProxy}} in DecayRpcScheduler should initiate its
{{delegate}} field in its Initialization method
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-12569) ZKFC should stop namenode before itself quit in some circumstances
Tao Jie created HADOOP-12569: Summary: ZKFC should stop namenode before itself quit in some circumstances Key: HADOOP-12569 URL: https://issues.apache.org/jira/browse/HADOOP-12569 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.6.0 Reporter: Tao Jie We have met such a HA scenario: NN1(active) and zkfc1 on node1; NN2(standby) and zkfc2 on node2. 1,Stop network on node1, NN2 becomes active. On node2, zkfc2 kills itself since it cannot connect to zookeeper, but leaving NN1 still running. 2,Several minutes later, network on node1 recovers. NN1 is running but out of control. NN1 and NN2 both run as active nn. Maybe zkfc should stop nn before quit in such circumstances. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12331) FSShell Delete and Rename should operate on symlinks rather than their target
Tao Jie created HADOOP-12331: Summary: FSShell Delete and Rename should operate on symlinks rather than their target Key: HADOOP-12331 URL: https://issues.apache.org/jira/browse/HADOOP-12331 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.6.0 Reporter: Tao Jie Priority: Minor Currently, we remove or rename a symlink in FSShell, symlink target is affected instead of symlink itself. However FileSystem#delete and FileSystem#rename can operate on symlinks rather than their targets. FSShell should have consistent behavior as FileSystem on symlink remove and rename -- This message was sent by Atlassian JIRA (v6.3.4#6332)
