[jira] [Created] (HADOOP-10687) improve Configuration.CACHE_CLASSES concurrentcy
Liang Xie created HADOOP-10687:
--
Summary: improve Configuration.CACHE_CLASSES concurrentcy
Key: HADOOP-10687
URL: https://issues.apache.org/jira/browse/HADOOP-10687
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 2.4.0, 3.0.0
Reporter: Liang Xie
Assignee: Liang Xie
In one of my high concurrent HBase in-memory read only testing scenario, the
following strace traces show CACHE_CLASSES becoming hotspot, since there's a
synchronized or synchronizedMap, but for most of hadoop applications, when
goes into getClassByNameOrNull(), most of code paths should expect there's a
cached class already, that means it's a read mostly scenario, the write should
be rare enough. So one possible optimization here is using concurrent
weakHashMap, the MapMaker could do this.
ps: i added more debug logging above cache hit and Class.forName and reran
my case, shows an expected result (almost all of them go to cache hit path),
and TestConfiguration also passed in my box.
{code}
IPC Reader 3 on port 11600 daemon prio=10 tid=0x7ff2fcb41190 nid=0x5db4
runnable [0x7ff238943000]
java.lang.Thread.State: RUNNABLE
at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
- locked 0x0002788ab4a0 (a java.util.Collections$SynchronizedMap)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
- locked 0x0002784211e8 (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
IPC Reader 1 on port 11600 daemon prio=10 tid=0x7ff2fc8cb2b0 nid=0x5db2
waiting for monitor entry [0x7ff29006c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
- waiting to lock 0x0002788ab4a0 (a
java.util.Collections$SynchronizedMap)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:617)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362)
at
org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:775)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:566)
- locked 0x000278434980 (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:541)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
IPC Reader 0 on port 11600 daemon prio=10 tid=0x7ff2fc9b7630 nid=0x5db1
waiting for monitor entry [0x7ff2900ad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Collections$SynchronizedMap.get(Collections.java:1979)
- waiting to lock 0x0002788ab4a0 (a
java.util.Collections$SynchronizedMap)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1505)
at
[jira] [Resolved] (HADOOP-10640) Implement Namenode RPCs in HDFS native client
[ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HADOOP-10640. --- Resolution: Fixed Fix Version/s: HADOOP-10388 Implement Namenode RPCs in HDFS native client - Key: HADOOP-10640 URL: https://issues.apache.org/jira/browse/HADOOP-10640 Project: Hadoop Common Issue Type: Sub-task Components: native Affects Versions: HADOOP-10388 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: HADOOP-10388 Attachments: HADOOP-10640-pnative.001.patch, HADOOP-10640-pnative.002.patch, HADOOP-10640-pnative.003.patch, HADOOP-10640-pnative.004.patch Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10447) Implement C code for parsing Hadoop / HDFS URIs
[ https://issues.apache.org/jira/browse/HADOOP-10447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HADOOP-10447. --- Resolution: Duplicate Assignee: Colin Patrick McCabe Implement C code for parsing Hadoop / HDFS URIs --- Key: HADOOP-10447 URL: https://issues.apache.org/jira/browse/HADOOP-10447 Project: Hadoop Common Issue Type: Sub-task Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe We need some glue code for parsing Hadoop or HDFS URIs in C. Probably we should just put a small 'Path' wrapper around a URI parsing library like http://uriparser.sourceforge.net/ (BSD licensed) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10446) native code for reading Hadoop configuration XML files
[ https://issues.apache.org/jira/browse/HADOOP-10446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HADOOP-10446. --- Resolution: Duplicate Fix Version/s: HADOOP-10388 Assignee: Colin Patrick McCabe Target Version/s: HADOOP-10388 native code for reading Hadoop configuration XML files -- Key: HADOOP-10446 URL: https://issues.apache.org/jira/browse/HADOOP-10446 Project: Hadoop Common Issue Type: Sub-task Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: HADOOP-10388 We need to have a way to read Hadoop configuration XML files in the native HDFS and YARN clients. This will allow those clients to discover the locations of NameNodes, YARN daemons, and other configuration settings, etc. etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10688) Expose thread-level FileSystem StatisticsData
Sandy Ryza created HADOOP-10688: --- Summary: Expose thread-level FileSystem StatisticsData Key: HADOOP-10688 URL: https://issues.apache.org/jira/browse/HADOOP-10688 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Sandy Ryza Assignee: Sandy Ryza FileSystems collect data on the bytes read and written by each thread. It would be helpful for it to make this data public. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
[ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-10663. Resolution: Won't Fix Thanks, Chris. Resolving. Path.getFileSystem should identify Windows drive (c:) like files and return the localFS --- Key: HADOOP-10663 URL: https://issues.apache.org/jira/browse/HADOOP-10663 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Remus Rusanu Assignee: Remus Rusanu Attachments: HADOOP-10663.1.patch While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking a Windows specific Path to get its file system should not return the default FS, but the localFS. Specifically, I believe that Path.initialize should identify such names as belonging to the 'file://' scheme (when null) and so create appropriate URIs. I do not have the know-how to evaluate the bigger impact of such a change. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
Ted Yu created HADOOP-10690:
---
Summary: Lack of synchronization on access to InputStream in
NativeAzureFileSystem#NativeAzureFsInputStream#close()
Key: HADOOP-10690
URL: https://issues.apache.org/jira/browse/HADOOP-10690
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void close() throws IOException {
in.close();
}
{code}
The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-10514) Common side changes to support HDFS extended attributes (HDFS-2006)
[ https://issues.apache.org/jira/browse/HADOOP-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu resolved HADOOP-10514. - Resolution: Fixed Hadoop Flags: Reviewed Common side changes to support HDFS extended attributes (HDFS-2006) Key: HADOOP-10514 URL: https://issues.apache.org/jira/browse/HADOOP-10514 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Uma Maheswara Rao G Assignee: Yi Liu This is an umbrella issue for tracking all Hadoop Common changes required to support HDFS extended attributes implementation -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-10682) Metrics are not output in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA resolved HADOOP-10682.
Resolution: Not a Problem
It's caused by the change of HADOOP-10468. Regarding the issue, we should set
the property case sensitive from 2.5.0. Closing.
Metrics are not output in trunk
---
Key: HADOOP-10682
URL: https://issues.apache.org/jira/browse/HADOOP-10682
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Akira AJISAKA
Metrics are not output in trunk by the following configuration:
{code}
*.sink.file.class=org.apache.Hadoop.metrics2.sink.FileSink
*.period=10
namenode.sink.file.filename=namenode-metrics.out
{code}
The below change worked well.
{code}
- namenode.sink.file.filename=namenode-metrics.out
+ NameNode.sink.file.filename=namenode-metrics.out
{code}
It means that an old configuration doesn't work on trunk. We should fix it or
document to use NameNode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10692) Update hadoop-metrics2.properties examples to be case sensitive
Akira AJISAKA created HADOOP-10692: -- Summary: Update hadoop-metrics2.properties examples to be case sensitive Key: HADOOP-10692 URL: https://issues.apache.org/jira/browse/HADOOP-10692 Project: Hadoop Common Issue Type: Bug Components: conf, metrics Affects Versions: 2.5.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA After HADOOP-10468, the prefix of the properties in metrics2 become case sensitive. We should also update hadoop-metrics2.properties examples to be case sensitive. -- This message was sent by Atlassian JIRA (v6.2#6252)
