[jira] [Created] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock
Gopal V created HADOOP-16461: Summary: Regression: FileSystem cache lock parses XML within the lock Key: HADOOP-16461 URL: https://issues.apache.org/jira/browse/HADOOP-16461 Project: Hadoop Common Issue Type: Bug Components: filecache Reporter: Gopal V {code} fs = createFileSystem(uri, conf); synchronized (this) { // refetch the lock again FileSystem oldfs = map.get(key); if (oldfs != null) { // a file system is created while lock is releasing fs.close(); // close the new file system return oldfs; // return the old file system } // now insert the new file system into the map if (map.isEmpty() && !ShutdownHookManager.get().isShutdownInProgress()) { ShutdownHookManager.get().addShutdownHook(clientFinalizer, SHUTDOWN_HOOK_PRIORITY); } fs.key = key; map.put(key, fs); if (conf.getBoolean( FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) { toAutoClose.add(key); } return fs; } {code} The lock now has a ShutdownHook creation, which ends up doing {code} HookEntry(Runnable hook, int priority) { this(hook, priority, getShutdownTimeout(new Configuration()), TIME_UNIT_DEFAULT); } {code} which ends up doing a "new Configuration()" within the locked section. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16341) ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
Gopal V created HADOOP-16341: Summary: ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 Key: HADOOP-16341 URL: https://issues.apache.org/jira/browse/HADOOP-16341 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.1.2 Reporter: Gopal V Attachments: shutdown-hook-removal.png !shutdown-hook-removal.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14054) NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls in it
[ https://issues.apache.org/jira/browse/HADOOP-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HADOOP-14054. -- Resolution: Won't Fix Closing as WONTFIX - will file issues on Azure github > NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls > in it > - > > Key: HADOOP-14054 > URL: https://issues.apache.org/jira/browse/HADOOP-14054 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Gopal V > > See the lock around the multiple buffered input streams with the read1 > java.io.BufferedInputStream.read() calls are all synchronized blocks. > {code} > IO-Elevator-Thread-9" #288 daemon prio=5 os_prio=0 tid=0x7fbf9c0aa000 > nid=0xa07d runnable [0x7faf3b4b4000] >java.lang.Thread.State: RUNNABLE > sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:140) > at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:127) > at > java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:584) > at java.security.MessageDigest.update(MessageDigest.java:325) > at > com.microsoft.azure.storage.core.Utility.writeToOutputStream(Utility.java:1329) > at > com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1410) > at > com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1310) > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:146) > at > com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1499) > at > com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255) > - eliminated <0x7fbdd5475b68> (a > com.microsoft.azure.storage.blob.BlobInputStream) > at > com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) > - locked <0x7fbdd5475b68> (a > com.microsoft.azure.storage.blob.BlobInputStream) > at > com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > - locked <0x7fbdd5475b38> (a java.io.BufferedInputStream) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:820) > - locked <0x7fbdd5475ac0> (a > org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > - locked <0x7fbdd5475a90> (a > org.apache.hadoop.fs.BufferedFSInputStream) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62) > at > org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174) > at > org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14054) NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls in it
Gopal V created HADOOP-14054: Summary: NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls in it Key: HADOOP-14054 URL: https://issues.apache.org/jira/browse/HADOOP-14054 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Gopal V See the lock around the multiple buffered input streams with the read1 java.io.BufferedInputStream.read() calls are all synchronized blocks. {code} IO-Elevator-Thread-9" #288 daemon prio=5 os_prio=0 tid=0x7fbf9c0aa000 nid=0xa07d runnable [0x7faf3b4b4000] java.lang.Thread.State: RUNNABLE sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:140) at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:127) at java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:584) at java.security.MessageDigest.update(MessageDigest.java:325) at com.microsoft.azure.storage.core.Utility.writeToOutputStream(Utility.java:1329) at com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1410) at com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1310) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:146) at com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1499) at com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255) - eliminated <0x7fbdd5475b68> (a com.microsoft.azure.storage.blob.BlobInputStream) at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) - locked <0x7fbdd5475b68> (a com.microsoft.azure.storage.blob.BlobInputStream) at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x7fbdd5475b38> (a java.io.BufferedInputStream) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:820) - locked <0x7fbdd5475ac0> (a org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream) at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x7fbdd5475a90> (a org.apache.hadoop.fs.BufferedFSInputStream) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62) at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174) at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-12956) Inevitable Log4j2 migration via slf4j
Gopal V created HADOOP-12956: Summary: Inevitable Log4j2 migration via slf4j Key: HADOOP-12956 URL: https://issues.apache.org/jira/browse/HADOOP-12956 Project: Hadoop Common Issue Type: Improvement Reporter: Gopal V {{5 August 2015 --The Apache Logging Services™ Project Management Committee (PMC) has announced that the Log4j™ 1.x logging framework has reached its end of life (EOL) and is no longer officially supported.}} https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces A whole framework log4j2 upgrade has to be synchronized, partly for improved performance brought about by log4j2. https://logging.apache.org/log4j/2.x/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11867) FS API: Add a high-performance vectored Read to FSDataInputStream API
Gopal V created HADOOP-11867: Summary: FS API: Add a high-performance vectored Read to FSDataInputStream API Key: HADOOP-11867 URL: https://issues.apache.org/jira/browse/HADOOP-11867 Project: Hadoop Common Issue Type: New Feature Affects Versions: 2.8.0 Reporter: Gopal V The most significant way to read from a filesystem in an efficient way is to let the FileSystem implementation handle the seek behaviour underneath the API to be the most efficient as possible. A better approach to the seek problem is to provide a sequence of read locations as part of a single call, while letting the system schedule/plan the reads ahead of time. This is exceedingly useful for seek-heavy readers on HDFS, since this allows for potentially optimizing away the seek-gaps within the FSDataInputStream implementation. For seek+read systems with even more latency than locally-attached disks, something like a {{readFully(long[] offsets, ByteBuffer[] chunks)}} would take of the seeks internally while reading chunk.remaining() bytes into each chunk (which may be {{slice()}}ed off a bigger buffer). The base implementation can stub in this as a sequence of seeks + read() into ByteBuffers, without forcing each FS implementation to override this in any way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Gopal V created HADOOP-11772: Summary: RPC Invoker relies on static ClientCache which has synchronized(this) blocks Key: HADOOP-11772 URL: https://issues.apache.org/jira/browse/HADOOP-11772 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Gopal V Attachments: sync-client-bt.png, sync-client-threads.png {code} private static ClientCache CLIENTS=new ClientCache(); ... this.client = CLIENTS.getClient(conf, factory); {code} Meanwhile in ClientCache {code} public synchronized Client getClient(Configuration conf, SocketFactory factory, Class valueClass) { ... Client client = clients.get(factory); if (client == null) { client = new Client(valueClass, conf, factory); clients.put(factory, client); } else { client.incCount(); } {code} All invokers end up calling these methods, resulting in IPC clients choking up. !sync-client-threads.png! !sync-client-bt.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11771) Configuration::getClassByNameOrNull synchronizes on a static object
Gopal V created HADOOP-11771: Summary: Configuration::getClassByNameOrNull synchronizes on a static object Key: HADOOP-11771 URL: https://issues.apache.org/jira/browse/HADOOP-11771 Project: Hadoop Common Issue Type: Sub-task Reporter: Gopal V {code} private static final Map>>> CACHE_CLASSES = new WeakHashMap>>>(); ... synchronized (CACHE_CLASSES) { map = CACHE_CLASSES.get(classLoader); if (map == null) { map = Collections.synchronizedMap( new WeakHashMap>>()); CACHE_CLASSES.put(classLoader, map); } } {code} !configuration-sync-cache.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11770) [Umbrella] locate static synchronized blocks in hadoop-common
Gopal V created HADOOP-11770: Summary: [Umbrella] locate static synchronized blocks in hadoop-common Key: HADOOP-11770 URL: https://issues.apache.org/jira/browse/HADOOP-11770 Project: Hadoop Common Issue Type: Improvement Components: conf, io, ipc Affects Versions: 2.7.0 Reporter: Gopal V There are several static synchronized blocks in the hadoop common functionality that hurts any multi-threaded processing system which uses the common APIs. Identify the static synchronized blocks and locate them for potential fixes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11239) [JDK8] azurenative tests fail builds on JDK8
[ https://issues.apache.org/jira/browse/HADOOP-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HADOOP-11239. -- Resolution: Not a Problem > [JDK8] azurenative tests fail builds on JDK8 > > > Key: HADOOP-11239 > URL: https://issues.apache.org/jira/browse/HADOOP-11239 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 2.6.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Trivial > > java.util.Base64 conflicts with > com.microsoft.windowsazure.storage.core.Base64 in Azure unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11239) [JDK8] azurenative tests fail builds on JDK8
Gopal V created HADOOP-11239: Summary: [JDK8] azurenative tests fail builds on JDK8 Key: HADOOP-11239 URL: https://issues.apache.org/jira/browse/HADOOP-11239 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Gopal V Assignee: Gopal V Priority: Trivial java.util.Base64 conflicts with com.microsoft.windowsazure.storage.core.Base64 in Azure unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11226) ipc.Client has to use setTrafficClass() with IPTOS_LOWDELAY
Gopal V created HADOOP-11226: Summary: ipc.Client has to use setTrafficClass() with IPTOS_LOWDELAY Key: HADOOP-11226 URL: https://issues.apache.org/jira/browse/HADOOP-11226 Project: Hadoop Common Issue Type: Bug Reporter: Gopal V -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
Gopal V created HADOOP-11223: Summary: Offer a read-only conf alternative to new Configuration() Key: HADOOP-11223 URL: https://issues.apache.org/jira/browse/HADOOP-11223 Project: Hadoop Common Issue Type: Bug Reporter: Gopal V new Configuration() is called from several static blocks across Hadoop. This is incredibly inefficient, since each one of those involves primarily XML parsing at a point where the JIT won't be triggered & interpreter mode is essentially forced on the JVM. The alternate solution would be to offer a {{Configuration::getDefault()}} alternative which disallows any modifications. At the very least, such a method would need to be called from # org.apache.hadoop.io.nativeio.NativeIO::() # org.apache.hadoop.security.SecurityUtil::() # org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider:: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11029) LocalFS Statistics performs thread local call per byte written
Gopal V created HADOOP-11029: Summary: LocalFS Statistics performs thread local call per byte written Key: HADOOP-11029 URL: https://issues.apache.org/jira/browse/HADOOP-11029 Project: Hadoop Common Issue Type: Bug Reporter: Gopal V Attachments: local-fs-locking.png This code is there in the hot-path of IFile writer via RawLocalFileSystem. !local-fs-locking.png! >From a preliminary glance, the lock prefix calls are coming from a >threadlocal.get() within FileSystem.Statistics {code} /** * Get or create the thread-local data associated with the current thread. */ private StatisticsData getThreadData() { StatisticsData data = threadData.get(); if (data == null) { data = new StatisticsData( new WeakReference(Thread.currentThread())); threadData.set(data); synchronized(this) { if (allData == null) { allData = new LinkedList(); } allData.add(data); } } return data; } /** * Increment the bytes read in the statistics * @param newBytes the additional bytes read */ public void incrementBytesRead(long newBytes) { getThreadData().bytesRead += newBytes; } {code} This is incredibly inefficient when used from FSDataOutputStream {code} public void write(int b) throws IOException { out.write(b); position++; if (statistics != null) { statistics.incrementBytesWritten(1); } } {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10694) Remove synchronized input streams from Writable deserialization
Gopal V created HADOOP-10694: Summary: Remove synchronized input streams from Writable deserialization Key: HADOOP-10694 URL: https://issues.apache.org/jira/browse/HADOOP-10694 Project: Hadoop Common Issue Type: Bug Components: io Reporter: Gopal V Assignee: Gopal V Attachments: writable-read-sync.png Writable deserialization is slowing down due to a synchronized block within DataInputBuffer$Buffer. ByteArrayInputStream::read() is synchronized and this shows up as a slow uncontested lock. Hive ships with its own faster thread-unsafe version with hive.common.io.NonSyncByteArrayInputStream. !writable-read-sync.png! The DataInputBuffer and Writable deserialization should not require a lock per readInt()/read(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10681) Compressor inner methods are all synchronized - within a tight loop
Gopal V created HADOOP-10681: Summary: Compressor inner methods are all synchronized - within a tight loop Key: HADOOP-10681 URL: https://issues.apache.org/jira/browse/HADOOP-10681 Project: Hadoop Common Issue Type: Bug Components: performance Affects Versions: 2.4.0, 2.2.0, 2.5.0 Reporter: Gopal V Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png The current implementation of SnappyCompressor spends more time within the java loop of copying from the user buffer into the direct buffer allocated to the compressor impl, than the time it takes to compress the buffers. !perf-top-spill-merge.png! The bottleneck was found to be java monitor code inside SnappyCompressor. The methods are neatly inlined by the JIT into the parent caller (BlockCompressorStream::write), which unfortunately does not flatten out the synchronized blocks. !compress-cmpxchg-small.png! The loop does a write of small byte[] buffers (each IFile key+value). I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10513) ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders
Gopal V created HADOOP-10513: Summary: ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders Key: HADOOP-10513 URL: https://issues.apache.org/jira/browse/HADOOP-10513 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 2.4.0 Reporter: Gopal V Any code which uses custom class-loaders needs to be aware of the ReflectionUtils::CONSTRUCTOR_CACHE {code} /** * Cache of constructors for each class. Pins the classes so they * can't be garbage collected until ReflectionUtils can be collected. */ private static final Map, Constructor> CONSTRUCTOR_CACHE = new ConcurrentHashMap, Constructor>(); {code} This is not a problem when using only 1 AppClassLoader. But in cases where the application uses multiple classloaders (to isolate UDFs), this holds onto class references and their associated class loaders by ref (& that leaks all the statics). The clear method for this cache is unfortunately marked only for testing. {code} // methods to support testing static void clearCache() { CONSTRUCTOR_CACHE.clear(); } {code} The cache shows up as the only reference for the class loaders. !class-loader-leak.png! -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-8921) ant build.xml in branch-1 ignores -Dcompile.native
[ https://issues.apache.org/jira/browse/HADOOP-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HADOOP-8921. - Resolution: Unresolved > ant build.xml in branch-1 ignores -Dcompile.native > -- > > Key: HADOOP-8921 > URL: https://issues.apache.org/jira/browse/HADOOP-8921 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 1.2.0 > Environment: Mac OS X 10.7.4 >Reporter: Gopal V >Assignee: Gopal V >Priority: Trivial > Labels: ant, autoconf, patch > Attachments: HADOOP-8921.4.patch, HADOOP-8921.5.patch, > HADOOP-8921.7.patch, hadoop-8921.6.patch > > > ant -Dcompile.native=false still runs autoconf and libtoolize > According to ant 1.8 manual, any conditions are checked only > after the dependencies are run through. The current if condition in code > fails to prevent the autoconf/libtool components from running. > Fixing it by moving the if condition up into the "compile-native" target and > changing it to a param substitution instead of being evaluated as a condition. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10047) Allow Compressor/Decompressor APIs to expose a Direct ByteBuffer API
Gopal V created HADOOP-10047: Summary: Allow Compressor/Decompressor APIs to expose a Direct ByteBuffer API Key: HADOOP-10047 URL: https://issues.apache.org/jira/browse/HADOOP-10047 Project: Hadoop Common Issue Type: New Feature Components: io Reporter: Gopal V Assignee: Gopal V With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform all I/O operations without copying data into byte[] buffers or other buffers which wrap over them. This is a proposal for adding new DirectCompressor and DirectDecompressor interfaces to the io.compress, to indicate codecs which want to surface the direct buffer layer upwards. The implementation may or may not copy the buffers passed in, but should work with direct heap/mmap buffers and cannot assume .array() availability. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-9612) HADOOP client IPC layer should differentiate between interrupted threads and interrupts occurring during operations
Gopal V created HADOOP-9612: --- Summary: HADOOP client IPC layer should differentiate between interrupted threads and interrupts occurring during operations Key: HADOOP-9612 URL: https://issues.apache.org/jira/browse/HADOOP-9612 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.5-beta Environment: Ubuntu / x86_64 Reporter: Gopal V Assignee: Gopal V Priority: Trivial Hadoop IPC throws misleading wrapped exceptions when operations are performed on threads which have already been interrupted. When a thread is already interrupted, the client IPC throws the following exception {code} Exception in thread "main" java.io.IOException: ... at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761) at org.apache.hadoop.ipc.Client.call(Client.java:1270) at org.apache.hadoop.ipc.Client.call(Client.java:1218) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) at $Proxy9.getBlockLocations(Unknown Source) {code} This is completely indistinguishable between a real interruption happening to the thread during the client call and very misleading. Before client calls, it would be prudent to check for the currentThread().isInterrupted() flag and throw a different exception to indicate the thread was already interrupted before the operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9570) Configuration.addResource() should only parse the new resource
Gopal V created HADOOP-9570: --- Summary: Configuration.addResource() should only parse the new resource Key: HADOOP-9570 URL: https://issues.apache.org/jira/browse/HADOOP-9570 Project: Hadoop Common Issue Type: Bug Components: conf Environment: Ubuntu LXC Reporter: Gopal V Assignee: Gopal V Priority: Minor Hadoop configuration parsing re-parses all configuration files when a new resource is added to the configuration object. This is wasteful and runs through every deprecation check unnecessarily. {code} JobConf clone = new JobConf(job); clone.addResource(...); {code} will reparse every file including core-site, hdfs-site etc. Resource addition can be taken care of cleanly, if the addResourceObject() call applies the new resource, followed by applying the overlay, without re-parsing any of the older files already loaded into the hashtable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9496) Bad merge of HADOOP-9450 on branch-2 breaks all bin/hadoop calls that need HADOOP_CLASSPATH
Gopal V created HADOOP-9496: --- Summary: Bad merge of HADOOP-9450 on branch-2 breaks all bin/hadoop calls that need HADOOP_CLASSPATH Key: HADOOP-9496 URL: https://issues.apache.org/jira/browse/HADOOP-9496 Project: Hadoop Common Issue Type: Bug Components: bin Affects Versions: 2.0.5-beta Reporter: Gopal V Assignee: Harsh J Priority: Critical Merge of HADOOP-9450 to branch-2 is broken for hadoop-config.sh on trunk http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh?r1=1453486&r2=1469214&pathrev=1469214 vs on branch-2 http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh?r1=1390222&r2=1469215 This is breaking all hadoop client code which needs HADOOP_CLASSPATH to be set correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8971) Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926)
Gopal V created HADOOP-8971: --- Summary: Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926) Key: HADOOP-8971 URL: https://issues.apache.org/jira/browse/HADOOP-8971 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gopal V Priority: Minor Backport (HADOOP-8926) cache-aware improvements made to PureJavaCrc32 to branch-1. With patch, observed significant improvement in crc throughput at larger chunk sizes on branch-1 java.version = 1.6.0_37 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_37-b06 java.vm.version = 20.12-b01 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 3.5.0-17-generic Performance Table (The unit is MB/sec) || Num Bytes ||CRC32 || PureJavaCrc32 || PureJavaCrc32+patch || |1 |16.548 |163.916 |122.289 | |2 |30.046 |213.591 |246.657 | |4 |59.853 |301.521 |322.862 | |8 | 107.435 |491.757 |512.487 | | 16 | 179.278 |509.099 |709.656 | | 32 | 265.496 |651.438 |908.673 | | 64 | 350.994 |766.566 | 1000.775 | |128 | 410.514 |795.719 | 1073.625 | |256 | 434.938 |828.010 | 1188.074 | |512 | 451.997 |883.219 | 1239.929 | | 1024 | 443.369 |902.318 | 1250.906 | | 2048 | 427.923 |864.701 | 1228.599 | | 4096 | 430.814 |863.258 | 1210.633 | | 8192 | 437.079 |883.766 | 1242.756 | |16384 | 433.349 |851.653 | 1211.161 | |32768 | 433.188 |906.142 | 1171.551 | |65536 | 437.420 |890.266 | 1089.157 | | 131072 | 431.189 |869.041 | 1180.171 | | 262144 | 430.636 |910.133 | 1212.344 | | 524288 | 429.365 |889.867 | 1197.842 | |1048576 | 433.012 |887.909 | 1153.719 | |2097152 | 432.520 |861.701 | 1179.221 | |4194304 | 439.791 |899.085 | 1178.736 | |8388608 | 442.422 |856.197 | 1211.953 | | 16777216 | 419.015 |866.560 | 1152.946 | -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
Gopal V created HADOOP-8926: --- Summary: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data Key: HADOOP-8926 URL: https://issues.apache.org/jira/browse/HADOOP-8926 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 2.0.3-alpha Environment: Ubuntu 10.10 i386 Reporter: Gopal V Priority: Trivial While running microbenchmarks for HDFS write codepath, a significant part of the CPU fraction was consumed by the DataChecksum.update(). The attached patch converts the static arrays in CRC32 into a single linear array for a performance boost in the inner loop. milli-seconds for 1Gig (16400 loop over a 64kb chunk) || platform || original || cache-aware || improvement || | x86 | 3894 | 2304 | 40.83 | | x86_64 | 2131 | 1826 | 14 | The performance improvement on x86 is rather larger than the 64bit case, due to the extra register/stack pressure caused by the static arrays. A closer analysis of the PureJavaCrc32 JIT code shows the following assembly fragment {code} 0x40f1e345: mov$0x184,%ecx 0x40f1e34a: mov0x4415b560(%ecx),%ecx ;*getstatic T8_5 ; - PureJavaCrc32::update@95 (line 61) ; {oop('PureJavaCrc32')} 0x40f1e350: mov%ecx,0x2c(%esp) {code} Basically, the static variables T8_0 through to T8_7 are being spilled to the stack because of register pressure. The x86_64 case has a lower likelihood of such pessimistic JIT code due to the increased number of registers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8921) ant build.xml in branch-1 ignores -Dcompile.native
Gopal V created HADOOP-8921: --- Summary: ant build.xml in branch-1 ignores -Dcompile.native Key: HADOOP-8921 URL: https://issues.apache.org/jira/browse/HADOOP-8921 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.2.0 Environment: Mac OS X 10.7.4 Reporter: Gopal V Priority: Trivial ant -Dcompile.native=false still runs autoconf and libtoolize According to ant 1.8 manual, any conditions are checked only after the dependencies are run through. The current if condition in code fails to prevent the autoconf/libtool components from running. Fixing it by moving the if condition up into the "compile-native" target and changing it to a param substitution instead of being evaluated as a condition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira