[jira] [Comment Edited] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531434#comment-17531434 ] John Zhuge edited comment on HADOOP-16159 at 5/3/22 11:33 PM: -- Here is the deadlock in SPARK-39094: thread-1 {noformat} t0: URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(conf)) when conf is not initialized. i.e. `properties` is null t1: conf.get(...) -> conf.getProps() acquires Configuration object -> conf.loadResources -> any code that acquires some sort of JAVA lock, e.g. `URL.streamHandlerLock`{noformat} thread-2 {noformat} t2: random Java code e.g. JceSecurity static initializer -> new URL(): acquire the same URL.streamHandlerLock -> URL.getURLStreamHandler -> FsUrlStreamHandlerFactory.createURLStreamHandler -> FileSystem.getFileSystemClass -> Configuration.getClass -> ... Configuration.getProps acquires Configuration object{noformat} was (Author: jzhuge): thread-1 {noformat} t0: URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(conf)) when conf is not initialized. i.e. `properties` is null t1: conf.get(...) -> conf.getProps() acquires Configuration object -> conf.loadResources -> any code that acquires some sort of JAVA lock, e.g. `URL.streamHandlerLock`{noformat} thread-2 {noformat} t2: random Java code e.g. JceSecurity static initializer -> new URL(): acquire the same URL.streamHandlerLock -> URL.getURLStreamHandler -> FsUrlStreamHandlerFactory.createURLStreamHandler -> FileSystem.getFileSystemClass -> Configuration.getClass -> ... Configuration.getProps acquires Configuration object{noformat} > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x000
[jira] [Comment Edited] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531437#comment-17531437 ] John Zhuge edited comment on HADOOP-16159 at 5/3/22 10:29 PM: -- I think HADOOP-15331 actually fixed this issue because it ensures the cloned Configuration is initialized with a call of getProps() while `FsUrlStreamHandlerFactory(conf)` clones the conf. {code:java} public Configuration(Configuration other) { synchronized(other) { // Make sure we clone a finalized state // Resources like input streams can be processed only once other.getProps(); this.resources = (ArrayList) other.resources.clone(); if (other.properties != null) { this.properties = (Properties)other.properties.clone(); } {code} Can we have another pair of eyes? was (Author: jzhuge): I think HADOOP-15331 actually fixed this issue because it ensures the cloned Configuration is initialized with a call of getProps() {code:java} public Configuration(Configuration other) { synchronized(other) { // Make sure we clone a finalized state // Resources like input streams can be processed only once other.getProps(); this.resources = (ArrayList) other.resources.clone(); if (other.properties != null) { this.properties = (Properties)other.properties.clone(); } {code} Can we have another pair of eyes? > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.
[jira] [Commented] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531437#comment-17531437 ] John Zhuge commented on HADOOP-16159: - I think HADOOP-15331 actually fixed this issue because it ensures the cloned Configuration is initialized with a call of getProps() {code:java} public Configuration(Configuration other) { synchronized(other) { // Make sure we clone a finalized state // Resources like input streams can be processed only once other.getProps(); this.resources = (ArrayList) other.resources.clone(); if (other.properties != null) { this.properties = (Properties)other.properties.clone(); } {code} Can we have another pair of eyes? > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:599) > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159 ] John Zhuge deleted comment on HADOOP-16159: - was (Author: jzhuge): Can we fix it by replacing `conf` with `this.conf` in this constructor? {code:java} public FsUrlStreamHandlerFactory(Configuration conf) { this.conf = new Configuration(conf); // force init of FileSystem code to avoid HADOOP-9041 try { FileSystem.getFileSystemClass("file", conf); === replace with this.conf } catch (IOException io) { throw new RuntimeException(io); } this.handler = new FsUrlStreamHandler(this.conf); } {code} In this way, no other thread can lock this private Configuration object. > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:599) > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531435#comment-17531435 ] John Zhuge edited comment on HADOOP-16159 at 5/3/22 10:19 PM: -- Can we fix it by replacing `conf` with `this.conf` in this constructor? {code:java} public FsUrlStreamHandlerFactory(Configuration conf) { this.conf = new Configuration(conf); // force init of FileSystem code to avoid HADOOP-9041 try { FileSystem.getFileSystemClass("file", conf); === replace with this.conf } catch (IOException io) { throw new RuntimeException(io); } this.handler = new FsUrlStreamHandler(this.conf); } {code} In this way, no other thread can lock this private Configuration object. was (Author: jzhuge): Is this a possible fix by replacing `conf` with `this.conf` in the following code? {code:java} public FsUrlStreamHandlerFactory(Configuration conf) { this.conf = new Configuration(conf); // force init of FileSystem code to avoid HADOOP-9041 try { FileSystem.getFileSystemClass("file", conf); === replace with this.conf } catch (IOException io) { throw new RuntimeException(io); } this.handler = new FsUrlStreamHandler(this.conf); } {code} In this way, no other thread can lock this private Configuration object. > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.
[jira] [Commented] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531435#comment-17531435 ] John Zhuge commented on HADOOP-16159: - Is this a possible fix by replacing `conf` with `this.conf` in the following code? {code:java} public FsUrlStreamHandlerFactory(Configuration conf) { this.conf = new Configuration(conf); // force init of FileSystem code to avoid HADOOP-9041 try { FileSystem.getFileSystemClass("file", conf); === replace with this.conf } catch (IOException io) { throw new RuntimeException(io); } this.handler = new FsUrlStreamHandler(this.conf); } {code} In this way, no other thread can lock this private Configuration object. > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:599) > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16159) Deadlock when using FsUrlStreamHandlerFactory
[ https://issues.apache.org/jira/browse/HADOOP-16159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531434#comment-17531434 ] John Zhuge commented on HADOOP-16159: - thread-1 {noformat} t0: URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(conf)) when conf is not initialized. i.e. `properties` is null t1: conf.get(...) -> conf.getProps() acquires Configuration object -> conf.loadResources -> any code that acquires some sort of JAVA lock, e.g. `URL.streamHandlerLock`{noformat} thread-2 {noformat} t2: random Java code e.g. JceSecurity static initializer -> new URL(): acquire the same URL.streamHandlerLock -> URL.getURLStreamHandler -> FsUrlStreamHandlerFactory.createURLStreamHandler -> FileSystem.getFileSystemClass -> Configuration.getClass -> ... Configuration.getProps acquires Configuration object{noformat} > Deadlock when using FsUrlStreamHandlerFactory > - > > Key: HADOOP-16159 > URL: https://issues.apache.org/jira/browse/HADOOP-16159 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.5, 3.1.2 >Reporter: Ajith S >Priority: Critical > > URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) > This induces the thread lock > Thread 1 : Does load class which will do below > - waiting to lock <0x0005c0c1e5e0> (a > org.apache.hadoop.conf.Configuration) > at > org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:420) > at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:812) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094) > at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091) > at java.security.AccessController.doPrivileged(Native Method) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090) > at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050) > at sun.misc.URLClassPath.getResource(URLClassPath.java:239) > at java.net.URLClassLoader$1.run(URLClassLoader.java:365) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > - locked <0x0005b7991168> (a org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > Thread 2 : Create new URL > - waiting to lock <0x0005b7991168> (a > org.apache.spark.util.MutableURLClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.parsers.DOMParser.(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.(Unknown Source) > at > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > Source) > at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737) > at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696) > at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579) > - locked <0x0005c0c1e5e0> (a org.apache.hadoop.conf.Configuration) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091) > at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363) > at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840) > at > org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74) > at java.net.URL.getURLStreamHandler(URL.java:1142) > at java.net.URL.(URL.java:599) > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop
[jira] [Comment Edited] (HADOOP-18132) S3A to support exponential backoff when throttled
[ https://issues.apache.org/jira/browse/HADOOP-18132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495352#comment-17495352 ] John Zhuge edited comment on HADOOP-18132 at 2/21/22, 2:49 PM: --- S3A already performs retries with exponential backoff on certain S3 errors. For details, please check out [https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Retry_and_Recovery]. was (Author: jzhuge): S3A already performs retries on S3 errors. For details, please check out https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Retry_and_Recovery. > S3A to support exponential backoff when throttled > - > > Key: HADOOP-18132 > URL: https://issues.apache.org/jira/browse/HADOOP-18132 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Holden Karau >Priority: Major > > S3 API has limits which we can exceed when using a large number of > writers/readers/or listers. We should add randomized-exponential back-off to > the s3 client when it encounters: > > com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request > rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18132) S3 exponential backoff
[ https://issues.apache.org/jira/browse/HADOOP-18132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved HADOOP-18132. - Resolution: Not A Problem S3A already performs retries on S3 errors. For details, please check out https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Retry_and_Recovery. > S3 exponential backoff > -- > > Key: HADOOP-18132 > URL: https://issues.apache.org/jira/browse/HADOOP-18132 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Holden Karau >Priority: Major > > S3 API has limits which we can exceed when using a large number of > writers/readers/or listers. We should add randomized-exponential back-off to > the s3 client when it encounters: > > com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request > rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15633: Fix Version/s: 3.1.2 > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, > HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) > at org.apache.hadoop.util.ToolRunner.run(To
[jira] [Updated] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[ https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14206: Fix Version/s: (was: 3.0.4) 2.10.1 2.9.3 > TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature > - > > Key: HADOOP-14206 > URL: https://issues.apache.org/jira/browse/HADOOP-14206 > Project: Hadoop Common > Issue Type: Test > Components: fs, test >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: Jim Brennan >Priority: Major > Fix For: 3.3.0, 2.9.3, 3.1.4, 3.2.2, 2.10.1 > > Attachments: HADOOP-14206-branch-2.10.001.patch, > HADOOP-14206.001.patch > > > https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt: > {noformat} > Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec > <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem > testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed: > 0.19 sec <<< ERROR! > java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: > java.security.SignatureException: Invalid encoding for signature > at com.jcraft.jsch.Session.connect(Session.java:565) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: c
[jira] [Comment Edited] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[ https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054495#comment-17054495 ] John Zhuge edited comment on HADOOP-14206 at 3/8/20, 6:45 PM: -- Pushed to trunk, branch-3.2, branch-3.1, branch-2.10, and branch-2.9. Thanks [~Jim_Brennan] for the contribution! was (Author: jzhuge): Pushed to trunk, branch-3, branch-3.2, branch-3.1, branch-2.10, and branch-2.9. Thanks [~Jim_Brennan] for the contribution! > TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature > - > > Key: HADOOP-14206 > URL: https://issues.apache.org/jira/browse/HADOOP-14206 > Project: Hadoop Common > Issue Type: Test > Components: fs, test >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: Jim Brennan >Priority: Major > Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2 > > Attachments: HADOOP-14206-branch-2.10.001.patch, > HADOOP-14206.001.patch > > > https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt: > {noformat} > Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec > <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem > testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed: > 0.19 sec <<< ERROR! > java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: > java.security.SignatureException: Invalid encoding for signature > at com.jcraft.jsch.Session.connect(Session.java:565) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626
[jira] [Commented] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[ https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054495#comment-17054495 ] John Zhuge commented on HADOOP-14206: - Pushed to trunk, branch-3, branch-3.2, branch-3.1, branch-2.10, and branch-2.9. Thanks [~Jim_Brennan] for the contribution! > TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature > - > > Key: HADOOP-14206 > URL: https://issues.apache.org/jira/browse/HADOOP-14206 > Project: Hadoop Common > Issue Type: Test > Components: fs, test >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: Jim Brennan >Priority: Major > Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2 > > Attachments: HADOOP-14206-branch-2.10.001.patch, > HADOOP-14206.001.patch > > > https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt: > {noformat} > Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec > <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem > testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed: > 0.19 sec <<< ERROR! > java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: > java.security.SignatureException: Invalid encoding for signature > at com.jcraft.jsch.Session.connect(Session.java:565) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) ---
[jira] [Updated] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[ https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14206: Fix Version/s: 3.2.2 3.1.4 3.3.0 3.0.4 Resolution: Fixed Status: Resolved (was: Patch Available) > TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature > - > > Key: HADOOP-14206 > URL: https://issues.apache.org/jira/browse/HADOOP-14206 > Project: Hadoop Common > Issue Type: Test > Components: fs, test >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: Jim Brennan >Priority: Major > Fix For: 3.0.4, 3.3.0, 3.1.4, 3.2.2 > > Attachments: HADOOP-14206-branch-2.10.001.patch, > HADOOP-14206.001.patch > > > https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt: > {noformat} > Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec > <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem > testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed: > 0.19 sec <<< ERROR! > java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: > java.security.SignatureException: Invalid encoding for signature > at com.jcraft.jsch.Session.connect(Session.java:565) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) -
[jira] [Commented] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[ https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053605#comment-17053605 ] John Zhuge commented on HADOOP-14206: - Give a few days for others to chime in, then I will commit. > TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature > - > > Key: HADOOP-14206 > URL: https://issues.apache.org/jira/browse/HADOOP-14206 > Project: Hadoop Common > Issue Type: Test > Components: fs, test >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: Jim Brennan >Priority: Major > Attachments: HADOOP-14206-branch-2.10.001.patch, > HADOOP-14206.001.patch > > > https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt: > {noformat} > Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec > <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem > testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed: > 0.19 sec <<< ERROR! > java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: > java.security.SignatureException: Invalid encoding for signature > at com.jcraft.jsch.Session.connect(Session.java:565) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hado
[jira] [Commented] (HADOOP-14206) TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature
[ https://issues.apache.org/jira/browse/HADOOP-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052596#comment-17052596 ] John Zhuge commented on HADOOP-14206: - [~Jim_Brennan] Thanks for finding the fix and submitting a patch! +1 LGTM > TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature > - > > Key: HADOOP-14206 > URL: https://issues.apache.org/jira/browse/HADOOP-14206 > Project: Hadoop Common > Issue Type: Test > Components: fs, test >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: Jim Brennan >Priority: Major > Attachments: HADOOP-14206-branch-2.10.001.patch, > HADOOP-14206.001.patch > > > https://builds.apache.org/job/PreCommit-HADOOP-Build/11862/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt: > {noformat} > Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 10.454 sec > <<< FAILURE! - in org.apache.hadoop.fs.sftp.TestSFTPFileSystem > testFileExists(org.apache.hadoop.fs.sftp.TestSFTPFileSystem) Time elapsed: > 0.19 sec <<< ERROR! > java.io.IOException: com.jcraft.jsch.JSchException: Session.connect: > java.security.SignatureException: Invalid encoding for signature > at com.jcraft.jsch.Session.connect(Session.java:565) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:168) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > at > org.apache.hadoop.fs.sftp.SFTPConnectionPool.connect(SFTPConnectionPool.java:180) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.connect(SFTPFileSystem.java:149) > at > org.apache.hadoop.fs.sftp.SFTPFileSystem.getFileStatus(SFTPFileSystem.java:663) > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1626) > at > org.apache.hadoop.fs.sftp.TestSFTPFileSystem.testFileExists(TestSFTPFileSystem.java:190) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-u
[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032831#comment-17032831 ] John Zhuge commented on HADOOP-12990: - Yes indeed. Actually 3 apps (lz4 tool, Hadoop, and Spark) claiming the `.lz4` extension. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:372) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268366#comment-15268366 ] John Zhuge edited comment on HADOOP-12990 at 2/8/20 6:36 AM: - [~cavanaug], lz4 command line and hadoop-lz4 use the same lz4 codec library. The difference is only the framing, see my comment and hack on 4/3. Questions for your use case: * Do your JSON files contain a single JSON object or many JSON records? * After ingesting into HDFS, how do you plan to use the data? * Have you considered these splittable container file formats with compression: SequenceFile, RCFile, ORC, Avro, Parquet? In the container, they can choose any Hadoop codec, including LZ4. was (Author: jzhuge): [~cavanaug], lz4 command line and hadoop-lz4 use the same lz4 codec library. The difference is only the framing, see my comment and hack on 4/3. Questions for your use case: * Do your JSON files contain a single JSON object or many JSON records? * After ingesting into HDFS, how do you plan to use the data? * Have considered these splittable container file formats with compression: SequenceFile, RCFile, ORC, Avro, Parquet? In the container, they can choose any Hadoop codec, including LZ4. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:372) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032199#comment-17032199 ] John Zhuge edited comment on HADOOP-12990 at 2/8/20 6:31 AM: - OOM usually indicates format mismatch, e.g., reading a large block size, then trying to allocate memory. After looking into Spark code, I realized I was wrong about using Hadoop codec. Spark uses its own LZ4 codec based on [lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124]. Its javadoc points out: {quote} * @note The wire protocol for this codec is not guaranteed to be compatible across versions * of Spark. This is intended for use as an internal compression utility within a single Spark * application.{quote} Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 tool. Your best bet may be writing a Java decompression application with a compatible version of lz4-java, e.g., 1.4.0 used by Spark 2.3. was (Author: jzhuge): OOM usually indicates format mismatch, e.g., reading a large block size, then trying to allocate memory. After looking into Spark code, I realized I was wrong about using Hadoop codec. Spark uses its own LZ4 codec based on [lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124]. Its javadoc points out: {quote} * @note The wire protocol for this codec is not guaranteed to be compatible across versions * of Spark. This is intended for use as an internal compression utility within a single Spark * application.{quote} Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 tool. Your best bet may be writing a Java decompression application with a compatible version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.ha
[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032202#comment-17032202 ] John Zhuge commented on HADOOP-12990: - [~redriver] since Hadoop is not involved, you might want to file a Spark JIRA and continue the discussion there. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:372) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032199#comment-17032199 ] John Zhuge edited comment on HADOOP-12990 at 2/7/20 8:35 AM: - OOM usually indicates format mismatch, e.g., reading a large block size, then trying to allocate memory. After looking into Spark code, I realized I was wrong about using Hadoop codec. Spark uses its own LZ4 codec based on [lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124]. Its javadoc points out: {quote} * @note The wire protocol for this codec is not guaranteed to be compatible across versions * of Spark. This is intended for use as an internal compression utility within a single Spark * application.{quote} Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 tool. Your best bet may be writing a Java decompression application with a compatible version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0. was (Author: jzhuge): OOM usually indicates format mismatch, e.g., reading a large block size, then trying to allocate memory. After looking into Spark code, I realized I was wrong about using Hadoop codec. Spark uses its own LZ4 codec base on [lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124]. Its javadoc points out: {quote} * @note The wire protocol for this codec is not guaranteed to be compatible across versions * of Spark. This is intended for use as an internal compression utility within a single Spark * application.{quote} Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 tool. Your best bet may be writing a Java decompression program with a matching or compatible version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at
[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032199#comment-17032199 ] John Zhuge commented on HADOOP-12990: - OOM usually indicates format mismatch, e.g., reading a large block size, then trying to allocate memory. After looking into Spark code, I realized I was wrong about using Hadoop codec. Spark uses its own LZ4 codec base on [lz4-java|https://github.com/lz4/lz4-java]. Check out [LZ4CompressionCodec in 2.3.4|https://github.com/apache/spark/blob/v2.3.4/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L113-L124]. Its javadoc points out: {quote} * @note The wire protocol for this codec is not guaranteed to be compatible across versions * of Spark. This is intended for use as an internal compression utility within a single Spark * application.{quote} Not sure whether lz4-java LZ4BlockOutputStream output can be read by Linux lz4 tool. Your best bet may be writing a Java decompression program with a matching or compatible version of lz4-java, e.g., Spark 2.3 uses lz4-java 1.4.0. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:372) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12990) lz4 incompatibility between OS and Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031853#comment-17031853 ] John Zhuge commented on HADOOP-12990: - [~redriver] Thanks for the info. {quote}But it is still confusing for me how did Spark achieves this, {quote} What Spark achieved? Write and/or read? Spark probably uses Hadoop codec on top of my head. > lz4 incompatibility between OS and Hadoop > - > > Key: HADOOP-12990 > URL: https://issues.apache.org/jira/browse/HADOOP-12990 > Project: Hadoop Common > Issue Type: Bug > Components: io, native >Affects Versions: 2.6.0 >Reporter: John Zhuge >Priority: Minor > > {{hdfs dfs -text}} hit exception when trying to view the compression file > created by Linux lz4 tool. > The Hadoop version has HADOOP-11184 "update lz4 to r123", thus it is using > LZ4 library in release r123. > Linux lz4 version: > {code} > $ /tmp/lz4 -h 2>&1 | head -1 > *** LZ4 Compression CLI 64-bits r123, by Yann Collet (Apr 1 2016) *** > {code} > Test steps: > {code} > $ cat 10rows.txt > 001|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 002|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 003|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 004|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 005|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 006|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 007|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 008|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 009|c1|c2|c3|c4|c5|c6|c7|c8|c9 > 010|c1|c2|c3|c4|c5|c6|c7|c8|c9 > $ /tmp/lz4 10rows.txt 10rows.txt.r123.lz4 > Compressed 310 bytes into 105 bytes ==> 33.87% > $ hdfs dfs -put 10rows.txt.r123.lz4 /tmp > $ hdfs dfs -text /tmp/10rows.txt.r123.lz4 > 16/04/01 08:19:07 INFO compress.CodecPool: Got brand-new decompressor [.lz4] > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119) > at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:106) > at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:101) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:315) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:372) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14038) Rename ADLS credential properties
[ https://issues.apache.org/jira/browse/HADOOP-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006221#comment-17006221 ] John Zhuge edited comment on HADOOP-14038 at 12/31/19 7:57 PM: --- [~hvanhovell] Before AdlConfKeys.addDeprecatedKeys is called in AdlFileSystem static initializer, somebody might have already created a Configuration instance and called `get` on a deprecated property, thus we need to reload configurations to apply the ADLS deprecations. Please note ADLS properties live in core-site. The cases may be extremely rare. was (Author: jzhuge): Before AdlConfKeys.addDeprecatedKeys is called in AdlFileSystem static initializer, somebody might have already created a Configuration instance and called `get` on a deprecated property, thus we need to reload configurations to apply the ADLS deprecations. Please note ADLS properties live in core-site. The cases may be extremely rare. > Rename ADLS credential properties > - > > Key: HADOOP-14038 > URL: https://issues.apache.org/jira/browse/HADOOP-14038 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > Attachments: HADOOP-14038.001.patch, HADOOP-14038.002.patch, > HADOOP-14038.003.patch, HADOOP-14038.004.patch, HADOOP-14038.005.patch, > HADOOP-14038.006.patch, HADOOP-14038.007.patch > > > Rename properties with prefix dfs.adls. to fs.adl. > Rename adl.dfs.enable.client.latency.tracker to > adl.enable.client.latency.tracker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14038) Rename ADLS credential properties
[ https://issues.apache.org/jira/browse/HADOOP-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006221#comment-17006221 ] John Zhuge commented on HADOOP-14038: - Before AdlConfKeys.addDeprecatedKeys is called in AdlFileSystem static initializer, somebody might have already created a Configuration instance and called `get` on a deprecated property, thus we need to reload configurations to apply the ADLS deprecations. Please note ADLS properties live in core-site. The cases may be extremely rare. > Rename ADLS credential properties > - > > Key: HADOOP-14038 > URL: https://issues.apache.org/jira/browse/HADOOP-14038 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > Attachments: HADOOP-14038.001.patch, HADOOP-14038.002.patch, > HADOOP-14038.003.patch, HADOOP-14038.004.patch, HADOOP-14038.005.patch, > HADOOP-14038.006.patch, HADOOP-14038.007.patch > > > Rename properties with prefix dfs.adls. to fs.adl. > Rename adl.dfs.enable.client.latency.tracker to > adl.enable.client.latency.tracker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16112) Delete the baseTrashPath's subDir leads to don't modify baseTrashPath
[ https://issues.apache.org/jira/browse/HADOOP-16112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863683#comment-16863683 ] John Zhuge commented on HADOOP-16112: - [~leosun08], thanks for finding and reporting the issue! I can take a look. Just echo [~hexiaoqiao]'s comment, the new unit test passes without any fix, is it valid? I understand race condition is hard to reproduce. BTW, have you or your users hit this issue often? What was the impact other than being surprised by the unexpected trash location? > Delete the baseTrashPath's subDir leads to don't modify baseTrashPath > - > > Key: HADOOP-16112 > URL: https://issues.apache.org/jira/browse/HADOOP-16112 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Lisheng Sun >Priority: Major > Attachments: HADOOP-16112.001.patch, HADOOP-16112.002.patch > > > There is race condition in TrashPolicyDefault#moveToTrash > try { > if (!fs.mkdirs(baseTrashPath, PERMISSION)) > { // create current LOG.warn("Can't create(mkdir) trash directory: " + > baseTrashPath); return false; } > } catch (FileAlreadyExistsException e) { > // find the path which is not a directory, and modify baseTrashPath > // & trashPath, then mkdirs > Path existsFilePath = baseTrashPath; > while (!fs.exists(existsFilePath)) > { existsFilePath = existsFilePath.getParent(); } > {color:#ff}// case{color} > {color:#ff} other thread deletes existsFilePath here ,the results > doesn't meet expectation{color} > {color:#ff} for example{color} > {color:#ff} there is > /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng/b{color} > {color:#ff} when delete /user/u_sunlisheng/b/a. if existsFilePath is > deleted, the result is > /user/u_sunlisheng/.Trash/Current/user/u_sunlisheng+timstamp/b/a{color} > {color:#ff} so when existsFilePath is deleted, don't modify > baseTrashPath. {color} > baseTrashPath = new Path(baseTrashPath.toString().replace( > existsFilePath.toString(), existsFilePath.toString() + Time.now()) > ); > trashPath = new Path(baseTrashPath, trashPath.getName()); > // retry, ignore current failure > --i; > continue; > } catch (IOException e) > { LOG.warn("Can't create trash directory: " + baseTrashPath, e); cause = e; > break; } -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16219) Hadoop branch-2 to set java language version to 1.8
[ https://issues.apache.org/jira/browse/HADOOP-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804234#comment-16804234 ] John Zhuge commented on HADOOP-16219: - +1 On Thu, Mar 28, 2019 at 12:33 PM Steve Loughran (JIRA) -- John > Hadoop branch-2 to set java language version to 1.8 > --- > > Key: HADOOP-16219 > URL: https://issues.apache.org/jira/browse/HADOOP-16219 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.10.0 >Reporter: Steve Loughran >Priority: Major > > Java 7 is long EOL; having branch-2 require it is simply making the release > process a pain (we aren't building, testing, or releasing on java 7 JVMs any > more, are we?). > Staying on java 7 complicates backporting, JAR updates for CVEs (hello > Guava!) &c are becoming impossible. > Proposed: increment javac.version = 1.8 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution
[ https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684427#comment-16684427 ] John Zhuge commented on HADOOP-12125: - Working on it [~medb]. > Retrying UnknownHostException on a proxy does not actually retry hostname > resolution > > > Key: HADOOP-12125 > URL: https://issues.apache.org/jira/browse/HADOOP-12125 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Jason Lowe >Assignee: John Zhuge >Priority: Major > > When RetryInvocationHandler attempts to retry an UnknownHostException the > hostname fails to be resolved again. The InetSocketAddress in the > ConnectionId has cached the fact that the hostname is unresolvable, and when > the proxy tries to setup a new Connection object with that ConnectionId it > checks if the (cached) resolution result is unresolved and immediately throws. > The end result is we sleep and retry for no benefit. The hostname resolution > is never attempted again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution
[ https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610935#comment-16610935 ] John Zhuge commented on HADOOP-12125: - My colleague has a fix for this issue. > Retrying UnknownHostException on a proxy does not actually retry hostname > resolution > > > Key: HADOOP-12125 > URL: https://issues.apache.org/jira/browse/HADOOP-12125 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Jason Lowe >Assignee: John Zhuge >Priority: Major > > When RetryInvocationHandler attempts to retry an UnknownHostException the > hostname fails to be resolved again. The InetSocketAddress in the > ConnectionId has cached the fact that the hostname is unresolvable, and when > the proxy tries to setup a new Connection object with that ConnectionId it > checks if the (cached) resolution result is unresolved and immediately throws. > The end result is we sleep and retry for no benefit. The hostname resolution > is never attempted again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14710) Uber-JIRA: Support AWS Snowball
[ https://issues.apache.org/jira/browse/HADOOP-14710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610081#comment-16610081 ] John Zhuge commented on HADOOP-14710: - [~kdzhao], I have added you as a contributor, just in case you'd like to take on this or any JIRA. > Uber-JIRA: Support AWS Snowball > --- > > Key: HADOOP-14710 > URL: https://issues.apache.org/jira/browse/HADOOP-14710 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > > Support data transfer between Hadoop and [AWS > Snowball|http://docs.aws.amazon.com/snowball/latest/ug/whatissnowball.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14710) Uber-JIRA: Support AWS Snowball
[ https://issues.apache.org/jira/browse/HADOOP-14710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610080#comment-16610080 ] John Zhuge commented on HADOOP-14710: - Thanks for the investigation [~kdzhao] > Uber-JIRA: Support AWS Snowball > --- > > Key: HADOOP-14710 > URL: https://issues.apache.org/jira/browse/HADOOP-14710 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > > Support data transfer between Hadoop and [AWS > Snowball|http://docs.aws.amazon.com/snowball/latest/ug/whatissnowball.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[ https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14060: Target Version/s: 3.3.0 (was: 3.0.1) > HTTP servlet /logs should require authentication and authorization > -- > > Key: HADOOP-14060 > URL: https://issues.apache.org/jira/browse/HADOOP-14060 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Priority: Critical > Attachments: HADOOP-14060-tmp.001.patch > > > HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine > for /conf, /jmx, /logLevel, and /stacks, but not for /logs. > The code in {{AdminAuthorizedServlet#doGet}} for /logs and > {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that > /logs should subject to the same access control as intended by the original > developer. > IMHO this could either be my misconfiguration or there is a bug somewhere in > {{HttpServer2}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[ https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595989#comment-16595989 ] John Zhuge commented on HADOOP-14060: - No time for 3.2. Moved out. > HTTP servlet /logs should require authentication and authorization > -- > > Key: HADOOP-14060 > URL: https://issues.apache.org/jira/browse/HADOOP-14060 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Priority: Critical > Attachments: HADOOP-14060-tmp.001.patch > > > HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine > for /conf, /jmx, /logLevel, and /stacks, but not for /logs. > The code in {{AdminAuthorizedServlet#doGet}} for /logs and > {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that > /logs should subject to the same access control as intended by the original > developer. > IMHO this could either be my misconfiguration or there is a bug somewhere in > {{HttpServer2}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[ https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14060: Target Version/s: 3.0.1 (was: 3.0.1, 3.2.0) > HTTP servlet /logs should require authentication and authorization > -- > > Key: HADOOP-14060 > URL: https://issues.apache.org/jira/browse/HADOOP-14060 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Priority: Critical > Attachments: HADOOP-14060-tmp.001.patch > > > HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine > for /conf, /jmx, /logLevel, and /stacks, but not for /logs. > The code in {{AdminAuthorizedServlet#doGet}} for /logs and > {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that > /logs should subject to the same access control as intended by the original > developer. > IMHO this could either be my misconfiguration or there is a bug somewhere in > {{HttpServer2}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594308#comment-16594308 ] John Zhuge commented on HADOOP-15633: - Committed to branch-3.1 and branch-3.0. > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, > HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShe
[jira] [Updated] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15633: Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk. Tranks [~ferhui] for discovering and fixing the issue! > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, > HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.C
[jira] [Assigned] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution
[ https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-12125: --- Assignee: John Zhuge (was: Rushabh S Shah) > Retrying UnknownHostException on a proxy does not actually retry hostname > resolution > > > Key: HADOOP-12125 > URL: https://issues.apache.org/jira/browse/HADOOP-12125 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Jason Lowe >Assignee: John Zhuge >Priority: Major > > When RetryInvocationHandler attempts to retry an UnknownHostException the > hostname fails to be resolved again. The InetSocketAddress in the > ConnectionId has cached the fact that the hostname is unresolvable, and when > the proxy tries to setup a new Connection object with that ConnectionId it > checks if the (cached) resolution result is unresolved and immediately throws. > The end result is we sleep and retry for no benefit. The hostname resolution > is never attempted again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583097#comment-16583097 ] John Zhuge commented on HADOOP-15633: - +1 Patch 005 LGTM > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, > HADOOP-15633.003.patch, HADOOP-15633.004.patch, HADOOP-15633.005.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) > at org.apache.hadoop.util.Tool
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582039#comment-16582039 ] John Zhuge commented on HADOOP-15633: - Great work [~ferhui]! TrashPolicyDefault.java * #158: Remove the space after “new Path ” * #162: Adding “continue” is not quite right, because it takes us to the “for” loop that only loops twice, and for a different purpose. As a quick hack, you might add a line “—I;” before “continue;” Any better solution? > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch, > HADOOP-15633.003.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581903#comment-16581903 ] John Zhuge commented on HADOOP-15633: - [~ferhui] Thanks for discovering the issue and creating a patch! TrashPolicyDefault.java * #152: The line exceeds 80 chars * #154: Add space after “while” * #158: Do not wrap a line before a comma * #160: Do not mkdir here inside catch block, loop back to the try block instead, for 2 reasons: 1) there might be more conflicts, 2) mkdirs may throw exception. TestTrash.java * The new unit test seems to be inserted into the middle of another test case “Verify old checkpoint format is recognized”. * Can we create a new test case if possible? Test case “trashShell” is way too long! > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apach
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579224#comment-16579224 ] John Zhuge commented on HADOOP-15633: - Thanks for patch 002. I will review. > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch, HADOOP-15633.002.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.h
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559932#comment-16559932 ] John Zhuge commented on HADOOP-15633: - After "hadoop fs -rm /user/hadoop/aaa/bbb/ccc", expect "/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc", since "/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb" is an existing file. > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559195#comment-16559195 ] John Zhuge commented on HADOOP-15633: - [~ferhui] Sorry, what I meant was "name conflict resolution is not done for a parent dir on the path." for your command: {noformat} hadoop fs -rm /user/hadoop/aaa/bbb/ccc{noformat} > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:114) > at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:95) > at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:118) > at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:105) > at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317) > at > org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289) > at > org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271) > at > org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255) > at > org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) > at org.apache.hadoop.fs.shell.Command.run(Command.java:165) > at org.apache.hadoop.fs.FsShe
[jira] [Comment Edited] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558647#comment-16558647 ] John Zhuge edited comment on HADOOP-15633 at 7/26/18 5:29 PM: -- If you run the following command after your test steps: {noformat} hadoop fs -rm -r /user/hadoop/aaa/bbb {noformat} You will see the name conflict resolution, "bbb" -> "bbb1532625817927": {noformat} # hadoop fs -ls -R /user/hadoop/.Trash drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user/hadoop drwx-- - hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa -rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb drwxr-xr-x - hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927 -rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc{noformat} Looks like name conflict resolution is not done for a parent dir on the path. was (Author: jzhuge): If you run the following command after your test steps: {noformat} hadoop fs -rm -r /user/hadoop/aaa/bbb {noformat} You will see the name conflict resolution, "bbb" -> "bbb1532625817927": {noformat} # hadoop fs -ls -R /user/hadoop/.Trash drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user/hadoop drwx-- - hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa -rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb drwxr-xr-x - hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927 -rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc{noformat} Looks like name conflict resolution is not done for a parent dir on the path. > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInst
[jira] [Commented] (HADOOP-15633) fs.TrashPolicyDefault: Can't create trash directory
[ https://issues.apache.org/jira/browse/HADOOP-15633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558647#comment-16558647 ] John Zhuge commented on HADOOP-15633: - If you run the following command after your test steps: {noformat} hadoop fs -rm -r /user/hadoop/aaa/bbb {noformat} You will see the name conflict resolution, "bbb" -> "bbb1532625817927": {noformat} # hadoop fs -ls -R /user/hadoop/.Trash drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user drwx-- - hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user/hadoop drwx-- - hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa -rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:21 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb drwxr-xr-x - hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927 -rw-r--r-- 3 hadoop hadoop 0 2018-07-26 17:23 /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb1532625817927/ccc{noformat} Looks like name conflict resolution is not done for a parent dir on the path. > fs.TrashPolicyDefault: Can't create trash directory > --- > > Key: HADOOP-15633 > URL: https://issues.apache.org/jira/browse/HADOOP-15633 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.8.3, 3.1.0, 3.0.3, 2.7.7 >Reporter: Fei Hui >Assignee: Fei Hui >Priority: Major > Attachments: HADOOP-15633.001.patch > > > Reproduce it as follow > {code:java} > hadoop fs -mkdir /user/hadoop/aaa > hadoop fs -touchz /user/hadoop/aaa/bbb > hadoop fs -rm /user/hadoop/aaa/bbb > hadoop fs -mkdir /user/hadoop/aaa/bbb > hadoop fs -touchz /user/hadoop/aaa/bbb/ccc > hadoop fs -rm /user/hadoop/aaa/bbb/ccc > {code} > Then we get errors > {code:java} > 18/07/26 17:55:24 WARN fs.TrashPolicyDefault: Can't create trash directory: > hdfs://xxx/user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > org.apache.hadoop.fs.FileAlreadyExistsException: Path is not a directory: > /user/hadoop/.Trash/Current/user/hadoop/aaa/bbb > at > org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:65) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3961) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2115) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2111) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1867) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) > at > org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3002) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2970) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) > at > org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) > at > org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) > at > org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:136) > at org.apache.hadoop.fs.Trash.moveT
[jira] [Commented] (HADOOP-14435) TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed
[ https://issues.apache.org/jira/browse/HADOOP-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506606#comment-16506606 ] John Zhuge commented on HADOOP-14435: - Saw it on 3.0.3 (RC0). > TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed > -- > > Key: HADOOP-14435 > URL: https://issues.apache.org/jira/browse/HADOOP-14435 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, fs/adl >Affects Versions: 2.9.0, 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > > Saw the following assertion failure in branch-2 and trunk: > {noformat} > Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 80.189 sec > <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive > testMkdirsWithUmask(org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive) > Time elapsed: 0.71 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<461> but was:<456> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:219) > at junit.framework.Assert.assertEquals(Assert.java:226) > at junit.framework.TestCase.assertEquals(TestCase.java:392) > at > org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:242) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at > org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive.runTest(TestAdlFileSystemContractLive.java:59) > Results : > Failed tests: > > TestAdlFileSystemContractLive.runTest:59->FileSystemContractBaseTest.testMkdirsWithUmask:242 > expected:<461> but was:<456> > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14435) TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed
[ https://issues.apache.org/jira/browse/HADOOP-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14435: Issue Type: Bug (was: Task) > TestAdlFileSystemContractLive#testMkdirsWithUmask assertion failed > -- > > Key: HADOOP-14435 > URL: https://issues.apache.org/jira/browse/HADOOP-14435 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, fs/adl >Affects Versions: 2.9.0, 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > > Saw the following assertion failure in branch-2 and trunk: > {noformat} > Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 80.189 sec > <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive > testMkdirsWithUmask(org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive) > Time elapsed: 0.71 sec <<< FAILURE! > junit.framework.AssertionFailedError: expected:<461> but was:<456> > at junit.framework.Assert.fail(Assert.java:57) > at junit.framework.Assert.failNotEquals(Assert.java:329) > at junit.framework.Assert.assertEquals(Assert.java:78) > at junit.framework.Assert.assertEquals(Assert.java:219) > at junit.framework.Assert.assertEquals(Assert.java:226) > at junit.framework.TestCase.assertEquals(TestCase.java:392) > at > org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsWithUmask(FileSystemContractBaseTest.java:242) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at junit.framework.TestCase.runTest(TestCase.java:176) > at > org.apache.hadoop.fs.adl.live.TestAdlFileSystemContractLive.runTest(TestAdlFileSystemContractLive.java:59) > Results : > Failed tests: > > TestAdlFileSystemContractLive.runTest:59->FileSystemContractBaseTest.testMkdirsWithUmask:242 > expected:<461> but was:<456> > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14175) NPE when ADL store URI contains underscore
[ https://issues.apache.org/jira/browse/HADOOP-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421110#comment-16421110 ] John Zhuge commented on HADOOP-14175: - [~keuler] Thanks for the contribution. I have added you as a contributor to Hadoop Common project. > NPE when ADL store URI contains underscore > -- > > Key: HADOOP-14175 > URL: https://issues.apache.org/jira/browse/HADOOP-14175 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: Xiaobing Zhou >Priority: Minor > Labels: newbie, supportability > > Please note the underscore {{_}} in the store name {{jzhuge_adls}}. Same NPE > wherever the underscore in the URI. > {noformat} > $ bin/hadoop fs -ls adl://jzhuge_adls.azuredatalakestore.net/ > -ls: Fatal internal error > java.lang.NullPointerException > at > org.apache.hadoop.fs.adl.AdlFileSystem.initialize(AdlFileSystem.java:145) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3257) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3306) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3274) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) > at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:249) > at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:232) > at > org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) > at org.apache.hadoop.fs.shell.Command.run(Command.java:176) > at org.apache.hadoop.fs.FsShell.run(FsShell.java:326) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) > at org.apache.hadoop.fs.FsShell.main(FsShell.java:389) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution
[ https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399707#comment-16399707 ] John Zhuge commented on HADOOP-12125: - [~shahrs87] and [~jlowe], any progress? We hit the same issue when the non-HA NN went down and AWS spun up another NN instance with a different IP address. Both Job History Server and Spark History Server were stuck because NameNodeProxy held on to the old IP address. > Retrying UnknownHostException on a proxy does not actually retry hostname > resolution > > > Key: HADOOP-12125 > URL: https://issues.apache.org/jira/browse/HADOOP-12125 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Jason Lowe >Assignee: Rushabh S Shah >Priority: Major > > When RetryInvocationHandler attempts to retry an UnknownHostException the > hostname fails to be resolved again. The InetSocketAddress in the > ConnectionId has cached the fact that the hostname is unresolvable, and when > the proxy tries to setup a new Connection object with that ConnectionId it > checks if the (cached) resolution result is unresolved and immediately throws. > The end result is we sleep and retry for no benefit. The hostname resolution > is never attempted again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12832) Implement unix-like 'FsShell -touch'
[ https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388350#comment-16388350 ] John Zhuge commented on HADOOP-12832: - Thanks [~bharatviswa] for taking this up. > Implement unix-like 'FsShell -touch' > - > > Key: HADOOP-12832 > URL: https://issues.apache.org/jira/browse/HADOOP-12832 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.6.4 >Reporter: Gera Shegalov >Assignee: Bharat Viswanadham >Priority: Major > > We needed to touch a bunch of files as in > https://en.wikipedia.org/wiki/Touch_(Unix) . > Because FsShell does not expose FileSystem#setTimes , we had to do it > programmatically in Scalding REPL. Seems like it should not be this > complicated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12832) Implement unix-like 'FsShell -touch'
[ https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388282#comment-16388282 ] John Zhuge commented on HADOOP-12832: - Unfortunately don't have time to work on it for a while. > Implement unix-like 'FsShell -touch' > - > > Key: HADOOP-12832 > URL: https://issues.apache.org/jira/browse/HADOOP-12832 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.6.4 >Reporter: Gera Shegalov >Priority: Major > > We needed to touch a bunch of files as in > https://en.wikipedia.org/wiki/Touch_(Unix) . > Because FsShell does not expose FileSystem#setTimes , we had to do it > programmatically in Scalding REPL. Seems like it should not be this > complicated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-12832) Implement unix-like 'FsShell -touch'
[ https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-12832: --- Assignee: (was: John Zhuge) > Implement unix-like 'FsShell -touch' > - > > Key: HADOOP-12832 > URL: https://issues.apache.org/jira/browse/HADOOP-12832 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.6.4 >Reporter: Gera Shegalov >Priority: Major > > We needed to touch a bunch of files as in > https://en.wikipedia.org/wiki/Touch_(Unix) . > Because FsShell does not expose FileSystem#setTimes , we had to do it > programmatically in Scalding REPL. Seems like it should not be this > complicated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-12832) Implement unix-like 'FsShell -touch'
[ https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-12832: --- Assignee: John Zhuge > Implement unix-like 'FsShell -touch' > - > > Key: HADOOP-12832 > URL: https://issues.apache.org/jira/browse/HADOOP-12832 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.6.4 >Reporter: Gera Shegalov >Assignee: John Zhuge >Priority: Major > > We needed to touch a bunch of files as in > https://en.wikipedia.org/wiki/Touch_(Unix) . > Because FsShell does not expose FileSystem#setTimes , we had to do it > programmatically in Scalding REPL. Seems like it should not be this > complicated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HADOOP-12832) Implement unix-like 'FsShell -touch'
[ https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-12832 stopped by John Zhuge. --- > Implement unix-like 'FsShell -touch' > - > > Key: HADOOP-12832 > URL: https://issues.apache.org/jira/browse/HADOOP-12832 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.6.4 >Reporter: Gera Shegalov >Assignee: John Zhuge >Priority: Major > > We needed to touch a bunch of files as in > https://en.wikipedia.org/wiki/Touch_(Unix) . > Because FsShell does not expose FileSystem#setTimes , we had to do it > programmatically in Scalding REPL. Seems like it should not be this > complicated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-12832) Implement unix-like 'FsShell -touch'
[ https://issues.apache.org/jira/browse/HADOOP-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-12832: --- Assignee: (was: John Zhuge) > Implement unix-like 'FsShell -touch' > - > > Key: HADOOP-12832 > URL: https://issues.apache.org/jira/browse/HADOOP-12832 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.6.4 >Reporter: Gera Shegalov >Priority: Major > > We needed to touch a bunch of files as in > https://en.wikipedia.org/wiki/Touch_(Unix) . > Because FsShell does not expose FileSystem#setTimes , we had to do it > programmatically in Scalding REPL. Seems like it should not be this > complicated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366130#comment-16366130 ] John Zhuge commented on HADOOP-13972: - Thanks [~ste...@apache.org] ! > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > Fix For: 3.1.0, 2.10.0, 3.0.2 > > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[ https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366129#comment-16366129 ] John Zhuge commented on HADOOP-14060: - Sorry, don't have time to work on it now. Unassigned the JIRA. > HTTP servlet /logs should require authentication and authorization > -- > > Key: HADOOP-14060 > URL: https://issues.apache.org/jira/browse/HADOOP-14060 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Priority: Critical > Attachments: HADOOP-14060-tmp.001.patch > > > HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine > for /conf, /jmx, /logLevel, and /stacks, but not for /logs. > The code in {{AdminAuthorizedServlet#doGet}} for /logs and > {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that > /logs should subject to the same access control as intended by the original > developer. > IMHO this could either be my misconfiguration or there is a bug somewhere in > {{HttpServer2}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14060) HTTP servlet /logs should require authentication and authorization
[ https://issues.apache.org/jira/browse/HADOOP-14060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-14060: --- Assignee: (was: John Zhuge) > HTTP servlet /logs should require authentication and authorization > -- > > Key: HADOOP-14060 > URL: https://issues.apache.org/jira/browse/HADOOP-14060 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Priority: Critical > Attachments: HADOOP-14060-tmp.001.patch > > > HADOOP-14047 makes KMS call {{HttpServer2#setACL}}. Access control works fine > for /conf, /jmx, /logLevel, and /stacks, but not for /logs. > The code in {{AdminAuthorizedServlet#doGet}} for /logs and > {{ConfServlet#doGet}} for /conf are quite similar. This makes me believe that > /logs should subject to the same access control as intended by the original > developer. > IMHO this could either be my misconfiguration or there is a bug somewhere in > {{HttpServer2}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363513#comment-16363513 ] John Zhuge commented on HADOOP-13972: - Thanks [~ssonker] for the PR. Added you as a contributor to Hadoop Common and assigned the Jira to you. > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13972) ADLS to support per-store configuration
[ https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-13972: --- Assignee: Sharad Sonker > ADLS to support per-store configuration > --- > > Key: HADOOP-13972 > URL: https://issues.apache.org/jira/browse/HADOOP-13972 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: Sharad Sonker >Priority: Major > > Useful when distcp needs to access 2 Data Lake stores with different SPIs. > Of course, a workaround is to grant the same SPI access permission to both > stores, but sometimes it might not be feasible. > One idea is to embed the store name in the configuration property names, > e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted > first, then fall back to the global keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[ https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved HADOOP-14961. - Resolution: Duplicate Fixed by HADOOP-14816. > Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed > -- > > Key: HADOOP-14961 > URL: https://issues.apache.org/jira/browse/HADOOP-14961 > Project: Hadoop Common > Issue Type: Bug > Components: build, test >Affects Versions: 3.1.0 >Reporter: John Zhuge >Priority: Major > > https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console > {noformat} > Downloading Oracle Java 8... > [0m[91m--2017-10-18 18:28:11-- > http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz > > [0m[91mResolving download.oracle.com (download.oracle.com)... > [0m[91m23.59.190.131, 23.59.190.130 > Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... > [0m[91mconnected. > HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily > Location: > https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz > [following] > --2017-10-18 18:28:11-- > https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz > > [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)... > [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e > Connecting to edelivery.oracle.com > (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected. > [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved > Temporarily > Location: > http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c > [following] > --2017-10-18 18:28:11-- > http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c > > Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... > [0m[91mconnected. > HTTP request sent, awaiting response... [0m[91m404 Not Found > [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found. > [0m[91mdownload failed > Oracle JDK 8 is NOT installed. > {noformat} > Looks like Oracle JDK 8u144 is no longer available for download using that > link. 8u151 and 8u152 are available. > Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs > failed the same way, all on build host H1 and H6. > [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial" > for a long term fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14961) Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed
[ https://issues.apache.org/jira/browse/HADOOP-14961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reopened HADOOP-14961: - > Docker failed to build yetus/hadoop:0de40f0: Oracle JDK 8 is NOT installed > -- > > Key: HADOOP-14961 > URL: https://issues.apache.org/jira/browse/HADOOP-14961 > Project: Hadoop Common > Issue Type: Bug > Components: build, test >Affects Versions: 3.1.0 >Reporter: John Zhuge >Priority: Major > > https://builds.apache.org/job/PreCommit-HADOOP-Build/13546/console > {noformat} > Downloading Oracle Java 8... > [0m[91m--2017-10-18 18:28:11-- > http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz > > [0m[91mResolving download.oracle.com (download.oracle.com)... > [0m[91m23.59.190.131, 23.59.190.130 > Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... > [0m[91mconnected. > HTTP request sent, awaiting response... [0m[91m302 Moved Temporarily > Location: > https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz > [following] > --2017-10-18 18:28:11-- > https://edelivery.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz > > [0m[91mResolving edelivery.oracle.com (edelivery.oracle.com)... > [0m[91m23.39.16.136, 2600:1409:a:39c::2d3e, 2600:1409:a:39e::2d3e > Connecting to edelivery.oracle.com > (edelivery.oracle.com)|23.39.16.136|:443... [0m[91mconnected. > [0m[91mHTTP request sent, awaiting response... [0m[91m302 Moved > Temporarily > Location: > http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c > [following] > --2017-10-18 18:28:11-- > http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz?AuthParam=1508351411_3d448519d55b9741af15953ef5049a7c > > Connecting to download.oracle.com (download.oracle.com)|23.59.190.131|:80... > [0m[91mconnected. > HTTP request sent, awaiting response... [0m[91m404 Not Found > [0m[91m2017-10-18 18:28:12 ERROR 404: Not Found. > [0m[91mdownload failed > Oracle JDK 8 is NOT installed. > {noformat} > Looks like Oracle JDK 8u144 is no longer available for download using that > link. 8u151 and 8u152 are available. > Many of last 10 https://builds.apache.org/job/PreCommit-HADOOP-Build/ jobs > failed the same way, all on build host H1 and H6. > [~aw] has a patch available in HADOOP-14816 "Update Dockerfile to use Xenial" > for a long term fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14651) Update okhttp version to 2.7.5
[ https://issues.apache.org/jira/browse/HADOOP-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343006#comment-16343006 ] John Zhuge commented on HADOOP-14651: - Fine with me. > Update okhttp version to 2.7.5 > -- > > Key: HADOOP-14651 > URL: https://issues.apache.org/jira/browse/HADOOP-14651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-14651.001.patch, HADOOP-14651.002.patch, > HADOOP-14651.003.patch, HADOOP-14651.004.patch > > > The current artifact is: > com.squareup.okhttp:okhttp:2.4.0 > That version could either be bumped to 2.7.5 (the latest of that line), or > use the latest artifact: > com.squareup.okhttp3:okhttp:3.8.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15134) ADL problems parsing JSON responses to include error details
[ https://issues.apache.org/jira/browse/HADOOP-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15134: Labels: supportability (was: ) > ADL problems parsing JSON responses to include error details > > > Key: HADOOP-15134 > URL: https://issues.apache.org/jira/browse/HADOOP-15134 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Affects Versions: 3.0.0 >Reporter: Steve Loughran > Labels: supportability > > Currently any failure of ADL's response JSON parsing results in the error > text like "Unexpected error happened reading response stream or parsing JSon > from rename()";". This is not useful. Fix by: including the exception text, > logging the Ex & info -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282508#comment-16282508 ] John Zhuge commented on HADOOP-15056: - +1 LGTM > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, > HADOOP-15056.003.patch, HADOOP-15056.004.patch, HADOOP-15056.005.patch, > HADOOP-15056.006.patch, HADOOP-15056.007.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278301#comment-16278301 ] John Zhuge commented on HADOOP-15056: - Here is a table of all scenarios and the comparisons of patches and original behavior. Logging and exception only occurs when explicitly indicated. || instanceof StreamCapabilities || has UNBUFFER capaility || instanceof CanUnbuffer || Patch 004 behavior || Patch 002 behavior || Original behavior || | Yes | Yes | Yes | Call unbuffer | Call unbuffer | Call unbuffer | | Yes | Yes | No | UOE | UOE | UOE | | Yes | No | N/A | LOG | No-op | No-op | | No | N/A | Yes | LOG | Call unbuffer | No-op | | No | N/A | No | No-op | UOE | No-op | > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch, > HADOOP-15056.003.patch, HADOOP-15056.004.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14294) Rename ADLS mountpoint properties
[ https://issues.apache.org/jira/browse/HADOOP-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276069#comment-16276069 ] John Zhuge commented on HADOOP-14294: - Excellent! I have added you as a Hadoop Common contributor and assigned this JIRA to you. > Rename ADLS mountpoint properties > - > > Key: HADOOP-14294 > URL: https://issues.apache.org/jira/browse/HADOOP-14294 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: Rohan Garg > > Follow up to HADOOP-14038. Rename the prefix of > {{dfs.adls..mountpoint}} and {{dfs.adls..hostname}} to > {{fs.adl.}}. > Borrow code from > https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch > and add a few unit tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14294) Rename ADLS mountpoint properties
[ https://issues.apache.org/jira/browse/HADOOP-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-14294: --- Assignee: Rohan Garg (was: John Zhuge) > Rename ADLS mountpoint properties > - > > Key: HADOOP-14294 > URL: https://issues.apache.org/jira/browse/HADOOP-14294 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: Rohan Garg > > Follow up to HADOOP-14038. Rename the prefix of > {{dfs.adls..mountpoint}} and {{dfs.adls..hostname}} to > {{fs.adl.}}. > Borrow code from > https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch > and add a few unit tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272192#comment-16272192 ] John Zhuge commented on HADOOP-15056: - IMHO, it is not a good idea to swallow CCE and just log. In the try block, StreamCapabilities#hasCapabilities has to be called, because this is the new algorithm introduced by HADOOP-15012.: an input stream should be queried about its capability UNBUFFER before unbuffer is called. Casting to CanUnbuffer should always succeed in this case. > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272100#comment-16272100 ] John Zhuge commented on HADOOP-15056: - No worry. Good discussion. IMPALA-5909 complained excessive UOE exceptions. > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271601#comment-16271601 ] John Zhuge commented on HADOOP-15056: - StreamCapabilities was added by HDFS-11644. It is considered a more flexible approach than {{instanceof}}. bq. it may prevent helpful debugging information to be passed back to the user Could you elaborate? > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15056: Attachment: HADOOP-15056.002.patch Hi [~jackbearden], I think StreamCapabilitiesPolicy#unbuffer should still call hasCapability if the input stream implements interface StreamCapabilities. Uploaded patch 002. Let me know what you think. Testing Done * TestCryptoStreams * TestCryptoStreamsForLocalFS * TestCryptoStreamsNormal * TestHdfsCryptoStreams * TestUnbuffer > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch, HADOOP-15056.002.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15056) Fix TestUnbuffer#testUnbufferException failure
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15056: Summary: Fix TestUnbuffer#testUnbufferException failure (was: Fix for TestUnbuffer#testUnbufferException Test) > Fix TestUnbuffer#testUnbufferException failure > -- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix for TestUnbuffer#testUnbufferException Test
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269484#comment-16269484 ] John Zhuge commented on HADOOP-15056: - Just back from vacation, sorry for the delay. I will review shortly. > Fix for TestUnbuffer#testUnbufferException Test > --- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.001.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15056) Fix for TestUnbuffer#testUnbufferException Test
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16260286#comment-16260286 ] John Zhuge commented on HADOOP-15056: - [~jackbearden] Welcome to the Hadoop Open Source community! Your contribution is very much appreciated! I have added you as a contributor and assigned the JIRA to you. Please take a look at https://wiki.apache.org/hadoop/HowToContribute. The section "Naming your patch" recommends the version number to be embedded in the patch file name. I will take a look at your patch shortly. > Fix for TestUnbuffer#testUnbufferException Test > --- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15056) Fix for TestUnbuffer#testUnbufferException Test
[ https://issues.apache.org/jira/browse/HADOOP-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-15056: --- Assignee: Jack Bearden > Fix for TestUnbuffer#testUnbufferException Test > --- > > Key: HADOOP-15056 > URL: https://issues.apache.org/jira/browse/HADOOP-15056 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Jack Bearden >Assignee: Jack Bearden >Priority: Minor > Attachments: HADOOP-15056.patch > > > Hello! I am a new contributor and actually contributing to open source for > the very first time. :) > I pulled down Hadoop today and when running the tests I encountered a failure > with the TestUnbuffer#testUnbufferException test. > The unbuffer code has recently gone through some changes and I believe this > test case may have been overlooked. Using today's git commit > (659e85e304d070f9908a96cf6a0e1cbafde6a434), and upon running the test case, > there is an expect mock for an exception UnsupportedOperationException that > is no longer being thrown. > It would appear that a test like this would be valuable so my initial > proposed patch did not remove it. Instead, I removed the conditions that were > guarding the cast from being able to fire -- as was the previous behavior. > Now when we encounter an object that doesn't have the UNBUFFERED > StreamCapability, it will throw an error as it did prior to the recent > changes. > Please review and let me know what you think! :D -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-15012: --- Assignee: John Zhuge > Add readahead, dropbehind, and unbuffer to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: John Zhuge > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15012: Target Version/s: 3.1.0 (was: 2.10.0) > Add readahead, dropbehind, and unbuffer to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.1.0 > > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved HADOOP-15012. - Resolution: Fixed Fix Version/s: 3.1.0 Committed to trunk together with HADOOP-14872. Code review was done there. {noformat} 6c32ddad302 HADOOP-14872. CryptoInputStream should implement unbuffer. Contributed by John Zhuge. bf6a660232b HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. {noformat} > Add readahead, dropbehind, and unbuffer to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.1.0 > > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~xiaochen] and [~steve_l] for the great reviews! > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.1.0 > > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246199#comment-16246199 ] John Zhuge commented on HADOOP-14872: - Ran "test-patch" locally and got all +1s except these expected javac deprecation warnings. Committing. {noformat} [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[42,46] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[251,30] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java:[252,33] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[32,46] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[40,46] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[201,16] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java:[204,16] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[362,25] [deprecation] StreamCapability in StreamCapabilities has been deprecated [WARNING] /home/jzhuge/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java:[364,25] [deprecation] StreamCapability in StreamCapabilities has been deprecated {noformat} > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Open (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Patch Available (was: Open) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Open (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Patch Available (was: Open) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: In Progress (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Patch Available (was: In Progress) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: In Progress (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Patch Available (was: In Progress) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Component/s: security > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, security >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Open (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Patch Available (was: Open) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Patch Available (was: Open) Tried to trigger pre-commit. > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: Open (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Target Version/s: 3.1.0 Status: Patch Available (was: In Progress) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Status: In Progress (was: Patch Available) > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15012: Summary: Add readahead, dropbehind, and unbuffer to StreamCapabilities (was: Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities) > Add readahead, dropbehind, and unbuffer to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15012) Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-15012: Summary: Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities (was: Enhance StreamCapabilities with READAHEAD, DROPBEHIND, and UNBUFFER) > Add READAHEAD, DROPBEHIND, and UNBUFFER to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge >Priority: Major > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14872) CryptoInputStream should implement unbuffer
[ https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14872: Attachment: HADOOP-14872.013.patch Thanks Xiao and Steve for the patience and great discussions. Patch 013 * Incorporate Xiao and Steve's comments * Add {{StreamCapabilitiesPolicy#unbuffer}} to implement the policy for unbuffer. Java 8 static method for interface CanUnbuffer could be a good choice but we can't use it due to backport concern. * Use a table in FileSystem.md to list capabilities * Plan to commit StreamCapabilities enhancements in HADOOP-15012 first, then check in CryptoInputStream changes here. > CryptoInputStream should implement unbuffer > --- > > Key: HADOOP-14872 > URL: https://issues.apache.org/jira/browse/HADOOP-14872 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.6.4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Major > Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch, > HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch, > HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch, > HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch, > HADOOP-14872.012.patch, HADOOP-14872.013.patch > > > Discovered in IMPALA-5909. > Opening an encrypted HDFS file returns a chain of wrapped input streams: > {noformat} > HdfsDataInputStream > CryptoInputStream > DFSInputStream > {noformat} > If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer, > FSDataInputStream#unbuffer will be called: > {code:java} > try { > ((CanUnbuffer)in).unbuffer(); > } catch (ClassCastException e) { > throw new UnsupportedOperationException("this stream does not " + > "support unbuffering."); > } > {code} > If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If > the application is not careful, tons of UOEs will show up in logs. > In comparison, opening an non-encrypted HDFS file returns this chain: > {noformat} > HdfsDataInputStream > DFSInputStream > {noformat} > DFSInputStream implements CanUnbuffer. > It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons: > * Release buffer, cache, or any other resource when instructed > * Able to call its wrapped DFSInputStream unbuffer > * Avoid the UOE described above. Applications may not handle the UOE very > well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15012) Enhance StreamCapabilities with READAHEAD, DROPBEHIND, and UNBUFFER
John Zhuge created HADOOP-15012: --- Summary: Enhance StreamCapabilities with READAHEAD, DROPBEHIND, and UNBUFFER Key: HADOOP-15012 URL: https://issues.apache.org/jira/browse/HADOOP-15012 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.9.0 Reporter: John Zhuge Priority: Major A split from HADOOP-14872 to track changes that enhance StreamCapabilities class with READAHEAD, DROPBEHIND, and UNBUFFER capability. Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org